JP2020091832A - Information processing system, information processing method, and program - Google Patents

Abstract

To provide an information processing system and others capable of preventing the occurrence of shoplifting without increasing burden to store workers and others.SOLUTION: A plurality of imaging devices are dispersedly arranged in a store. A server detects whether a person whose image has been taken in image data has picked up a product displayed in the store or not on the basis of image data obtained by each of the plurality of imaging devices. When the person in the image data is detected to have picked up the product, the server determines whether the person who has picked up the product has performed stealing or not on the basis of the image data obtained by the imaging device, by using a learning completion model which learns to output information indicating whether the person whose image has been taken in image data has performed stealing or not when the image data has been input.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to an information processing system, an information processing method, and a program.

A system for detecting the occurrence of shoplifting has been proposed. For example, Patent Document 1 proposes a system for detecting whether or not the photographed person has performed a shoplifting suspicious action based on image data photographed by a surveillance camera provided in a store. When the system disclosed in Patent Document 1 detects the occurrence of a shoplifting suspicious act, the system notifies the mobile terminal of the store clerk in the store, and the clerk who receives the notification calls out to a person suspected of shoplifting. By doing so, shoplifting can be prevented in advance.

JP, 2017-76171, A

In the system disclosed in Patent Document 1, in order to prevent shoplifting, the clerk who has received the notification needs to speak to a person suspected of shoplifting, which may increase the burden on the clerk. There is.

The present disclosure has been made in view of such circumstances, and an object of the present disclosure is to provide an information processing system or the like that can suppress the occurrence of shoplifting without increasing the burden on a clerk or the like. To provide.

In an information processing system according to an aspect of the present disclosure, a plurality of image capturing devices distributed in a store and a person imaged in the image data based on the image data acquired by the plurality of image capturing devices are A detection unit that detects whether or not the product displayed in the store is picked up, and information indicating whether or not the person imaged in the image data performed the theft when the image data was input. Using a learned model that has been learned to output, based on image data acquired by the plurality of imaging devices, a theft determination unit that determines whether or not a theft has been performed, and the theft determination unit When it is detected that the person in the image data picks up the product, it is determined whether or not the person who picked up the product performed the theft.

According to the present disclosure, shoplifting can be prevented from occurring without increasing the burden on the clerk or the like.

It is a schematic diagram which shows the structural example of an information processing system. It is a schematic diagram which shows the example of arrangement|positioning of the camera installed in the store, a warning device, and a display device. It is a block diagram which shows the structural example of a server. It is a schematic diagram which shows the structural example of a shoplifting discrimination model. It is a schematic diagram which shows the structural example of DB memorize|stored in a server. It is a block diagram which shows the function implement|achieved by the control part of a server. It is a schematic diagram for demonstrating the process which a server performs. It is a schematic diagram for demonstrating the process which a server performs. It is a schematic diagram which shows an example of a warning screen. It is a flowchart which shows the procedure of the shoplifting act monitoring process by an information processing system. 9 is a block diagram showing functions implemented by a control unit of a server according to the second embodiment. FIG. FIG. 9 is a schematic diagram for explaining a process performed by the server of the second embodiment. 9 is a schematic diagram showing a configuration example of a shoplifting discrimination model of Embodiment 2. FIG. 9 is a schematic diagram showing a configuration example of a shoplifting discrimination model of Embodiment 2. FIG. 11 is a flowchart showing a procedure of shoplifting monitoring processing by the information processing system according to the second embodiment. 9 is a flowchart showing the procedure of shoplifting monitoring processing by the information processing system according to the third embodiment. FIG. 13 is a schematic diagram showing an arrangement example of each device of the information processing system of the fourth exemplary embodiment. 16 is a flowchart showing the procedure of shoplifting monitoring processing by the information processing system according to the fourth embodiment. 16 is a flowchart illustrating a procedure of shoplifting monitoring processing by the information processing system according to the fifth embodiment. It is a schematic diagram which shows the example of the captured image for learning of a violent behavior discrimination model.

Hereinafter, the information processing system, the information processing method, and the program of the present disclosure will be described in detail based on the drawings showing the embodiments.

(Embodiment 1)
An embodiment applied to an information processing system that suppresses the occurrence of shoplifting (theft) in shops and stores will be described. FIG. 1 is a schematic diagram showing a configuration example of an information processing system. The information processing system 100 according to the present embodiment is provided, for example, in a store, and includes a server 10, a plurality of cameras 20, a plurality of warning devices 30, and two display devices (first display device 41 and second display device 42). including. Each of these devices is connected to, for example, a LAN (Local Area Network) provided in the store, and transmits and receives information via the LAN. Note that each of these devices may be connected to a network such as the Internet and may transmit and receive information via the network.

FIG. 2 is a schematic diagram showing an arrangement example of the camera 20, the warning device 30, and the display devices 41 and 42 installed in the store. The camera 20 is an image pickup apparatus having a lens, an image pickup element, and the like, and photoelectrically converts light incident through the lens by the image pickup element to obtain image data. The camera 20 sequentially transmits the image data (photographed image) acquired by photographing to the server 10. The camera 20 is installed, for example, on the upper part or the ceiling of the display shelf so that a person in front of the display shelf displaying the product picks up the product. In the example shown in FIG. 2, one camera 20 is provided at each end of each display shelf, but it is sufficient if the cameras 20 are dispersed and arranged so that the entire area of the store can be photographed. It is not limited to the example shown in.

Each of the warning devices 30 includes, for example, one or more of a display device, a light emitting device, and a voice output device, and gives a warning (attention) by the display device, the light emitting device, or the voice output device. The display device is, for example, a liquid crystal display, an organic EL display, or the like, and when receiving display data from the server 10, issues a warning by displaying the received display data. The light emitting device is, for example, a lamp, a rotating light, or the like, and when receiving a lighting or blinking execution instruction from the server 10, the light emitting device lights or blinks to issue a warning. The voice output device is, for example, a speaker or a buzzer, and when receiving the voice data for output from the server 10, issues a voice warning by outputting the received voice data. The voice output device may be configured to issue a warning by outputting a predetermined sound or voice when receiving a voice output execution instruction from the server 10. Therefore, the warning device 30 gives a warning by displaying the warning data on the display device, turning on or blinking the light emitting device, and outputting the warning sound by the voice output device. When the warning device 30 is composed of a plurality of devices of a display device, a light emitting device, and a voice output device, the warning device 30 in which the plurality of devices are integrated may be used, or may be configured separately. It is also possible to collectively use a plurality of the above-mentioned devices as the warning device 30.

The warning device 30 is installed at an appropriate position so as to issue a warning to a person in front of the display shelf. For example, when the warning device 30 is a display device or a light emitting device, the warning device 30 is installed at the position of the line of sight of a person standing in front of the display shelf so that the display screen of the display device and the lighting or blinking state of the light emitting device can be grasped. Further, when the warning device 30 is a voice output device, it is installed, for example, on the upper part or the ceiling of the display rack so that the voice from the voice output device reaches a person in front of the display rack. In the example shown in FIG. 2, one warning device 30 is provided at the center of each display shelf, but it is sufficient if each warning device 30 is distributed and arranged so as to warn people in the store. 2, but is not limited to the example shown in FIG. Further, each of the warning devices 30 may be installed by selecting one or more of the display device, the light emitting device, and the audio output device according to the installation position in the store. That is, it is possible to configure such that different devices issue warnings depending on the installation position.

The first display device 41 and the second display device 42 are, for example, liquid crystal displays, organic EL displays, etc., and receive the display data transmitted from the server 10 and display the received display data. Therefore, when the first display device 41 and the second display device 42 receive and display warning data from the server 10, they also function as a warning device. The first display device 41 is installed in, for example, a cashier counter in a store or in the vicinity of the cashier counter, and is provided so that a person lined up at the cashier counter or a person lined up at the head of the cashier counter can be visually recognized. The second display device 42 is installed, for example, in the vicinity of an entrance/exit (exit), and is provided so that a person trying to exit the store from the entrance can visually recognize it.

Although not shown in FIG. 2, the server 10 may be provided at any place in the store, and when the server 10 is connected to other devices 20, 30, 41, 42 via a network. , May not be provided in the store. The server 10 is an information processing device such as a server computer or a personal computer, and may be provided with a plurality of units, may be realized with a plurality of virtual machines provided in one server, or may be a cloud server. It may be realized by using. The server 10 accumulates the image data sequentially transmitted from the cameras 20, processes to detect the occurrence of shoplifting based on the image data, and outputs warning information to the warning device 30 when detecting the occurrence of shoplifting. Various information processing such as output processing is performed.

FIG. 3 is a block diagram showing a configuration example of the server 10. The server 10 includes a control unit 11, a storage unit 12, a communication unit 13, a display unit 14, an input unit 15, a reading unit 16 and the like, and these units are mutually connected via a bus. The control unit 11 includes one or more processors such as a CPU (Central Processing Unit), an MPU (Micro-Processing Unit) or a GPU (Graphics Processing Unit). The control unit 11 appropriately executes the control program 12P stored in the storage unit 12 to perform various types of information processing, control processing, and the like that the server 10 should perform.

The storage unit 12 includes a RAM (Random Access Memory), a flash memory, a hard disk, an SSD (Solid State Drive), and the like. The storage unit 12 stores in advance the control program 12P executed by the control unit 11 and various data necessary for executing the control program 12P. The storage unit 12 also temporarily stores data and the like generated when the control unit 11 executes the control program 12P. Further, the storage unit 12 stores a shoplifting discrimination model 121, which is a neural network constructed by machine learning processing, for example. The shoplifting discrimination model 121 uses image data obtained by shooting a person (captured image) and teacher data including information indicating whether or not the person in the shot image has shoplifted It is a learned model that has been learned so as to output information indicating whether a person photographed in the image data has performed shoplifting (theft) when input. The learned model performs a predetermined calculation on the input value and outputs the calculation result, and the storage unit 12 stores the data such as the coefficient of the function defining the calculation and the threshold value in the shoplifting discrimination model. It is stored as 121. Further, the storage unit 12 stores a camera DB (database) 12a and a warning device DB 12b. The camera DB 12a and the warning device DB 12b may be stored in an external storage device connected to the server 10, or may be stored in a storage device that can communicate with the server 10 via a network.

The communication unit 13 is an interface for connecting to a network by wire communication or wireless communication, and transmits/receives information to/from an external device via the network. The display unit 14 is a liquid crystal display, an organic EL display, or the like, and displays various kinds of information according to an instruction from the control unit 11. The input unit 15 includes a mouse, a keyboard, and the like, receives an operation input by the user, and sends a control signal corresponding to the operation content to the control unit 11. The display unit 14 and the input unit 15 may be a touch panel integrally configured.

The reading unit 16 reads information stored in a portable storage medium 1a including a CD (Compact Disc)-ROM or a DVD (Digital Versatile Disc)-ROM. The control program and data stored in advance in the storage unit 12 may be read by the control unit 11 from the portable storage medium 1 a via the reading unit 16 and stored in the storage unit 12. The control program and data stored in advance in the storage unit 12 may be downloaded from the external device via the network via the communication unit 13 by the control unit 11 and stored in the storage unit 12. Furthermore, the control unit 11 may read the control program and data from the semiconductor memory 1b.

FIG. 4 is a schematic diagram showing a configuration example of the shoplifting discrimination model 121. The shoplifting discrimination model 121 is a deep learning model learned using a deep learning algorithm, and is configured by a convolutional neural network (CNN: Convolution Neural Network) model as shown in FIG. 4, for example. The shoplifting discrimination model 121 may be configured by a recurrent neural network (RNN) model in addition to the CNN model. Further, the shoplifting discrimination model 121 is not limited to the multilayer neural network (deep learning) as shown in FIG. 4, and may be configured by using another machine learning algorithm.

The shoplifting discrimination model 121 shown in FIG. 4 includes an input layer, an intermediate layer, and an output layer. The intermediate layer includes a convolutional layer, a pooling layer and a fully bonded layer. In the shoplifting discrimination model 121 of this embodiment, image data is input via the input layer. The image data input through each node of the input layer is input to the intermediate layer, and in the intermediate layer, the image feature amount is extracted by the filtering process in the convolutional layer to generate the feature map, which is compressed in the pooling layer. The amount of information is reduced. A plurality of convolutional layers and pooling layers are repeatedly provided, and the feature maps generated by the plurality of convolutional layers and pooling layers are input to the fully connected layer. The fully connected layers are provided in a plurality of layers (two layers in FIG. 4). Based on the input feature map, the output values of the nodes in each layer are calculated using various functions and thresholds, and the calculated output values Are sequentially input to the nodes of the next layer. The fully connected layer sequentially inputs the output values of the nodes in each layer to the nodes in the subsequent layer, and finally gives the respective output values to each node in the output layer. The number of layers of each of the convolutional layer, the pooling layer, and the total coupling layer is not limited to the example shown in FIG. In the shoplifting discrimination model 121 of the present embodiment, the output layer has two nodes. For example, the node 0 indicates the probability that the person photographed in the input image data should discriminate that the shoplifting action is performed. Then, the node 1 outputs the probability of determining that the shoplifting action is not performed (the normal action is performed). The output value of each node of the output layer is, for example, a value of 0 to 1.0, and the total of the probabilities output from the two nodes is 1.0 (100%). In the shoplifting discrimination model 121 of the present embodiment, the situation in which it is determined that the shoplifting action is not performed includes, for example, an action of putting a product picked up by a person in the captured image into a shopping cart. Further, the situation in which it is determined that a shoplifting act is performed includes an act of opening the product or opening a beverage bottle, in addition to an act of putting the product in a bag or a pocket of clothes.

The shoplifting discrimination model 121 includes, for example, image data (photographed image) obtained by photographing a person who puts a product picked up from a product shelf in his or her bag or clothes pocket, and Learning is performed using teacher data in which information (correct answer label) indicating that a person has shoplifted is set as one set. Further, the shoplifting discrimination model 121 is, for example, a photographed image obtained by photographing a person who performs an operation of opening a product picked up from a product shelf, a person who opens a beverage bottle of the product, and performs a drinking operation. And learning data using a set of the information (correct answer label) indicating that the person in the photographed image has shoplifted. In the shoplifting discrimination model 121, when the captured image included in the teacher data is input, the output value from the node 0, which outputs the probability that the shoplifting action should be discriminated, approaches 1.0, and the other node 1 Learning is performed so that the output value from is close to 0. With this, the person who has taken the action of putting the product taken from the product shelf into his or her bag or the pocket of clothes, the person who has performed the operation of opening the taken product, and the beverage bottle of the product are opened. When a photographed image of a person who performs a drinking motion is input, the shoplifting discrimination model 121 in which the output value from the node 0 (probability to discriminate that a shoplifting action has been performed) becomes a value close to 1.0 can be generated. ..

Further, the shoplifting discrimination model 121 includes image data (photographed image) obtained by photographing a person who puts a product picked up from a product shelf in a shopping basket prepared in a store You may learn using the teacher data which made one set the information (correct answer label) which shows that the person is not shoplifting. In this case, in the shoplifting discrimination model 121, when the captured image included in the teacher data is input, the output value from the node 1 that outputs the probability that it should be discriminated that the shoplifting action is not performed approaches 1.0. , And learn so that the output value from the other node 0 approaches 0. As a result, when the photographed image of the person who puts the product picked up from the product shelf into the shopping basket is input, the output value from the node 1 (probability to determine that shoplifting is not performed) The shoplifting discrimination model 121 having a value close to 1.0 can be generated. In the learning process, the shoplifting discrimination model 121 optimizes data such as coefficients and thresholds of various functions that define a predetermined calculation performed on an input value. Thereby, the learned shoplifting discrimination model 121 learned based on the image data and learned to output information indicating whether or not the person photographed in the image data has shoplifted. The shoplifting discrimination model 121 may be learned by the server 10, or may be stored in the storage unit 12 of the server 10 after being learned by a different learning device.

FIG. 5 is a schematic diagram showing a configuration example of the DBs 12a to 12b stored in the server 10. 5A shows the camera DB 12a, and FIG. 5B shows the warning device DB 12b. The camera DB 12a stores information on the cameras 20 distributed in the store. The camera DB 12a illustrated in FIG. 5A includes a camera ID column, an installation place column, a captured image column, and the like. The camera ID column stores identification information assigned in advance to each camera 20. The installation location column stores information indicating the installation location of each camera 20. The information on the installation location is, for example, the identification information of the display shelf closest to the installation location of the camera 20 or the coordinate value represented by the coordinate system defined based on an arbitrary position in the store, which can identify the installation location. To use. The captured image sequence stores image data (captured image) captured and acquired by each camera 20. The captured image may be stored in a predetermined area of the storage unit 12 or an external storage device connected to the server 10, in addition to being stored in the camera DB 12a. In this case, the captured image sequence stores information (for example, a file name indicating a storage location of image data) for reading the captured image from a predetermined area of the storage unit 12 or an external storage device. The camera ID and the installation location stored in the camera DB 12a are input by the employee of the store through the input unit 15 when the new camera 20 is installed in the store, and are stored by the control unit 11. The captured image stored in the camera DB 12a is stored by the control unit 11 every time the communication unit 13 obtains from each camera 20. The storage content of the camera DB 12a is not limited to the example shown in FIG. 5A, and various information regarding the camera 20 may be stored. For example, the date on which the camera 20 was installed, the date on which the camera 20 was inspected, the date on which the camera 20 was repaired, etc. may be stored.

The warning device DB 12b stores information on the warning devices 30 distributed in the store. The warning device DB 12b illustrated in FIG. 5B includes a warning device ID column, an installation location column, an installation device column, and the like. The warning device ID column stores identification information pre-assigned to each warning device 30. The installation location column stores information indicating the installation location of each warning device 30. The information on the installation location stored in the warning device DB 12b may be any information as long as it can identify the installation location, but the same information as the installation location stored on the camera DB 12a is used. Preferably. The installed device row stores information indicating the devices installed as the warning devices 30. For example, information indicating whether the installed device is a display device, a light emitting device, or an audio output device is stored. In addition, when the warning device 30 is a device in which a plurality of devices among the display device, the light emitting device, and the sound output device are integrally configured, information indicating each device included in the warning device 30 is stored in the installed device row. May be done. Moreover, when each device configured as a separate body is installed in the same installation place, information indicating each device may be collectively stored in the installation device row. Each information stored in the warning device DB 12b is input by the employee of the store via the input unit 15 when a new warning device 30 is installed in the store, and is stored by the control unit 11. The storage content of the warning device DB 12b is not limited to the example shown in FIG. 5B, and various information regarding the warning device 30 may be stored. For example, the execution history of the warning process by the warning device 30, the date when the warning device 30 was installed, the date when the warning device 30 was inspected, the date when the warning device 30 was repaired, and the like may be stored.

Next, a function realized by the control unit 11 of the server 10 executing the control program 12P will be described. FIG. 6 is a block diagram showing functions realized by the control unit 11 of the server 10, FIGS. 7 to 8 are schematic diagrams for explaining processing performed by the server 10, and FIG. 9 is a schematic diagram showing an example of a warning screen. It is a figure. When executing the control program 12P stored in the storage unit 12, the control unit 11 of the server 10 realizes the functions of the image acquisition unit 101, the shoplifting detection unit 102, the tracking processing unit 103, and the warning processing unit 107. The tracking processing unit 103 also includes a tracking unit 104, an action detection unit 105, and a position detection unit 106. In the present embodiment, each of these functions is realized by the control unit 11 executing the control program 12P, but some of these functions may be realized by a dedicated hardware circuit.

The image acquisition unit 101 sequentially acquires the image data obtained by the cameras 20 from the respective cameras 20. The image acquisition unit 101 sequentially stores the image data (captured image) acquired from each camera 20 in the camera DB 12a and sends it to the shoplifting detection unit 102. The shoplifting detection unit (theft determination unit) 102 uses the shoplifting discrimination model 121, and based on the captured image (image data) acquired by each camera 20, the person photographed in the captured image performs shoplifting action. It detects whether or not. Specifically, the shoplifting detection unit 102 inputs the captured image of each camera 20 acquired by the image acquisition unit 101 to the shoplifting determination model 121, and acquires the output value from the shoplifting determination model 121. As the output value of the shoplifting discrimination model 121, the probability that it should be discriminated that the person in the input captured image performed the shoplifting action is output from the node 0 of the output layer, and the node 1 of the output layer performs the shoplifting action. The probability that should be discriminated as no (performing normal actions) is output.

The shoplifting detection unit 102 determines whether a shoplifting action has occurred, based on the output value from the shoplifting discrimination model 121. For example, the shoplifting detection unit 102 performs shoplifting when the output value from the node 0 in the output layer of the shoplifting discrimination model 121 (probability of determining that a shoplifting action has been performed) is a predetermined value (for example, 0.8) or more. Determine that an action has occurred. The shoplifting detection unit 102 determines that the output value from the node 0 in the output layer of the shoplifting discrimination model 121 exceeds the output value from the node 1 in the output layer (probability that it should be determined that shoplifting is not performed). Alternatively, it may be determined that a shoplifting act has occurred when the number exceeds a predetermined value. The shoplifting detection units 102 may be provided as many as the number of cameras 20 in the store, and each shoplifting detection unit 102 may perform shoplifting detection processing based on an image captured by the corresponding camera 20. Further, one shoplifting detection unit 102 may be configured to sequentially switch shoplifting detection processing based on images captured by a plurality of cameras 20.

Further, in the shoplifting discrimination model 121, instead of the node 0 of the output layer, a node that outputs the probability that a person in the input photographed image should perform the act of putting the product in the bag, and a person in the photographed image May output the probability that should be determined to have carried out the act of putting the product in the pocket of the clothes. With such a configuration, the shoplifting detection unit 102 can determine whether the person in the input captured image has put the product in the bag or put it in the pocket of the clothes. The shoplifting detection unit 102 may determine that the shoplifting action has occurred when the output value from either of the two nodes is equal to or greater than a predetermined value (for example, 0.8). In addition, the shoplifting discrimination model 121, instead of the node 1 in the output layer, outputs a probability that should be discriminated that the person in the input photographed image does not pick up the product, and the person in the photographed image. It may be configured to have a node that outputs a probability that should be determined that an act of putting a product into a shopping cart of a store has been performed. With such a configuration, the shoplifting detection unit 102 can determine whether the person in the input photographed image is in a state where he or she does not pick up the product or whether the product has been put in the shopping cart. The shoplifting detection unit 102 may determine that the shoplifting action has not occurred when the output value from either of these two nodes is equal to or greater than a predetermined value (for example, 0.5). ..

When the shoplifting detection unit 102 detects the occurrence of the shoplifting action, the tracking processing unit 103 performs a process of tracking the movement of the person who performed the shoplifting action. The person who is the target of the tracking process is called a tracking target person. The tracking unit 104 tracks the movement of the tracking target person based on the captured image of each camera 20. FIG. 7 shows an example of a photographed image of a tracking target person. In the captured image, the tracking unit 104 sets, for example, a rectangular area including the person to be tracked as the tracking area R, and in the tracking area R, specifies the feature amount such as the color of clothes or belongings of the person to be tracked. Then, the tracking unit 104 determines whether or not the tracking region R is included in the captured images sequentially acquired by the cameras 20 in time series, based on the identified characteristic amount in the tracking region R, and the tracking region R is included. The movement of the tracking region R is tracked based on the captured image taken. If the person to be tracked is within the shooting range of the plurality of cameras 20, tracking processing is performed based on the images shot by the plurality of cameras 20. As a result, it is possible to prevent the tracking target person from failing to lose track of the tracking target person.

The behavior detection unit (tracking determination unit) 105 displays, on the display shelf, a product put in a bag or clothes pocket of the person to be tracked by the person to be tracked based on the captured image of the person to be tracked taken by each camera 20. It is detected whether or not a returning action (action) is performed. For example, the behavior detection unit 105 detects whether or not the product has been returned to the product shelf based on the tracking region R that the tracking unit 104 sequentially tracks. For example, when the tracking target person returns the product to the display shelf by performing the operation shown in FIGS. 8A, 8B, and 8C, the behavior detection unit 105 detects the tracking target person's movement based on the tracking region R. Detects that the product has been returned to the display shelf.

Further, for example, in the shoplifting discrimination model 121, in addition to the probability of discriminating whether or not a person in the input captured image has shoplifted, the person in the input captured image may be, for example, his or her own pocket or pocket of clothes. It is also possible to output the determination probability of whether or not the behavior of taking out a product from the product and returning it to the display shelf is performed. By using the shoplifting discrimination model 121 as described above, the behavior detection unit 105 can detect whether or not the tracking target person has performed the action of returning the merchandise to the display shelf based on the captured image of the tracking target person. In addition to the shoplifting discrimination model 121, the action detection unit 105, when a photographed image is input, has the person in the photographed image take out the product from his/her bag or clothes pocket and return it to the display shelf? Using a learned model that has been learned to output information indicating whether or not, based on the captured image of the tracking target person, it is detected whether the tracking target person has performed the action of returning the product to the display shelf. Good. When detecting using the shoplifting discrimination model 121 or another learned model, the behavior detection unit 105 captures the tracking target person by each camera 20 with respect to the shoplifting discrimination model 121 or another learned model. An image is input and an output value from the shoplifting discrimination model 121 or another learned model is acquired. The behavior detection unit 105 detects, based on the output value from the shoplifting discrimination model 121 or another learned model, whether or not the tracking target person has performed the behavior of returning the merchandise to the display rack.

The position detection unit 106 detects the position of the tracking target person and acquires position information indicating the position. The position information is, for example, the identification information of the display rack closest to the person to be tracked, the camera 20, the warning device 30, or the coordinate value represented by the coordinate system defined based on an arbitrary position in the store. Use information that can identify the position. When the identification information of the camera 20 is used as the position information, the position detection unit 106 identifies, for example, the camera 20 that has the largest area of the tracking target person in the captured image among the cameras 20 that have captured the tracking target person. Then, the camera ID of this camera 20 is acquired as the position information of the tracking target person. Further, the position detection unit (detection unit) 106 detects, for example, whether or not the tracking target person is within a predetermined range from the cashier counter, is within a predetermined range from the doorway, and the like, and detects a predetermined range from the cashier counter or the doorway. When inside, the identification information of the cashier counter or the entrance/exit may be acquired as the position of the tracking target person.

The tracking processing unit 103 can track the movement of the tracking target person who has performed the shoplifting action detected by the shoplifting detection unit 102 in the store by the processing of each unit described above, and can grasp the position of the tracking target person in real time. In addition, the tracking processing unit 103 can grasp whether or not the tracking target person has returned the product put in the bag or the pocket of the clothes to the display shelf by the shoplifting act. When it is detected that the tracking target person returns the product to the display shelf, the tracking processing unit 103 ends the tracking processing for the tracking target person. When the tracking processing unit 103 ends the tracking processing for the tracking target person, the tracking processing unit 103 may output a message indicating that the shoplifting action is stopped by the warning device 30 closest to the tracking target person. For example, the tracking processing unit 103 may transmit a display instruction of a message such as “Thank you for returning the product” to the warning device 30 of the display device and display it on the warning device 30. Further, the tracking processing unit 103 may transmit an output instruction of a voice message such as “Thank you for returning the product” to the warning device 30 of the voice output device to cause the warning device 30 to output a voice. ..

The warning processing unit 107 detects that the shoplifting detection unit 102 has performed a shoplifting action, and issues a warning to the tracking target person who is being tracked by the tracking processing unit 103. The warning processing unit 107 is based on the installation location of each of the warning devices 30 registered in the warning device DB 12b and the position of the tracking target person detected by the position detection unit 106, and is closest to the tracking target person. The warning device 30 is identified. The warning device 30 here also includes a first display device 41 and a second display device 42. The warning processing unit 107 transmits a warning processing execution instruction (warning instruction, warning information) to cause the specified warning device 30 to perform the warning processing. Specifically, if the warning device 30 is a display device, the warning processing unit 107 transmits the warning data for display and a display instruction (warning instruction, warning information) for the warning data to the warning device 30. When receiving the display instruction from the server 10, the warning device 30, which is a display device, displays a warning screen based on the received warning data.

FIG. 9A shows an example of a warning screen. The warning screen has a warning message such as "Shoplifting is a crime" and a message such as "Please return the product" to stop the shoplifting action. Is displayed. Further, as shown in FIG. 9A, the warning screen may display a captured image when the shoplifting detection unit 102 detects the occurrence of shoplifting. Note that the warning screen displays the captured image by superimposing the mosaiced region P on the region of the face of the tracking target person in the captured image so that the individual cannot be identified, for example, as shown in FIG. 9B. Alternatively, as shown in FIG. 9C, the mosaiced area P may be displayed in an overlapping manner on the entire area of the tracking target person in the captured image. Further, as shown in FIG. 9D, the area P of the character image may be displayed in a superimposed manner on the area of the face of the tracking target person in the captured image. The warning screen may display an image of a product taken by a shoplifting action, for example, a product that the tracking target person put in his/her bag or clothes pocket. In this case, the server 10 stores the image of the package of each product in the storage unit 12 in advance, specifies the product taken by the shoplifting action based on the photographed image, and stores the image of the specified product from the storage unit 12. It is read and displayed on the warning device 30.

Further, if the warning device 30 is a light emitting device, the warning processing unit 107 transmits a lighting or blinking execution instruction (warning instruction, warning information) to the warning device 30. In this case, the warning device 30, which is a light emitting device, receives a lighting or blinking execution instruction from the server 10 and lights or blinks based on the received execution instruction. Further, if the warning device 30 is a voice output device, the warning processing unit 107 transmits warning data for voice output and a voice output instruction (warning instruction, warning information) of the warning data to the warning device 30. In this case, the warning device 30, which is a voice output device, receives a voice output instruction from the server 10 and outputs a voice based on the received warning data. If the warning device 30 is a buzzer or a voice output device that outputs a predetermined voice, the warning processing unit 107 transmits a voice output instruction (warning instruction, warning information) to the warning device 30, and in this case, a warning is issued. The device 30 outputs a predetermined sound based on the output instruction received from the server 10. The warning device 30, which is a voice output device, for example, "Is the product in the bag?", "Please pay at the cashier", "Thank you for coming to the store. Please output a voice message such as "Shoplifting AI is in operation now."

Next, a shoplifting monitoring process performed by each device of the information processing system 100 will be described. FIG. 10 is a flowchart showing a procedure of shoplifting monitoring processing by the information processing system 100. In FIG. 10, the processing performed by the camera 20 is shown on the left side, the processing performed by the server 10 is shown in the center, and the processing performed by the warning device 30 is shown on the right side. In the information processing system 100 that monitors shoplifting in a store, the camera 20 acquires image data at a predetermined timing (S11), and sequentially transmits the acquired image data to the server 10 (S12). The camera 20 acquires image data of 20 to 30 frames per second, for example, but the number of frames is not limited.

The server 10 sequentially receives the image data transmitted from the camera 20 (S13). The server 10 stores the received image data in the camera DB 12a and performs a detection process for detecting the occurrence of shoplifting based on the received image data (S14). In the detection process, the server 10 inputs the received image data (captured image) to the shoplifting discrimination model 121 and acquires the output value from the shoplifting discrimination model 121. Then, the server 10 determines whether a shoplifting act has occurred, based on the output value from the shoplifting discrimination model 121, and determines whether a shoplifting action has been detected (S15). For example, when the output value from the node 0 of the output layer of the shoplifting discrimination model 121 (probability of determining that a shoplifting action has occurred) is a predetermined value (for example, 0.8) or more, or from the node 1 of the output layer If the output value (probability of determining that the shoplifting action has not occurred) exceeds a predetermined value, the server 10 detects that the shoplifting action has occurred.

When determining that the occurrence of shoplifting has not been detected (S15: NO), the server 10 returns to the process of step S13 and repeats the processes of steps S13 to S15. When determining that the occurrence of shoplifting has been detected (S15: YES), the server 10 starts the tracking process of the person (tracking target) who has carried out the detected shoplifting (S16). Specifically, the server 10 determines whether or not the captured image of each person to be tracked (tracking region R) is included in the captured image of each camera 20, and the captured image including the tracking region R (this captured image is The movement of the tracking region R is tracked based on the captured position of the camera 20).

The server 10 determines whether or not the person to be tracked performs a shoplifting action, for example, to return the product put in his/her bag to the display shelf based on the captured image of the person to be tracked taken by each camera 20. A judgment is made (S17). For example, the server 10 detects the movement of the tracking target person photographed in the tracking area R based on the tracking area R sequentially tracked, and whether the tracking target person returns the product acquired by the shoplifting action to the product shelf. To detect. The server 10 operates the shoplifting discrimination model 121 or another learned model (when the photographed image is input, the person in the photographed image takes out the product from his/her bag or clothes pocket and returns it to the display shelf. It is also possible to determine whether or not the tracking target person has returned the product to the display shelf by using a learned model that has been learned to output information indicating whether or not the product has been performed. When the tracking target person determines that the merchandise has been returned to the display shelf (S17: YES), the server 10 ends the tracking process started in step S16 (S18), and returns to the process of step S13. Thereby, for example, when the tracking target person returns the product put in his/her bag or the like to the display shelf by shoplifting, the tracking processing of the tracking target person is ended, and the warning processing by the warning device 30 is ended.

When it is determined that the tracking target person has not returned the merchandise to the display shelf (S17: NO), the server 10 performs a process of detecting the current position of the tracking target person (S19). Here, the server 10 specifies, for example, the camera 20 that has the largest shooting area (tracking area R) of the tracking target person in the captured image among the cameras 20 that have captured the tracking target person. The ID is acquired as information indicating the current position of the tracking target person. When the server 10 detects that the tracking target person is within a predetermined range from the cashier counter or the entrance/exit, the server 10 may acquire the identification information of the cashier counter or the entrance/exit as information indicating the current position of the tracking target person.

The server 10 identifies the warning device to be subjected to the warning process based on the detected current position of the tracking target person and the installation location of each warning device 30 registered in the warning device DB 12b (S20). For example, the server 10 identifies the warning device 30 installed at the position closest to the current position of the tracking target person. The warning device 30 includes a first display device 41 and a second display device 42. Then, the server 10 outputs a warning process execution instruction to the specified warning device 30 (S21). For example, the server 10 transmits a display instruction of display warning data to the warning device 30 if the warning device 30 is a display device, and sends a lighting or blinking execution instruction to the warning device 30 if the warning device 30 is a light emitting device. Send to 30. If the warning device 30 is a voice output device, the server 10 transmits a voice output instruction of warning data for voice output or a predetermined voice output instruction to the warning device 30. It should be noted that the server 10 may cause each of the warning devices 30 to perform different warning processing depending on the specified position of the warning device 30. For example, even if the two warning devices 30 are both display devices, the server 10 may transmit different warning data for display to each warning device 30 depending on the installation position of the warning device 30. As a result, it is possible to issue a warning according to the installation position of the warning device 30.

The warning device 30 performs a warning process according to the execution instruction acquired from the server 10 (S22). If the warning device 30 is a display device, a warning screen is displayed based on the warning data for display received from the server 10, and if the warning device 30 is a light emitting device, based on the execution instruction received from the server 10. Lights up or flashes. Further, when the warning device 30 is a voice output device, when the warning data for voice output is received from the server 10, a voice output is performed based on the received warning data, and a predetermined voice output instruction is received from the server 10. In this case, a predetermined sound is output based on the received output instruction.

After the processing of step S21, the server 10 returns to the processing of step S17, and repeats the processing of steps S19 to S21 until the tracking target person returns the merchandise to the display shelf. By the processing described above, when the occurrence of shoplifting is detected, the warning processing can be performed by the warning device 30 near the person (tracking target person) who is likely to have shoplifted. Therefore, immediately after the shoplifting action is performed, or after the shoplifting action is performed, it is possible to periodically warn and alert the tracking target person. For example, it is possible to warn the person to be tracked that a shoplifting action has been detected, and also to notify a message that discourages the execution of the shoplifting action. As a result, it can be expected that the pursuit target person who has carried out the shoplifting action is urged to return the product to the display shelf and stop the shoplifting action. Further, when the person to be traced returns the product put in his/her bag or the like to the display shelf by shoplifting, the tracking processing for this person to be tracked is ended, and the warning processing by the warning device 30 is also ended. Therefore, it is possible to suppress the occurrence of shoplifting without increasing the burden on employees of the store. When the tracking process for the tracking target person is terminated, the server 10 outputs a message such as "Thank you for returning the product" by the warning device 30 closest to the tracking target person (display, voice output). ) May be done.

In the information processing system 100 of the present embodiment, not only the warning device 30 and the display devices 41 and 42 issue a warning to the tracking target person, but also when the tracking target person is detected, the server 10 causes a police station, a security company, or the like. May be configured to notify. For example, the control unit 11 of the server 10 determines whether or not the tracking processing by the tracking processing unit 103 for the same tracking target person has been continuously performed for a predetermined time (for example, 5 minutes, 10 minutes, etc.), and continues for the predetermined time. When it is done, it may have a configuration to output the notification information to the terminal of a predetermined security company. In addition, the control unit 11 of the server 10 determines whether or not the tracking target person has gone out of the store by the tracking processing by the tracking processing unit 103, and when it goes out of the store, outputs the notification information to the terminal of the police station. You may have a structure. When the report information is acquired from the server 10, the terminal of the security company and the terminal of the police station, for example, turn on or blink a lamp or output a predetermined voice message, so that the guard of the security company or the police of the police station can receive the information. Notify the official that there was a report of shoplifting. By providing such a configuration, for example, when the person to be traced who has shoplifted is staying in the store with a product for a predetermined time, it is possible to notify the security company, and the person to be traced can store the information. If you go out, you can call the police station. A security company or a police station can deal with shoplifting when receiving a report from the server 10. The timing of notifying the security company and the timing of notifying the police station are not limited to the above timing.

In addition to the police station and the security company, the server 10 may notify the head office of the store, the headquarters having jurisdiction over the store, or the like that the tracking target person has been detected. For example, the control unit 11 of the server 10 informs the head office of the store or the terminal of the head office when the same tracking target person continues the tracking process for a predetermined time or when the tracking target person goes out of the store. May be output. Even in this case, the head office of the store or the terminal of the head office notifies the person in charge of the head office or the head office of the fact of the shoplifting action, for example, by turning on or off the lamp or outputting a predetermined voice message. .. By providing such a configuration, the occurrence of shoplifting can be understood even at the head office or the headquarters of the store. It is possible to appropriately change the timing of notification to the security company, the police station, the head office and headquarters of the store. For example, not only the duration of the tracking process for the same tracking target person but also the timing according to the number of times of executing the warning process for the tracking target person during tracking may be notified. That is, if the tracking process is continuously performed for a predetermined time period or if the warning process for the tracking target person is performed for a predetermined number of times, the tracking target person is likely to exit the store without returning the product to the display shelf. It may be configured to report to a station or the like.

In the present embodiment, AI (Artificial Intelligence) can be used to perform a process of detecting whether or not a shoplifting action has occurred based on the image captured by the camera 20 provided in the store. Therefore, it is not necessary for an employee or the like of the store to monitor the image captured by the camera 20. Further, it is very difficult for a person to visually check all the captured images, but since the server 10 detects the shoplifting action based on all the captured images, the captured images can be effectively used.

In the present embodiment, the camera 20 may be provided with a function of detecting whether or not shoplifting has been performed based on the image data captured by the camera 20. For example, the camera 20 is configured to perform a shooting process, perform a shoplifting action detection process based on the acquired image data, and notify the server 10 when a shoplifting action is detected. In this case, the server 10 is configured to perform the tracking process of the tracking target person after being notified by the camera 20 that the shoplifting action has been performed. Even with such a configuration, the same effect as that of the above-described first embodiment can be obtained.

(Embodiment 2)
Performs processing to detect whether a person (customer) in the store picks up the product, and when detecting that the product is picked up, performs processing to detect whether shoplifting has been performed The information processing system 100 will be described. The information processing system 100 of this embodiment is configured by the same devices as the devices of the information processing system 100 of the first embodiment. FIG. 11 is a block diagram showing functions realized by the control unit 11 of the server 10 of the second embodiment, FIG. 12 is a schematic diagram for explaining processing performed by the server 10 of the second embodiment, FIGS. 13 and 14 FIG. 9 is a schematic diagram showing a configuration example of a shoplifting discrimination model 121 according to the second embodiment.

In the information processing system 100 of this embodiment, when the control unit 11 of the server 10 executes the control program 12P stored in the storage unit 12, in addition to the functions shown in FIG. To be realized. Based on the image data (captured image) acquired by the image acquisition unit 101 from each camera 20, the grip detection unit 108 determines whether or not the person photographed in the captured image picks up the product (whether or not the product is grasped). Or) is detected. For example, an image of a package of a product sold in a store is stored in the storage unit 12 in advance, and the grip detection unit 108 stores image data captured by each camera 20 and each product stored in the storage unit 12. It is detected whether or not the person in the photographed image picks up any product based on the image of the package. For example, the grip detection unit 108 detects the shooting area of the product in the sequentially acquired shooting images, and when the shooting area of the product moves, determines that the product is held by the customer. Further, the grip detection unit 108 detects the photographing area of the product and the photographing area of the human hand in the sequentially acquired photographed images, and when the photographing area of the commodity overlaps the photographing area of the human hand, It may be determined that it has been taken by the customer. The process of detecting whether the person in the captured image picks up the product by the grip detection unit 108 is not limited to these.

The shoplifting detection unit 102 according to the present embodiment uses the shoplifting discrimination model 121 to detect a person in a captured image when the grip detection unit 108 detects that the person in the captured image picks up a product. Detects whether shoplifting has been performed. The shoplifting discrimination model 121 of the present embodiment has the same configuration as the shoplifting discrimination model 121 of the first embodiment. In the shoplifting determination model 121 of the first embodiment, the image data (image data) captured by the camera 20 is the input data, but in the shoplifting determination model 121 of the present embodiment, the customer region R1 extracted from the captured image is the input data. Becomes

When the shoplifting detection unit 102 detects that the person in the captured image picks up the product by the grip detection unit 108, as illustrated in FIG. 12, the customer region R1 including the detected person is detected from the captured image. To extract. Then, the shoplifting detection unit 102 inputs the customer region R1 to the shoplifting determination model 121 via the input layer, as shown in FIG. Note that the shoplifting discrimination model 121 of the present embodiment, as shown in, for example, FIGS. 14A and 14B, stores the product region R2 including the product detected by the grip detection unit 108 as being held by the customer in the input layer. The configuration may be such that the shoplifting determination model 121 is input via this. Even with such a configuration, the shoplifting detection unit 102 can detect whether the person who picks up the product has shoplifted.

FIG. 15 is a flowchart showing the procedure of shoplifting monitoring processing by the information processing system 100 according to the second embodiment. FIG. 15 shows the processing performed by the camera 20 on the left side and the processing performed by the server 10 on the right side. The process shown in FIG. 15 is obtained by adding the processes of steps S31 to S32 between steps S13 and S14 in the process shown in FIG. Therefore, description of steps similar to those in FIG. 10 will be omitted. Further, in FIG. 15, the illustration of steps S15 to S22 in FIG. 10 is omitted.

In the information processing system 100 of the present embodiment, when the server 10 receives the image data transmitted from the camera 20 (S13), the server 10 stores the received image data in the camera DB 12a, and based on the received image data. A grip detection process is performed to detect whether a person (customer in the store) photographed in the image data (photographed image) picks up the product (S31). The server 10 determines whether the person in the captured image picks up the product by the grip detection process (S32), and determines that the person in the captured image does not pick up the product (S32: NO), and the process returns to step S13. When the server 10 determines that the person in the captured image picks up the product (S32: YES), the process proceeds to step S14, and a detection process for detecting the occurrence of shoplifting action is performed based on the received image data. Perform (S14). At this time, the server 10 extracts the photographing area (customer area R1) of the person who is detected to have picked up the product from the photographed image, and performs shoplifting detection processing based on the extracted customer area R1. ..

In this embodiment, the same effect as that of the first embodiment can be obtained. Further, in the present embodiment, the photographing region (customer region R) of the person who picks up the product is extracted from the photographed image acquired by the camera 20, and the discrimination processing by the shoplifting discrimination model 121 is performed based on the customer region R. To do. Therefore, since the customer area R1 having a small amount of data is used, it is possible to perform efficient and highly accurate determination processing.

(Embodiment 3)
The information processing system 100 in which the warning device 30 performs different warning processes depending on the installation position will be described. The information processing system 100 of this embodiment is configured by the same devices as the devices of the information processing system 100 of the first embodiment. Since the first display device 41 and the second display device 42 are regarded as one of the warning devices 30, the information of the first display device 41 and the second display device 42 is also stored in the warning device DB 12b. In the information processing system 100 of the present embodiment, when the control unit 11 of the server 10 executes the control program 12P stored in the storage unit 12, the control unit 11 realizes the same functions as those shown in FIG. In this embodiment, the processing performed by the warning processing unit 107 is slightly different from that of the first embodiment, and only different processing will be described.

As with the first embodiment, the warning processing unit 107 according to the present embodiment, for the tracking target person who is being tracked by the tracking processing unit 103, is a warning device 30 installed at a position close to the tracking target person. Warn by. Here, when the warning device 30 close to the person to be tracked is the first display device 41 or the second display device 42, the warning processing unit 107 of the present embodiment is another warning device 30 (a warning device 30 that is a display device). ) And a warning screen different from that are displayed to warn. Specifically, when the warning processing unit 107 specifies the warning device 30 at the installation location closest to the person to be tracked, whether the specified warning device 30 is the first display device 41 or the second display device 42. To judge. When the warning processing unit 107 determines that the specified warning device 30 is not the first display device 41 and the second display device 42, that is, when the tracking target person is not near the cashier counter or the entrance/exit, A warning is issued by the warning device 30 near the person to be tracked. When the warning device 30 here is a display device, a warning message such as "Please pay at the cash register" is displayed. When the warning processing unit 107 determines that the specified warning device 30 is the first display device 41, that is, when the tracking target person is in the vicinity of the cashier counter, the first display device 41 displays, for example, “Shoplifting”. Therefore, please display a warning message such as "Please pay at the cash register" (perform the first warning process). Further, when the warning processing unit 107 determines that the specified warning device 30 is the second display device 42, that is, when the tracking target person is in the vicinity of the entrance/exit, the warning display unit 107 may display the second display device 42, for example, as shown in FIG. 9A or 9B. The warning screen as shown in is displayed (the second warning process is performed). By thus displaying different warning screens on the first display device 41 and the second display device 42, it is possible to perform warning processing with different warning levels. The warning message displayed on the warning device 30 and the display devices 41 and 42, which are display devices, is not limited to the above-mentioned message, and for example, a warning such as "Shoplifting prevention AI is operating" or "Shoplifting is a crime". It may be a message or a warning screen in which a warning message is displayed in a speech bubble spoken by the character. In addition, a paper sticker or a sticker on which such a warning message or warning screen is written may be attached to the cashier counter, the vicinity of the entrance, the wall in the store, or the like.

The traceable person pays some products at the cashier counter when shoplifting in front of a display shelf in the store, when going out of the store from the doorway, and when shoplifting in the store. After going, it is possible that you will go out of the store from the entrance. In the present embodiment, such a tracking target person is first warned in the vicinity of the display shelf, and if the product is not returned to the display shelf in response to the warning, a warning is issued in the vicinity of the cashier counter, and the product is still displayed. If it is not returned to the shelf, a warning can be issued near the doorway. In this way, it is possible to call attention while switching the warning device 30 as the person to be tracked moves. Further, in the present embodiment, by changing the content displayed on the warning screen when the person to be tracked is in the vicinity of the display shelf, in the vicinity of the cashier counter, and in the vicinity of the doorway. , It is possible to give a warning according to the location of the person to be tracked. For example, when the tracked person is near the display shelf, when the tracked person comes to the cashier counter, and when the tracked person comes to the entrance/exit, a warning message with an increased warning level is displayed in order. be able to.

FIG. 16 is a flowchart illustrating a procedure of shoplifting monitoring processing by the information processing system 100 according to the third embodiment. FIG. 16 shows processing performed by the server 10, the first display device 41, the second display device 42, and the warning device 30 in order from the left side. The process shown in FIG. 16 is obtained by adding the processes of steps S41 to S46 between steps S20 and S21 in the process shown in FIG. Therefore, description of steps similar to those in FIG. 10 will be omitted. Further, in FIG. 16, the illustration of steps S11 to S16 and S18 in FIG. 10 is omitted.

In the information processing system 100 of this embodiment, the server 10 identifies the warning device 30 in step S20, and then determines whether the identified warning device 30 is the first display device 41 (S41). When it is determined that the specified warning device 30 is the first display device 41 (S41: YES), the server 10 transmits the first warning data for display to the first display device 41 (S42). The first display device 41 receives the first warning data transmitted from the server 10 and displays a warning screen based on the received first warning data (S43).

When it is determined that the specified warning device 30 is not the first display device 41 (S41: NO), the server 10 determines whether the specified warning device 30 is the second display device 42 (S44). When it is determined that the specified warning device 30 is the second display device 42 (S44: YES), the server 10 transmits the second warning data for display to the second display device 42 (S45). The second display device 42 receives the second warning data transmitted from the server 10 and displays a warning screen based on the received second warning data (S46). When it is determined that the specified warning device 30 is not the second display device 42 (S44: NO), the server 10 shifts the processing to step S21.

After the processing of step S42, S45 or S21, the server 10 returns to the processing of step S17 and repeats the processing of steps S19 to S21, S41 to S42 and S44 to S45 until the person to be traced returns the product to the display shelf. By the processing described above, when the occurrence of shoplifting is detected, the warning device 30 near the person (tracking target) who is likely to have shoplifted can give a warning and the tracking target A warning can be issued by a predetermined warning message when the user comes near the cashier counter or the doorway.

In the present embodiment, for example, immediately after performing the shoplifting action, it is possible to warn and alert the person (tracking target) who has performed the shoplifting action. In addition, when the person to be tracked comes to the cashier counter or to the entrance/exit, by displaying a predetermined warning screen on the first display device 41 and the second display device 42, respectively, the location and the situation of the person to be tracked are displayed. A corresponding warning can be given. Further, in each of the warning devices 30 distributed in the store, for example, as the elapsed time after the shoplifting action becomes longer, the warning process having a higher warning level may be performed. For example, immediately after a shoplifting action, a warning by lighting or blinking light is given by a warning device 30 which is a light emitting device, and when the product is not returned to the display shelf thereafter, the warning device 30 which is a display device is displayed. If a warning screen is displayed to give a warning and the product is still not returned to the display rack, the warning message may be output as a sound from the warning device 30 which is a voice output device to give a warning.

In the present embodiment, the warning device 30 of the light emitting device is provided near the first display device 41 provided on the cashier counter, and when the person to be tracked comes to the cashier counter, the warning device 30 turns on or off the light. By blinking, an employee at the cashier counter may be notified of the existence of the person to be tracked. Further, a warning device 30 such as a light emitting device or a voice output device is provided near the second display device 42 provided at the entrance and exit, and when the tracking target person tries to go out of the store, the warning process by the warning device 30 is performed. Alternatively, the employee of the store may be notified that the person to be tracked is about to leave the store. Thereby, the employee of the store can grasp the existence of the tracking target person, and can secure the identity of the tracking target person when the tracking target person leaves the store.

In the present embodiment, whether or not the tracking target person has come near the cashier counter or the entrance/exit is determined based on, for example, an image captured by a camera that photographs a person who is lined up at the cashier counter or a person who is about to exit the store from the entrance/exit. You may make it detect by. For example, the tracking processing unit 103 of the server 10 acquires a captured image (shooting region R) of the tracking target person. Then, a photographed image including the photographing region R of the person to be tracked is searched from the photographed images of the person lined up at the cashier counter or the person trying to go out of the store from the entrance. When the photographed image including the photographing region R of the person to be tracked can be retrieved, it may be detected that the person to be tracked is lined up at the cashier counter or is about to go out of the store from the entrance.

In addition, in the information processing system 100 of the present embodiment, the server 10 performs warning processing by the warning device 30 and the display devices 41 and 42, and notifies a pre-registered smartphone or mobile terminal of the existence of the tracking target person. May be. For example, when the occurrence of shoplifting is detected, or when the tracking target person comes near the cashier counter or the entrance/exit, the presence of the tracking target person may be notified to the smartphone or the mobile terminal managed by the store manager. This allows the store manager or employee of the store to monitor the behavior of the tracked person. Further, the server 10 may display a photographed image in which the occurrence of shoplifting has been detected, for example, on a display device provided in the employee waiting room in the store to notify the employee or the like in the employee waiting room. ..

(Embodiment 4)
An information processing system 100 that monitors not only shoplifting activities but also internal shoplifting activities (theft activities) by store employees (store staff) will be described. FIG. 17 is a schematic diagram illustrating an arrangement example of each device of the information processing system 100 according to the fourth embodiment. The information processing system 100 according to the present embodiment includes, in addition to the devices of the information processing system 100 according to the first embodiment illustrated in FIG. 2, a camera 50 and a warning device 30 provided in a backyard in a store, a store manager and an employee. A waiting room display device 43 provided in the waiting room is included. The camera 50 and the warning device 30 provided in the backyard and the waiting room display device 43 provided in the waiting room are both connected to a network such as a LAN or the Internet provided in the store. The backyard is, for example, a warehouse, a room, a refrigerator, or the like that stores products that are not displayed in the store, and the waiting room is, for example, a room where employees take a break.

The camera 50 provided in the backyard is installed, for example, on the ceiling so that a person in the backyard can take a picture of a product being picked up. In the example shown in FIG. 17, two cameras 50 are provided in the backyard, but the number of cameras 50 is not limited to two as long as the entire area of the backyard can be captured. Further, in the example shown in FIG. 17, one warning device 30 is provided in the backyard, but two or more warning devices 30 may be provided. In addition, a display shelf installed adjacent to the backyard is a walk-in refrigerator that has a door in the front for customers to take out products and a product slot for store employees to replenish products in the rear. In some cases, a camera 50 may be provided at the rear product insertion slot. In this case, the camera 50 is installed so as to be able to take a picture of the employee near the product slot.

In the information processing system 100 of the present embodiment, when the control unit 11 of the server 10 executes the control program 12P stored in the storage unit 12, the control unit 11 realizes the same functions as those shown in FIG. In this embodiment, the processing performed by the warning processing unit 107 is slightly different from that of the first embodiment, and only different processing will be described.

As with the first embodiment, the warning processing unit 107 according to the present embodiment, for the tracking target person who is being tracked by the tracking processing unit 103, is a warning device 30 installed at a position close to the tracking target person. Warn by. Here, the warning processing unit 107 of the present embodiment determines whether the tracking target person is an employee of the store. For example, a photographed image of the face of each employee is registered in advance in the storage unit 12 of the server 10, and the warning processing unit (specification unit) 107 includes the photographing region R of the tracking target person tracked by the tracking processing unit 103 and Based on the photographed image of the employee's face, the tracked person is identified as one of the employees. The warning processing unit 107 of the present embodiment performs the same processing as that of the first embodiment when it is determined that the tracking target person is not an employee of the store, that is, when a shoplifting act is performed. On the other hand, when the warning processing unit 107 of the present embodiment determines that the tracking target person is an employee of the store, that is, when an internal deduction action is performed, the tracking target person (of the store While performing warning processing for employees, a notification screen is displayed on the waiting room display device 43. As a result, it is possible to warn the employee who has performed the internal deduction and to notify the store manager in the waiting room of the occurrence of the internal deduction. In addition, when the occurrence of shoplifting is detected based on the image taken by the camera 20 provided in the backyard, it is determined that the shoplifting has occurred, and the tracking target person You may specify who you are. The determination as to whether or not the tracking target person is an employee of the store may be performed by the shoplifting detection unit 102 when the shoplifting detection unit 102 detects the occurrence of the shoplifting action.

FIG. 18 is a flowchart showing the procedure of shoplifting monitoring processing by the information processing system 100 according to the fourth embodiment. In FIG. 18, the process performed by the server 10 is shown on the left side, the process performed by the waiting room display device 43 is shown in the center, and the process performed by the warning device 30 is shown on the right side. The process shown in FIG. 18 is obtained by adding the processes of steps S51 to S54 after the process of step S21 in the process shown in FIG. Therefore, description of steps similar to those in FIG. 10 will be omitted. Further, in FIG. 18, steps S11 to S16 and S18 in FIG. 10 are omitted.

In the information processing system 100 of the present embodiment, the server 10 outputs the warning process execution instruction to the warning device 30 at the position closest to the current position of the tracking target person in step S21, and then the tracking target person stores To determine if you are an employee of. That is, the server 10 determines whether the shoplifting act detected in step S14 is an internal pulling act by the employee (S51). For example, the server 10 uses the photographed image of the face of each employee registered in the storage unit 12 in advance and the photographed image (photographing region R) obtained by photographing the person to be traced by the cameras 20 and 50 to trace the person to be traced. Determine whether you are an employee. When the server 10 determines that the tracking target person is an employee of the store, that is, when it determines that the internal deduction action has been performed (S51: YES), the tracking target person who performed the internal deduction action is the employee. It is specified from the inside (S52).

The server 10 generates notification data for notifying the specified employee of the tracking target person, and transmits the notification data to the waiting room display device 43 (S53). The waiting room display device 43 receives the notification data transmitted from the server 10 and displays a notification screen based on the received notification data (S54). In this way, it is possible to notify the store manager or the like in the waiting room when the internal deduction is performed. When the server 10 determines that the person to be tracked is not an employee of the store, that is, it is not an internal act (S51: NO), or after the process of step S53, the process of step S17 is performed. The process of steps S19 to S21 and S51 to S53 is repeated until the person to be traced returns the product to a predetermined position in the display shelf or the backyard. By the above-described processing, it is possible to give a warning to the tracking target person who has performed the shoplifting act or the internal shoplifting act, and to notify the store manager in the waiting room of the occurrence of the internal shoplifting act. Instead of the waiting room display device 43, for example, a smartphone or mobile terminal used only by the store manager may be notified of the occurrence of the internal drawing act.

In the present embodiment, it is possible to monitor and warn not only the shoplifting action that occurs in the sales floor of the store, but also the internal pulling action that occurs in the backyard of the store, for example. Therefore, by giving warnings and cautions to the traceable persons (employees of the store) who have a high possibility of performing internal deductions, the actions performed by the traceable persons are not internal deductions. It can warn you of something and encourage you to stop performing internal deductions. Further, regarding the tracking target person who has performed the internal deduction action, the tracking process for the tracking target person is ended by returning the product acquired by the internal deduction action to the original position, for example, and the warning process by the warning device 30 is ended. To be done.

In the present embodiment, when the occurrence of the internal pulling action is detected, the captured image determined to be the internal pulling action is associated with the employee ID of the employee identified as having performed the internal pulling action. It may be stored in the storage unit 12. In this case, it is possible to confirm the occurrence history of the internal pulling action based on the captured image. Further, when each employee leaves the store (goes home), there is a case where the employee leaves the store, for example, by reading the ID card of each employee with a reader. In this way, when the employee ID is read by the reader, if the read employee ID matches the employee ID of the employee who has performed the internal deduction action, warning processing may be performed. Good. For example, a display device may be provided near the reader, and a warning screen as shown in FIG. 9A or 9B may be displayed on this display device to warn the employee. Thereby, the employee can know that the act of himself/herself is an internal pulling act, and that it has been discovered that the internal pulling act has been performed, and therefore, the employee can respond to the performing of the internal pulling act.

The configuration of this embodiment can be applied to the information processing system 100 of the second and third embodiments, and the same effect can be obtained even when applied to the information processing system 100 of the second and third embodiments.

(Embodiment 5)
An information processing system 100 that causes a warning device 30, which is a display device, to display a predetermined screen when the occurrence of shoplifting is not detected will be described. The information processing system 100 of this embodiment is configured by the same devices as the devices of the information processing system 100 of the first embodiment. In the information processing system 100 of this embodiment, the control unit 11 of the server 10 realizes each function illustrated in FIG. 6 when the control program 12P stored in the storage unit 12 is executed. In addition, in the information processing system 100 of this embodiment, when the server 10 does not detect the occurrence of the shoplifting action based on the image data acquired by the camera 20, it suppresses the image (moving image) for advertising and the shoplifting action. A process of displaying an announcement image or the like on the warning device 30 of the display device is performed. The image data for displaying an image for advertisement (advertising data) and the image data for displaying an announcement image (announcement data) may be stored in the storage unit 12 of the server 10 in advance, The server 10 may receive from an external device via the network and store it in the storage unit 12.

FIG. 19 is a flowchart showing the procedure of shoplifting act monitoring processing by the information processing system 100 according to the fifth embodiment. The process shown in FIG. 19 is obtained by adding the processes of steps S61 to S65 to the process shown in FIG. 10 before step S13 performed by the server 10. Therefore, description of steps similar to those in FIG. 10 will be omitted. Further, in FIG. 19, illustration of steps S19 to S22 in FIG. 10 is omitted.

In the information processing system 100 of the present embodiment, the server 10 reads out image data (advertising data) for displaying an image for advertisement stored in the storage unit 12 and sends it to the warning device 30 which is a display device. It is transmitted (S61). The warning device 30, which is a display device, receives the advertisement data sequentially transmitted from the server 10 and displays an advertisement screen based on the received advertisement data (S62). The server 10 determines whether or not a predetermined timing for displaying the announcement image has come while displaying the advertisement screen on the warning device 30 which is a display device (S63). The announcement image may be displayed every 30 minutes, every hour, or the like. The server 10 determines whether or not a predetermined time has passed since the advertisement screen was displayed on the warning device 30, which is a display device, or whether a predetermined time has passed since the announcement image was displayed last time. If so, it is determined that the timing for displaying the announcement image has arrived.

When it is determined that the timing for displaying the announcement image has arrived (S63: YES), the server 10 reads and displays the image data (announcement data) for displaying the announcement image stored in the storage unit 12, for example. It transmits to the warning device 30 which is a device (S64). The alerting device 30, which is a display device, receives the announcement data transmitted from the server 10 and displays the announcement screen based on the received announcement data (S65). The announcement screen displays warning messages such as "Shoplifting is a crime" and "I will report as soon as I find it". When the server 10 receives the image data transmitted from the camera 20 while displaying the advertisement screen or the announcement screen on the warning device 30 which is a display device (S13), the server 10 performs the shoplifting action based on the received image data. A detection process for detecting the occurrence is performed (S14). The server 10 of the present embodiment returns to the process of step S61 when the shoplifting action is not detected (S15: NO) and when the tracking process is completed (S18).

By the above-described processing, when the server 10 does not detect the occurrence of shoplifting or after the person to be traced returns the product to the display shelf, the warning device 30 displays an advertisement screen or an announcement screen for suppressing shoplifting. (Display device) can be displayed. Therefore, when shoplifting has not occurred, it is possible to improve the customer's willingness to purchase by displaying the advertisement screen, and be careful not to perform shoplifting by displaying the announcement screen. Can be called. When the server 10 detects the occurrence of shoplifting, a warning screen can be displayed on the warning device 30 (display device) to warn the tracking target person. Since the warning screen is displayed on the warning device 30 (display device) near the person to be shoplifted, the other warning devices 30 (display devices) will continue to display advertisement screens and announcement screens. May be. The time for displaying the advertisement screen or the announcement screen for suppressing shoplifting (that is, the moving image reproduction time) can be changed as appropriate. Accordingly, in the information processing system 100 in which the advertising screen and the announcement screen are displayed in the normal state where the shoplifting is not performed, when the shoplifting is performed, the warning device 30 near the person who has performed the shoplifting is performed. A warning screen can be displayed by issuing a warning.

The configuration of the present embodiment can be applied to the information processing system 100 of the second to fourth embodiments, and the same effect can be obtained even when applied to the information processing system 100 of the second to fourth embodiments.

(Embodiment 6)
In the information processing system 100 of each of the above-described embodiments, when image data is input instead of the shoplifting discrimination model 121 or together with the shoplifting discrimination model 121, is the person photographed in the image data performing a violent act? You may use the violent behavior discrimination model learned so that the information which shows whether or not may be output. The violent behavior discrimination model is a learned model learned using teacher data including a photographed image of a person and information indicating whether or not the person in the photographed image committed a violent behavior. FIG. 20 is a schematic diagram showing an example of a captured image for learning the violent behavior discrimination model. The violent behavior discrimination model is image data (photographed image) obtained by photographing a person who is likely to commit violent behavior or a person who is committing violent behavior as shown in FIG. Learning is performed by using the teaching data in which one set of information (correct answer label) indicating that the above has been performed is set. Thereby, in the information processing system 100 of each of the above-described embodiments, when the captured image acquired by each camera 20 is input to the violent behavior discrimination model, it indicates whether or not the person in the captured image is performing the violent behavior. Information (discrimination probability) is output.

The violent behavior discrimination model learned in this way is stored in the storage unit 12 of the server 10, and the control unit (violence determination unit) 11 uses the violent behavior discrimination model and the captured image acquired by each camera 20. Based on the (image data), it is detected whether or not the person photographed in the photographed image committed a violent act. Further, when the control unit (notification unit) 11 detects that the person photographed in the photographed image is performing a violent act, for example, the warning device 30 provided in the store is used to generate the violent act. May be notified. In the case of such a configuration, not only can shoplifting or internal shoplifting be detected, but also violence in a store can be detected and a warning can be issued. The shoplifting determination model 121 may be configured not only to determine whether or not a shoplifting action has been performed, but also to determine whether or not a violent action has been performed. When the learned model that can detect the occurrence of violent behavior is used in this way, it can be applied not only to a store but also to a system that detects the occurrence of violent behavior at a window such as a public office or a police station and warns.

In addition, in the information processing system 100 according to the present embodiment, the server 10 may have a configuration that notifies the police station, the security company, the head office of the store, the headquarters, or the like when the occurrence of violence is detected. For example, when the control unit 11 of the server 10 detects that a person in the captured image acquired by each camera 20 has performed a violent act using the violent act discrimination model, the police station, the predetermined security company, the store, etc. It has a configuration to output the notification information to the terminal at the head office or the headquarters. The terminal of the security company, the terminal of the police station, the terminal of the store headquarters or the headquarters, when the notification information is acquired from the server 10, for example, by turning on or blinking a lamp or outputting a predetermined voice message, the security company. Notifying the security officers, police officers of the police station, personnel at the head office and headquarters, etc. of the report of violence. With such a configuration, for example, when violence is committed, it is possible to notify the security company, police station, store head office or headquarters at an early stage, and each person in charge can know the occurrence of violence. .. When receiving a report from the server 10, the security company, the police station, the head office of the store, or the headquarters can deal with violent acts.

In each of the above-described embodiments, a discrimination model for discriminating whether or not the person in the captured image is performing shoplifting, and whether or not the person (employee) in the captured image is performing internal pulling A discrimination model for discriminating and a discrimination model for discriminating whether or not a person in the captured image is committing violence may be separately provided. Further, the discriminant model (shoplifting discriminant model 121) may be configured such that the respective discriminant processes can be collectively output by one discriminant model. Also, in shoplifting and internal shoplifting, a discrimination model for determining whether or not an act of opening a product is performed, a discrimination model for determining whether or not an act of opening a beverage bottle and drinking is performed, etc. May be separately provided, or the discriminant model may be configured such that the respective discriminant processes can be collectively output as one discriminant model.

It should be considered that the embodiments disclosed this time are exemplifications in all points and not restrictive. The scope of the present disclosure is shown not by the meanings described above but by the claims, and is intended to include meanings equivalent to the claims and all modifications within the scope.

10 server 11 control unit 12 storage unit 20 camera 30 warning device 41 first display device 42 second display device 100 information processing system 101 image acquisition unit 102 shoplifting detection unit 103 tracking processing unit 104 tracking unit 105 action detection unit 106 position detection unit 107 Warning processing unit

In an information processing system according to an aspect of the present disclosure, a plurality of image capturing devices distributed in a store and a person imaged in the image data based on the image data acquired by the plurality of image capturing devices are A detection unit that detects whether or not the product displayed in the store is picked up, and when image data is input when it is detected that a person in the image data picks up the product. the image data person imaged in by using the learned learned models to output information indicating whether performed theft, based on the image data by the plurality of imaging devices has been acquired, the If the person who picked up the product was determined and determining theft determination section whether or not subjected to theft and theft by the theft determination unit is performed, based on the image data to which the imaging apparatus acquires Then, based on image data acquired by the tracking unit that tracks the person who has committed the theft, and the person that the tracking unit tracks by the imaging device, the product that has been stolen is displayed on the display shelf. The tracking unit ends the tracking process when it determines that the product taken by the theft has been returned to the display shelf .

Claims (8)

A plurality of imaging devices distributed in the store,
Based on the image data acquired by the plurality of imaging devices, a detection unit that detects whether or not the person imaged in the image data picks up the product displayed in the store,
When the image data is input, the plurality of image capturing devices are acquired by using a learned model learned to output information indicating whether or not the person imaged in the image data has committed theft. And a theft determination section that determines whether or not a theft has been performed based on the image data
The information processing system, wherein the theft determination unit determines, when it is detected that a person in the image data picks up a product, a person who picks up the product has performed a theft. When it is detected that a person in the image data picks up a product, an extraction unit that extracts an imaging region of a person who picks up the product from the image data is provided.
The information processing system according to claim 1, wherein the theft determination unit determines whether or not a theft has been performed based on an imaging region extracted from the image data. Multiple warning devices distributed in the store,
The information processing system according to claim 1 or 2, further comprising: a warning processing unit that, when it is determined that a theft has been performed, issues a warning by the warning device arranged near a person who has performed the theft. The warning device includes a light emitting device or a voice output device,
The information processing system according to claim 3, wherein the warning processing unit causes the light emitting device to turn on or blink light, or causes the sound output device to output a predetermined sound to give a warning. When it is determined that the theft has been performed by the theft determination unit, a tracking unit that tracks the person who has performed the theft based on image data acquired by the imaging device,
A tracking determination unit that determines whether or not the product taken by the theft has been returned to the display shelf based on image data acquired by the imaging device capturing the person tracked by the tracking unit. ,
The information processing system according to claim 1, wherein the tracking unit ends the tracking process when it is determined that the product taken by the theft has been returned to the display shelf. When the theft determination unit determines that a theft has been performed, the display unit that displays the image data determined to have the theft performed on the display device arranged in the employee waiting room in the store is further included. The information processing system according to any one of claims 1 to 5, further comprising: The information processing device
Based on the image data obtained by a plurality of imaging devices distributed in the store, it is detected whether or not the person imaged in the image data picked up the product displayed in the store,
When the image data is input, using a learned model that is learned to output information indicating whether the person imaged in the image data has committed theft, the person in the image data is An information processing method for executing a process of determining whether or not a person who picks up the product has committed a theft, based on the image data, when it is detected that the product is picked up. Based on the image data obtained by a plurality of imaging devices distributed in the store, it is detected whether or not the person imaged in the image data picked up the product displayed in the store,
When the image data is input, using a learned model that is learned to output information indicating whether the person imaged in the image data has committed theft, the person in the image data is A program that causes a computer to execute a process for determining whether or not a person who picks up the product has committed a theft based on the image data when it is detected that the product is picked up.