JP7161920B2 - store equipment - Google Patents

Description

The present invention relates to a store device for monitoring a store.

Conventionally, a store device installed in a store analyzes a photographed image of the inside of the store, and when it is necessary to deal with a person detected from the photographed image, an image for reporting to a monitoring device installed in a monitoring center. A monitoring system is used.

For example, Patent Literature 1 discloses an image monitoring system that monitors the status of a monitoring target using an image at a monitoring center. In this image monitoring system, an image monitoring device sends an emergency report to a monitoring center and a current image when an emergency report operation is performed by a user.

In addition, Patent Document 2 discloses a reporting device that determines whether or not an emergency report is necessary based on a robbery occurrence situation factor when the posture of a person detected from an image of a monitoring area matches the posture stored in a storage means. disclosed.

JP-A-2001-109973 JP 2012-3595 A

A store device that monitors a store is required to more accurately determine whether or not it is necessary to deal with a person detected from a photographed image of the inside of the store.

SUMMARY OF THE INVENTION An object of the present invention is to provide a store apparatus capable of calculating with high accuracy the degree of necessity of dealing with a person detected from a photographed image.

In order to solve such problems, the present invention provides a store apparatus for monitoring a store, comprising: an image acquiring section for acquiring a photographed image of the inside of the store; a person detecting section for detecting a person from the photographed image; a handling requirement calculation unit that calculates a handling requirement level indicating the degree of necessity of handling the detected person from the photographed image using the model obtained from the model; An automatic notification unit that transmits an automatic notification, a manual notification unit that transmits a manual notification to the monitoring device according to the notification instruction by the store clerk, and when the manual notification unit transmits the manual notification, before the notification instruction is executed An update unit for updating a model based on a photographed image photographed in a predetermined period is provided.

In this store device, the automatic notification unit prompts the store clerk to execute the notification instruction when the degree of required action is less than the threshold and is equal to or greater than the second threshold smaller than the threshold, and the update unit issues the notification instruction. It is preferable to change the method of updating the model depending on whether the manual notification is sent after prompting execution or when the manual notification is sent without prompting execution of the notification instruction.

In this store device, the updating unit considers that it is not necessary to deal with the detected person when the automatic notification unit urges the store clerk to execute the notification instruction and the manual notification unit does not send the manual notification. Preferably, the model is updated based on the captured image used when prompting execution of the notification instruction.

The store apparatus further includes a congestion determination unit that determines whether the store is crowded, the update unit, the automatic notification unit prompting the store clerk to execute a notification instruction, and the manual notification unit If no manual notification is sent, it is preferable to update the model only when the congestion determination unit determines that the store is not congested.

The store apparatus according to the present invention can calculate with high accuracy the degree of necessity of dealing with a person detected from a photographed image.

1 is a diagram showing an overall system configuration of an image monitoring system 1; FIG. It is a figure which shows an example of the flowchart which shows operation|movement of a monitoring process. 4 is a diagram showing an example of an operation sequence of the image monitoring system 1; FIG. It is an example of a display screen displayed on the monitoring device 30 .

An image monitoring system according to an embodiment will be described below with reference to the drawings.

FIG. 1 is a diagram showing the overall system configuration of an image monitoring system 1 according to an embodiment. As shown in FIG. 1, an image monitoring system 1 connects one or more store devices 10 installed in one or more stores, and each store device 10 via a communication network such as a local LAN or the Internet. and a monitoring device 30 . In addition, one or a plurality of imaging devices 21 , a sound output device 22 , an operation device 23 , and one or a plurality of terminal devices 24 are connected to each store device 10 .

The store apparatus 10 is, for example, a personal computer or the like, and has an interface section 11 , a first communication section 12 , a first storage section 13 and a first control section 14 .

The interface unit 11 has an interface circuit conforming to a serial bus standard such as USB, and communicates with the imaging device 21, the sound output device 22, the operation device 23 and/or the terminal device 24 to transmit and receive various information. Note that the interface unit 11 may have an interface circuit conforming to a short-range wireless communication standard such as WiFi, Bluetooth (registered trademark), NFC, etc., instead of the interface circuit conforming to the serial bus standard.

The first communication unit 12 has various communication interface circuits such as Ethernet (registered trademark), and transmits and receives various signals to and from the monitoring device 30 via a communication network.

The first storage unit 13 has a semiconductor memory such as ROM and RAM, a magnetic disk (HDD), or an optical disk drive such as CD-ROM and DVD-RAM and recording media thereof. The first storage unit 13 stores computer programs and various data for controlling the store apparatus 10 and inputs/outputs such information to/from the first control unit 14 . The computer program may be installed in the first storage unit 13 from a computer-readable portable recording medium such as CD-ROM, DVD-ROM, etc. using a known setup program or the like. The storage unit 15 also stores a model 131, a background image 132, and the like as data.
The model 131 is a determination model that is set in advance and is used to calculate a degree of need to deal with a person detected from a captured image of the inside of the store. A person who needs to be dealt with is, for example, a suspicious person or a person in trouble. Suspicious people include, for example, people who have stayed for a long time, people who have weapons such as knives, people who make strange noises, people who cover their faces with sunglasses or masks, or people who have committed theft such as shoplifting. It is a person or the like who is observing the surroundings by plotting. A person in need is, for example, a person who is looking for something, or a person who is crouched due to poor physical condition or the like.
The background image 132 is an image captured inside the store by each imaging device 21 when there is no moving object.

The first control unit 14 has processors such as CPU and MPU, memories such as ROM and RAM, and their peripheral circuits, and executes various signal processing of the store apparatus 10 . The first control unit 14 includes manual notification means 141, image acquisition means 142, person detection means 143, response level calculation means 144, automatic notification means 145, update means 146, which are implemented as functional modules of a program that operates on the processor. and congestion determination means 147 and the like. A DSP, LSI, ASIC, FPGA, or the like may be used as the first control unit 14 .

The imaging device 21 is a camera that is fixedly installed in the store where the store device 10 is installed and captures the inside of the store. The imaging device 21 includes a photoelectric conversion element sensitive to visible light, such as a CCD element or a C-MOS element, an imaging optical system that forms an image on the photoelectric conversion element, and an electric current output from the photoelectric conversion element. and an A/D converter that amplifies and analog-to-digital (A/D) converts the signal. The imaging device 21 converts the captured RGB color image into a digital captured image in which each pixel has a luminance value in the range of 0 to 255, and outputs the digital captured image to the store device 10 .

The sound output device 22 is a speaker that is installed in the store where the store device 10 is installed and that outputs sound inside the store. The sound output device 22 receives an audio signal from the store device 10 and outputs audio to a person in the store, such as a salesclerk or a customer, according to the received audio signal.

The operation device 23 is a button that is installed in the store where the store device 10 is installed, particularly around a cash register operated by a store clerk, and receives a notification instruction from the store clerk. The operation device 23 outputs to the store device 10 a notification instruction signal corresponding to the received notification instruction. In this embodiment, the operation device 23 is a report button, and outputs a report instruction signal to the store device 10 when pressed by the store clerk.

The terminal device 24 is a terminal carried by a store clerk of the store device 10, and is, for example, an intercom (intercommunication), a smart phone, or the like. The terminal device 24 has an interface for connecting with the store device 10, a notification unit such as a display device and/or an audio output device for notifying the store clerk, and a processor for controlling them. The terminal device 24 receives the notification signal from the store device 10 and notifies the store clerk by image or sound according to the received notification signal.

Note that the imaging device 21 , the sound output device 22 , the operation device 23 and/or the terminal device 24 may not be provided separately from the store device 10 but may be provided integrally with the store device 10 .

The monitoring device 30 is, for example, a server or the like, and has a second communication section 31 , an operation section 32 , a display section 33 , a second storage section 34 and a second control section 35 .

The second communication section 31 has a communication interface circuit similar to that of the first communication section 12, and transmits/receives various signals to/from the store apparatus 10 via the communication network.

The operation unit 32 is, for example, a touch panel, a mouse, a keyboard, etc., receives operations by the monitoring staff of the monitoring center, and sends signals corresponding to the received operations to the second control unit 35 .

The display unit 33 is, for example, a liquid crystal display, an organic EL display, or the like, and displays information, captured images, and the like received from the store device 10 according to instructions from the second control unit 35 .

The second storage section 34 has a semiconductor memory similar to that of the first storage section 13, a magnetic disk or an optical disk drive, and its recording medium. The second storage unit 34 stores computer programs and various data for controlling the monitoring device 30 and inputs/outputs such information to/from the second control unit 35 . The computer program may be installed in the second storage unit 34 from a computer-readable portable recording medium such as CD-ROM, DVD-ROM, etc. using a known setup program or the like.

The second control unit 35 has the same processor, memory and peripheral circuits as those of the first control unit 14 and executes various signal processing of the monitoring device 30 . The second control unit 35 has evaluation means 351 and the like implemented as functional modules of a program that runs on the processor.

FIG. 2 is a diagram showing an example of a flowchart showing the operation of monitoring processing by the store apparatus 10. As shown in FIG. The operation of the registration process according to this embodiment will be described below with reference to the flowchart shown in FIG. The operation flow described below is executed mainly by the first control unit 14 in cooperation with each unit of the store apparatus 10 based on a program pre-stored in the first storage unit 13 .

First, the manual notification means 141 determines whether or not a notification instruction is input from the store clerk from the operation device 23 via the interface section 11 (step S101). When the notification instruction is input by the store clerk, the manual notification means 141 determines to transmit a manual notification indicating that the notification has been instructed by the store clerk to the monitoring device 30 (step S102), and the process proceeds to step S101. back to The manual notification includes the identification information of the input store. Furthermore, the manual notification may include identification information based on the work schedule information of the clerk who instructed the notification. In this case, the manual notification means 141 transmits a manual notification to the monitoring device 30 in a process to be described later. In this manner, the manual reporting means 141 transmits manual reporting to the monitoring device 30 in accordance with instructions from the store clerk.

On the other hand, when the store clerk has not input a notification instruction from the operation device 23, the image acquiring means 142 acquires the image captured by the image capturing device 21 of the inside of the store from the image capturing device 21 via the interface unit 11 ( step S103). In this embodiment, the image acquisition unit 142 acquires a plurality of captured images captured during the immediately preceding predetermined period (for example, 10 seconds).

Next, the person detection means 143 detects a person from the photographed image (step S104). The person detection means 143 obtains the luminance difference between corresponding pixels between the photographed image and the background image 132 photographed by the imaging device 21 that photographed the photographed image, and the pixel value of each pixel is the absolute value of the luminance difference. Generates a difference image represented by a value. The person detection unit 143 detects, in the captured image, an area in which the pixel value in the corresponding difference image is equal to or greater than a predetermined threshold as a moving object area. For example, the predetermined threshold can be the average value of each pixel value of the difference image. When the detected moving object area has a size within a predetermined range corresponding to a person, the person detecting means 143 detects the moving object area as a person area containing a person.
Further, the person detection unit 143 executes tracking processing using a known tracking technique for each person region detected in the photographed images sequentially acquired by the image acquisition unit 142 . For example, for each person area in the latest captured image, the person detection unit 143 calculates the distance between the centroid position of the person area and the centroid position of the person area detected from the immediately preceding captured image. A person area that is equal to or less than a predetermined threshold value is associated as a person area of the same person. Note that the person detection unit 143 may perform tracking processing using methods such as optical flow and particle filtering.
If a person is not detected from the captured image, the person detection unit 143 returns the process to step S101.

On the other hand, when a person is detected from the photographed image, the handling requirement calculating means 144 calculates the handling requirement level of each detected person from the photographed image using the preset model 131 (step S105).

式（１）において、ｓ_t及びａ_tはそれぞれ時刻ｔにおける状態及び行動であり、行動ａ_tにより状態はｓ_tからｓ_t+1に変化する。ｒ_t+1は、状態がｓ_tからｓ_t+1に変化したことで得られる報酬である。ｍａｘＱの項は、時刻ｔ＋１で最大の価値Ｑになる（と時刻ｔで考えられている）行動ａを行ったときのＱを意味する。α及びγはそれぞれ学習係数及び割引率であり、０＜α≦１、０＜γ≦１で任意に設定される。 As the model 131, the store apparatus 10 uses a learning model learned in advance by reinforcement learning. The store apparatus 10 uses, for example, Q-learning as reinforcement learning and an action-value function as a learning model. In Q-learning, the action value function Q(s, a) representing the action value when the action a is selected in the state s is obtained by using the environmental state s and the action a selected in the state s as independent variables. be learned. In Q-learning, learning is started in a state where the correlation between state s and action a is unknown, and by repeating trial and error to select various actions a in arbitrary state s, action-value function Q repeats Updated. In addition, in Q-learning, when an action a in a certain state s is selected, a reward r is obtained, and an action value function Q is learned.
An update formula for the action-value function Q is expressed as the following formula (1).

In equation (1), s _t and a _t are the state and action at time t, respectively, and the state changes from s _t to s _t+1 by action a _t . r _t+1 is the reward obtained by changing the state from s _t to s _t+1 . The term maxQ means the Q when the action a that becomes the maximum value Q at the time t+1 (considered at the time t) is performed. α and γ are a learning coefficient and a discount rate, respectively, and are arbitrarily set within 0<α≦1 and 0<γ≦1.

The store device 10 is set with an action value function _Q1 for an image containing a person who needs to be dealt with, and an action value function _Q2 for an image without a person who needs to be dealt with. The action-value function Q ₁ is learned in advance using a plurality of sample image groups in which a person requiring action is shown, and the action-value function Q ₂ is learned in advance by using a group of sample images in which a person requiring action is not shown. Pre-learned using multiple. Each sample image group is a set of consecutive captured images captured by each imaging device 21 during a predetermined period. The shop apparatus 10 sets the time when each photographed image was photographed to be time t, the barycentric position of the person region detected from the photographed image at time t to state s _t , and the barycentric position of each person region to be the photographed at time t. The movement from the centroid position of the human region detected from the image to the centroid position of the human region detected from the captured image at time _t +1 is used as the action at. Also, the store apparatus 10 sets a value greater than 0 as the reward r for the last action a in the sample image group, and sets 0 as the reward r for the other actions a.

The store device 10 randomly sets the initial value of Q(s, a) for all combinations of all states s and actions a that can be taken in each state s. The shop apparatus 10 refers to each image included in one sample image group in chronological order, and updates Q(s, a) corresponding to the human region detected from each image according to the above equation (1). go. The shop device 10 updates the action value function Q ₁ (s, a) using all the sample image groups showing the person who needs to be dealt with, and finally updates each Q ₁ (s, A combination of a) values (behavioral values) is stored in the first storage unit 13 as a model 131 . Similarly, the store device 10 updates the action-value function Q ₂ (s, a) using all the sample image groups that do not include a person who needs to be dealt with, and finally updates each Q ₂ A combination of (s, a) values (behavior value) is stored in the first storage unit 13 as a model 131 .

The degree-of-measures-required calculation means 144 sets the barycenter position of the person region of each person detected from each captured image acquired by the image acquisition means 142 to state s, and uses the barycentric position to determine the person detected from the next captured image. movement to the position of the center of gravity of the person region is identified as action a. The action required degree calculation means 144 refers to the model 131, the value (action value) of Q ₁ (s, a) corresponding to the combination of the identified state s and the action a, and the value of Q ₂ (s, a) Identify value (behavioral value). The action required degree calculation means 144 subtracts the value of Q ₂ (s, a) (action value) from the identified value of Q ₁ (s, a) (action value), and calculates the value as the action value corresponding to the combination. Calculate as The handling requirement calculating means 144 calculates, for each detected person, the sum of the action values for the person calculated for each photographed image, and calculates the maximum value among the calculated sums as the handling requirement level. That is, the degree of need to be dealt with is calculated so as to be higher as the probability that a person who needs to be dealt with is shown in the photographed image is higher.

Next, the automatic notification means 145 determines whether or not the calculated degree of need for action is equal to or greater than a threshold (step S106). The threshold is set in advance based on experimental results from previous experiments. If the degree of required action calculated for any person is equal to or greater than the threshold, the automatic notification means 145 sends an automatic notification to the monitoring device 30 indicating that the store device 10 has determined that a notification is necessary based on the photographed image. (step S107), and the process returns to step S101. The automatic notification includes the identification information of the input store. Furthermore, the automatic notification may include identification information (for example, the coordinates of the person area in the captured image) of the person whose total behavioral value is equal to or greater than the threshold or the second threshold. In this case, the automatic notification means 145 transmits an automatic notification to the monitoring device 30 in a process to be described later.
On the other hand, when the degree of action required calculated for all persons is less than the threshold, the automatic reporting means 145 determines whether or not the degree of action required is equal to or larger than a second threshold, which is smaller than the threshold (step S108). The second threshold is set in advance based on experimental results from previous experiments. If the degree of need for action calculated for all persons is less than the second threshold, the automatic notification means 145 determines not to transmit the notification to the monitoring device 30 (step S109), and returns the process to step S101.
On the other hand, if the degree of required action calculated for any person is equal to or higher than the second threshold, the automatic notification means 145 sends a message to each store clerk prompting execution of the notification instruction to each store clerk via the interface unit 11. is transmitted to the terminal device 24 possessed by and notified (step S110). When each terminal device 24 receives the message prompting execution of the notification instruction, it displays or outputs the received message by voice, and notifies each store clerk. In this way, the automatic notification means 145 transmits an automatic notification to the monitoring device 30 of the monitoring center according to a predetermined standard when the level of action required is equal to or higher than the second threshold, and the terminal device 24 of the store clerk prompting the user to execute the notification instruction. In particular, the automatic notification means 145 selects to send an automatic notification to the monitoring device 30 when the degree of action required is equal to or higher than the threshold, and to the terminal device 24 of the store clerk when the degree of action required is less than the threshold. Select to prompt execution of notification instructions.

Next, the automatic notification means 145 determines whether or not a notification instruction signal has been received from the operation device 23 or the terminal device 24 via the interface section 11 (step S111). When the notification instruction signal is received, the automatic notification means 145 determines to transmit a manual notification to the monitoring device 30 (step S112), and returns the process to step S101. In this case, the manual notification means 141 transmits a manual notification to the monitoring device 30 in a process to be described later. The manual notification in this case may include the identification information of the person whose total behavioral value is equal to or greater than the second threshold.
On the other hand, if the notification instruction signal has not been received, the automatic notification means 145 determines whether or not a predetermined time (for example, 10 seconds) has passed since the message was sent to the terminal device 24 (step S113). If the predetermined time has not elapsed, the automatic notification means 145 returns the process to step S111 and determines again whether or not the notification instruction signal has been received.

On the other hand, when the predetermined time has passed, that is, when the store clerk does not execute the notification instruction, the congestion determination means 147 calculates the degree of congestion indicating the degree of congestion in the store from the photographed image (step S114). For example, the congestion determination unit 147 calculates the number of persons detected from the photographed image by the person detection unit 143 as the degree of congestion. Note that the congestion determination unit 147 may calculate the number of people who are located within a preset range, such as in front of a cash register, among the people detected from the photographed image, as the degree of congestion. Further, the congestion determination unit 147 may calculate the degree of congestion by weighting each person detected from the captured image so that the weight of the person located within the predetermined range is increased.
Next, the congestion determining means 147 determines whether the store is crowded based on the calculated congestion degree (step S115). The congestion determining means 147 determines that the store is crowded when the degree of congestion is equal to or greater than a predetermined threshold, and determines that the store is not crowded when the degree of congestion is less than the predetermined threshold.

When it is determined that the store is crowded, the updating means 146 returns the process to step S101. On the other hand, when it is determined that the store is not crowded, the updating means 146 updates the model 131 stored in the first storage unit 13 based on the photographed image used when prompting the execution of the notification instruction. (step S116), and the process returns to step S101.
That is, when the automatic notification means 145 prompts the store clerk to execute a notification instruction and the manual notification means 141 does not send a manual notification to the monitoring device 30, the update means 146 updates the person detected from the captured image. The model 131 is updated assuming no action is required. However, only when the congestion determination means 147 determines that the store is not congested when the automatic notification means 145 urges the store clerk to execute the notification instruction and the manual notification means 141 does not transmit the manual notification. Update the model 131. As a result, the updating means 146 can prevent the model 131 from being erroneously updated when the clerk is too busy to execute the notification instruction.

The updating means 146 updates the model 131 in the same way as when the model 131 is initialized. However, the updating means 146 updates the model 131 by using the photographed images photographed during a predetermined period immediately before and when a message prompting each store clerk to execute the notification instruction is sent to the terminal device 24 . Note that the updating unit 146 may update the model 131 using only the person regions of persons whose total behavior value is equal to or greater than the second threshold.
The update unit 146 sets the time when each photographed image was photographed to be time t, the barycentric position of the person region detected from the photographed image at time t to state s _t , and the barycentric position of each person region to be the photographed at time t. The movement from the centroid position of the human region detected from the image to the centroid position of the human region detected from the captured image at time _t +1 is used as the action at. The updating means 146 sets a value greater than 0 as the reward r for the action a related to the last photographed image among the respective photographed images corresponding to the received determination result, and sets 0 as the reward r for the other action a. .
The updating means 146 refers to each photographed image corresponding to the received determination result in chronological order, and updates Q(s, a) corresponding to the human region detected from each photographed image according to the above equation (1). To go. Since it is deemed unnecessary to deal with the person appearing in the photographed image, the updating means 146 updates the action value function Q ₂ (s, a), and the model 131 is finally updated to each Update to a combination of Q ₂ (s, a) values (action values).
As a result, the updating unit 146 can update the model 131 so that it is difficult for a report to be sent for a photographed image in which a person who needs to be dealt with is not shown.

FIG. 3 is a diagram showing an example of an operation sequence of the image monitoring system 1. As shown in FIG. This operation sequence is executed mainly by each control unit of each device in cooperation with each element of the store device 10 and the monitoring device 30, based on a program stored in advance in each storage unit of each device. . This operation sequence is executed when the store device 10 decides to transmit a manual report or an automatic report to the monitor device 30 in the monitoring process.

First, when the automatic notification means 145 of the store device 10 determines to transmit the automatic notification in step S109 of FIG. to the monitoring device 30 via the first communication unit 12 (step S201). In this case, as the photographed images, images photographed during a predetermined period immediately before and when the automatic notification was transmitted, and live images are transmitted together.

Next, when the evaluation unit 351 of the monitoring device 30 receives the automatic notification and the captured image from the store device 10 via the second communication unit 31, the information indicating the received notification and the captured image are displayed on the display unit 33. (Step S202).

FIG. 4 is an example of a display screen displayed on the monitoring device 30. As shown in FIG.
As shown in FIG. 4, the display screen 400 includes information 401 indicating the received report, an image 402 photographed at the time of report by automatic report or manual report, a live image 403 displaying a live image, a voice notification button 404, a report button 405, an end button 406, and the like are displayed.
The information 401 displays the identification information of the shop where the store device 10 that sent the report is installed, and whether the report received from the store device 10 is a manual report or an automatic report. When the report received from the store device 10 is a manual report, the information 401 may further display the identification information of the clerk or the terminal device 24 who instructed the report. Further, when the report received from the store device 10 includes the identification information of a person whose total behavioral value is equal to or greater than the threshold or the second threshold, the information 401 further displays the identification information of the person. good too.
By pressing the voice notification button 404 , the surveillance staff can cause the sound output device 22 to output a predetermined voice via the store device 10 .
By pressing the report button 405, the surveillance staff can cause the monitoring device 30 to send a report to another device (not shown) installed in the police or the like.
The observer can terminate the display of the display screen 400 by pressing the end button 406 .

The manual notification means 141 of the store device 10, when it is determined to transmit the manual notification in step S102 or S112 of FIG. to the monitoring device 30 via the first communication unit 12 (step S203). If it is determined in step S102 to transmit the manual notification, an image in which a person is detected and photographed before the manual notification is transmitted as the photographed image. If it is determined in step S112 that the manual report is to be sent, the photographed image is taken at the time when a message prompting each store clerk to execute the report instruction was sent to the terminal device 24 and for a predetermined period immediately before that. Image is sent.

Next, when the evaluation unit 351 of the monitoring device 30 receives the manual report and the photographed image from the store device 10 via the second communication unit 31, the display screen 400 including the information indicating the received report and the photographed image is displayed. It is displayed on the unit 33 (step S204).

When the manual notification is transmitted to the monitoring device 30, the updating means 146 of the store device 10 considers that the person detected from the captured image needs to be dealt with, and based on the captured image corresponding to the manual notification, updates the 1 The model 131 stored in the storage unit 13 is updated (step S205).
If it is decided to send a manual report in step S112 of FIG. set. Further, the updating means 146 uses, as images to be used for updating the model 131, photographed images photographed during a predetermined period immediately before and at the time when a message prompting each store clerk to execute a notification instruction is transmitted to the terminal device 24. set. The updating means 146 may set only the human region of the person whose total behavioral value is equal to or greater than the second threshold as the image used for updating the model 131 .
If it is decided to send a manual report in step S102 of FIG. Set degree. The second degree is greater than the first degree. Further, the updating means 146 sets, as an image used for updating the model 131, a photographed image in which a person is detected and photographed before the notification instruction is executed, that is, before the manual notification.
In this way, when the evaluation result corresponds to manual notification, the updating means 146 sets the captured image captured during the predetermined period before the notification instruction is executed as the image used for updating the model 131 .

The update means 146 updates the model 131 in the same manner as the process of step S116 in FIG. However, the updating means 146 updates the model 131 based on the set degree and the photographed image corresponding to the manual notification. In this case, since it is considered necessary to deal with the person appearing in the photographed image, the updating means 146 updates the action-value function Q ₁ (s, a), and finally updates the model 131. update to the combination of each Q ₁ (s, a) value (behavior value) obtained.
Further, when the degree of updating the model 131 is set to the first degree, the updating means 146 sets the first value as the reward r for the action a related to the last photographed image among the photographed images corresponding to the evaluation result. do. Further, when the degree of updating the model 131 is set to the second degree, the updating means 146 sets the reward r of the action a related to the last photographed image among the photographed images corresponding to the evaluation result to be larger than the first value. Sets the second value.

In this way, the updating means 146 changes the updating method of the model 131 depending on whether the execution of the notification instruction is prompted and the manual notification is sent or when the manual notification is sent without prompting the execution of the notification instruction. do. In particular, the updating means 146 updates the model 131 more than the case where the execution of the notification instruction is prompted and the manual notification is sent when the evaluation result indicates that the manual notification is sent without prompting the execution of the notification instruction. increase the degree to which If a manual report is voluntarily sent by the store clerk even though the execution of the report instruction is not prompted, there is a high possibility that there is someone in the store who needs to be dealt with. Therefore, the updating means 146 can update the model 131 appropriately.
Note that the updating means 146 updates the model 131 to a greater extent than when the manual notification is sent without prompting the execution of the notification instruction when the manual notification is sent with the execution of the notification instruction being prompted. You may In that case, the second degree is set to a value smaller than the first degree. When the model 131 is sufficiently learned, it is more likely that a person who needs to be dealt with is shown in the captured image when the execution of the notification instruction is prompted. Therefore, the updating means 146 can update the model 131 appropriately.

Thus, the operation sequence of the image monitoring system 1 ends.
Note that the store device 10 may omit the processing of steps S108 to S116 of FIG. Alternatively, the store apparatus 10 may omit the processing of steps S114 to S115 in FIG. 2 and update the model 103 regardless of whether the store is crowded.

Further, the monitoring device 30 may transmit the monitoring time to the store device 10 when the store device 10 manually reports in step S203 of FIG. In that case, in step S205, the updating means 146 may change the degree of updating the model 131 based on the monitoring time received from the monitoring device. When the monitoring time is long, it takes a long time for the monitor to determine whether or not the person in the photographed image needs to be dealt with, and the person's behavior is conspicuous in the photographed image. It is highly likely that it is not reflected in Conversely, if the monitoring time is short, the monitor quickly determines whether or not the person in the captured image needs to be dealt with. likely to be
Therefore, the updating means 146 increases the degree of updating the model 131 as the monitoring time becomes shorter, and decreases the degree of updating the model 131 as the monitoring time becomes longer. For example, the updating unit 146 sets a larger value as the reward r for the action a related to the last captured image among the captured images corresponding to the determination result as the monitoring time becomes shorter, A small value is set as the reward r for the action a related to the last captured image among the captured images corresponding to .
Thereby, the updating means 146 can update the model 131 more efficiently.

As described above, the store apparatus 10 according to the present invention transmits an automatic notification to the monitoring apparatus 30 based on the degree of action required calculated from the photographed image using the model 131, and according to the notification instruction from the store clerk. A manual notification is sent to the monitoring device 30 . When the store device 10 transmits the manual notification, it considers that the person detected from the captured image needs to be dealt with, and updates the model 131 . As a result, the store apparatus 10 can appropriately update the model 131, and as a result, can calculate the required action level with high accuracy.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments. In this embodiment, an example in which the model 131 is stored in the store device 10 has been described, but the model 131 may be stored in an external device and used in the form of cloud computing. In that case, the handling level calculation means 144 transmits the image including the person area to the external device via the first communication unit 12, and sends the handling level calculated by the external device to the external device via the first communication unit 12. Receive from device. In addition, the updating means 146 transmits the captured image to the external device via the first communication unit 12 and causes the external device to update the model 131 . As a result, the image monitoring system can use a device that does not have sufficient processing speed and storage capacity as the store device 10 .

In addition, in step S106 of FIG. 2, the automatic notification means 145 determines whether the person appearing in the photographed image is a suspicious person or a person in need, not whether or not the degree of need to be dealt with is equal to or greater than the threshold. You can judge.
In this case, the store device 10 is set with an action value function _Q3 for an image containing a suspicious person and an action value function _Q4 for an image containing a troubled person. The action value function _Q3 is learned in advance using a plurality of sample image groups showing suspicious persons, and the action value function _Q4 is learned in advance using a plurality of sample image groups showing troubled persons. be done. Action-value functions _Q3 and _Q4 are set in the same manner as action-value functions _Q1 and _Q2 . The action required degree calculation means 144 calculates the value of Q ₃ (s, a) (behavioral value) and the value of Q ₄ (s,a) (behavioral value) for each human region detected from each photographed image. Identify. If the sum of Q ₃ (s, a) values (behavior value) for each person calculated for each photographed image is equal to or greater than the sum of Q ₄ (s, a) values (behavior value). If there is, the person is determined to be a suspicious person. On the other hand, if the sum of the values of Q ₃ (s, a) (behavioral value) is less than the sum of the values of Q ₄ (s, a) (behavioral value), the handling requirement level calculating means 144 determines that the person is in trouble. It is determined that the person is
If the person appearing in the photographed image is a suspicious person, the automatic notification means 145 determines in step S107 to transmit an automatic notification to the monitoring device 30, and the person appearing in the photographed image is a person in trouble. If so, the process proceeds to step S108, and it is determined whether or not the degree of need for action is equal to or greater than the second threshold.
As a result, when it is estimated that a suspicious person appears in the photographed image, it is automatically reported to the monitoring device 30, and appropriate measures are taken by the monitoring staff. On the other hand, when it is presumed that a person in trouble is shown in the photographed image, the clerk is first notified, and the clerk takes measures. be done. Therefore, in the image monitoring system, it is possible to prevent the monitoring device from being notified to the monitoring device 30 and placing a burden on the monitoring personnel even though there is no need for countermeasures.

In addition, the store device 10 does not set the learning model in advance, but based only on the photographed image captured during operation, the execution result of the manual notification, and the evaluation result transmitted from the monitoring device 30, the model 131 may be generated and updated. When building a new image monitoring system, the administrator of the image monitoring system does not need to learn the learning model in advance, so the image monitoring system can be built easily in a short period of time.

Further, in step S203 of FIG. 3, the evaluation means 351 sends the evaluation result, the photographed image and/or the type of report (manual report or automatic report) to another store apparatus 10 different from the store apparatus 10 that transmitted the report. You may send. In that case, the updating means 146 of the other shop device 10 updates the model 131 stored in the own device based on the received evaluation result, photographed image, and report type. As a result, when the image monitoring system has a plurality of nationwide chain stores as stores, the data acquired by the store devices 10 of each store is centrally managed and horizontally deployed, and the data is used by each store device 10. The model 131 can be updated efficiently.

Further, in step S105 of FIG. 2, the store device 10 uses a learning model (discriminator) determined by another learning method such as supervised learning instead of the learning model (action value function) determined by reinforcement learning. may be used to determine whether the captured image contains a person that needs to be addressed.
In this case, the handling requirement calculation unit 144 calculates the handling requirement level using a discriminator pre-trained to output the handling requirement level when an image is input. This discriminator is pre-learned, for example, by deep learning, etc., using a plurality of sample images containing a person who needs to be dealt with, and a plurality of sample images not containing a person who needs to be dealt with. 131 is stored in advance in the first storage unit 13 . The handling requirement calculation unit 144 calculates the handling requirement level by inputting the image including the person region detected by the person detection unit 143 into the classifier and obtaining the handling level output from the classifier.
On the other hand, in step S116 of FIG. 2 and step S205 of FIG. When a person appearing in the captured image needs to be dealt with, the updating means 146 adds the person area in the captured image to the sample image showing the person who needs to be dealt with, and re-identifies the classifier. learn. On the other hand, if the person in the captured image does not need to be dealt with, the updating means 146 adds the person area in the captured image to the sample image showing the person who does not need to be dealt with. to relearn.
In this case as well, the image monitoring system can appropriately update the model 131, and as a result, can calculate the degree of action required with high accuracy.

Alternatively, the shop apparatus 10 may determine whether or not a person who needs to be dealt with is shown in the captured image by pattern matching.
In this case, the shop apparatus 10 first stores, as models 131, a plurality of first image patterns showing a person who needs to be dealt with and a plurality of second image patterns showing a person who does not need to be dealt with. Store in section 13 . The action required degree calculation means 144 calculates the degree of similarity between the human region detected from the photographed image and each image pattern stored as the model 131 . The degree of similarity can be, for example, a normalized cross-correlation value between the person area and an image obtained by enlarging/reducing the image pattern to the same size as the person area. The handling requirement calculating means 144 calculates the handling requirement level by dividing the maximum value of similarities between the person region and each first image pattern by the maximum value of similarities between the person region and each second image pattern. Calculate as
On the other hand, in step S116 of FIG. 2 and step S205 of FIG. When the person appearing in the photographed image needs to be dealt with, the updating means 146 adds the person area in the photographed image to the first image pattern. On the other hand, if the evaluation result indicates that no action is required for the person appearing in the photographed image, the updating means 146 adds the person area in the photographed image to the second image pattern.
In this case as well, the image monitoring system can appropriately update the model 131, and as a result, can calculate the degree of action required with high accuracy.

As described above, various modifications can be made within the scope of the present invention according to the embodiment.

10 store device, 141 manual notification means, 142 image acquisition means, 143 person detection means, 144 response level calculation means, 145 automatic notification means, 146 update means, 147 congestion determination means, 30 monitoring device, 351 evaluation means

Claims (4)

A store device for monitoring a store,
an image acquisition unit that acquires a photographed image of the inside of the store;
a person detection unit that detects a person from the captured image;
a handling requirement level calculation unit that calculates a handling requirement level indicating the degree of necessity of handling the detected person from the captured image using a preset model;
an automatic notification unit that transmits an automatic notification to a monitoring device when the degree of action required is equal to or greater than a threshold;
A manual report unit that transmits a manual report to a monitoring device according to a report instruction by a store clerk;
an updating unit that, when the manual notification unit transmits the manual notification, updates the model based on a captured image captured during a predetermined period before the notification instruction is executed;
A store device characterized by having: The automatic notification unit prompts a store clerk to execute the notification instruction when the degree of need to be dealt with is less than the threshold and is equal to or greater than a second threshold smaller than the threshold,
The updating unit changes a method of updating the model depending on whether the manual notification is sent after prompting execution of the notification instruction or when the manual notification is sent without prompting execution of the notification instruction. The store device according to claim 1, wherein When the automatic notification unit urges the store clerk to execute the notification instruction and the manual notification unit does not send the manual notification, the update unit determines that it is not necessary to deal with the detected person. 3. The store apparatus according to claim 2, wherein said model is updated based on a photographed image used when execution of said notification instruction is prompted. It further has a congestion determination unit that determines whether the store is crowded,
The update unit determines that the store is not congested when the automatic notification unit urges the store clerk to execute the notification instruction and the manual notification unit does not send the manual notification. 4. The store apparatus according to claim 3, wherein said model is updated only when said model is updated.

Images