JP2024040203A - Person detection device, person tracking apparatus, person tracking system, person detection method, person tracking method, person detection program and person tracking program - Google Patents

Abstract

To allow a monitor to easily recognize accuracy of an authentication result.SOLUTION: A detection/tracking result determination function unit 34 acquires from a monitoring/tracking execution function unit 23, similarity to an image of a similarity calculation object person being an object image and an image of a specific person being a specific image extracted respectively from monitor images periodically acquired respectively by a plurality of monitor cameras 10, detects an image similar to the image of the specific person from the images of the similarity calculation object person extracted from the monitor images of the plurality of monitor cameras 10 on the basis of a plurality of thresholds stored in a camera information table storage unit 321, and detects one of a plurality of attention stages which the detected image of the similarity calculation object person belongs to. A detection alert output function unit 3521 generates a detection alert screen 42 for presenting the detected image similar to the image of the specific person together with a stage display indicating the detected attention stage.SELECTED DRAWING: Figure 3

Description

One aspect of the present invention relates to a person detection device, a person tracking device, a person tracking system, a person detection method, a person tracking method, a person detection program, and a person tracking program.

2. Description of the Related Art In recent years, with the development of technology for person authentication based on the similarity of features of a person's face or whole body extracted from an image, various methods of using the authentication technology have been proposed. For example, Patent Document 1 discloses a technology that transmits a face authentication result of a person extracted from a surveillance image obtained by a surveillance camera to an external terminal such as a surveillance device via a communication interface section and displays it on the screen display section. is suggesting.

Patent No. 6627894

Depending on the type and performance of the surveillance camera that acquires surveillance images, the output results of face authentication and whole body verification will vary. Therefore, it is difficult for the supervisor to recognize the reliability of the authentication result from the authentication result displayed on the screen display section of the monitoring device.

This invention was made in view of the above-mentioned circumstances, and its purpose is to enable a supervisor to easily recognize the reliability of authentication results.

In order to achieve the above object, one aspect of the person detection device according to the present invention is based on the similarity calculated by a similarity calculation unit that calculates the similarity of a target image to a specific image using one similarity calculation method. a threshold storage unit that stores a plurality of threshold values for classifying the target image into a plurality of stages based on the target image, and the target image extracted from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras. Obtain one degree of similarity of the image of the person to be calculated from the similarity calculation unit to the image of the specific person, which is the specific image, and based on the plurality of thresholds stored in the threshold storage unit, Detecting an image similar to the image of the specific person from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras, and detecting the detected similarity calculation target person. a detection unit that detects which of a plurality of attention stages the image is in, and an image similar to the image of the specific person detected by the detection unit, together with a stage display indicating the attention stage detected by the detection unit. An output screen generation unit that generates an output screen for presentation.

One aspect of the person tracking device according to the present invention includes the person detecting device according to the present invention, and a plurality of images similar to the specific person image presented on the output screen by the person detecting device together with the stepwise display. A designation of a face image of the person to be tracked to be tracked from among the images of the person to be tracked for similarity calculation is received from a monitoring terminal operated by a monitor, the specified image is registered as a tracking image, and the The tracking image, which is the registered image of the tracking target person to be tracked, is used as the specific image, and the similarity calculation unit uses the similarity calculation method to calculate the similarity calculation image of the tracking target person. a registration unit that extracts the degree of similarity; and a registration unit that extracts the degree of similarity; and a tracking output screen generation unit that generates a tracking output screen to be presented together with a stage display indicating the caution level detected by the user.

Alternatively, in one aspect of the person tracking device according to the present invention, the target image is detected based on the similarity calculated by a similarity calculation unit that calculates the similarity of the target image with respect to a specific image using one similarity calculation method. A threshold storage unit that stores a plurality of threshold values for classifying images into a plurality of stages; and a registration unit that registers, as the specific image, an image of a person to be tracked who is to be tracked as specified by a monitoring terminal operated by a monitoring person. and obtaining from the similarity calculation unit the similarity of the image of the person to be calculated, which is the target image extracted from each of the surveillance images periodically acquired by each of the plurality of surveillance cameras, with respect to the specific image. Based on the plurality of threshold values stored in the threshold storage unit, the tracking target person is selected from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras. a detection unit that detects an image similar to the image of the person and detects which of a plurality of attention stages the detected image of the similarity calculation target person is in, and the detection unit acquires information from the similarity calculation unit; a tracking output screen generation unit that generates a tracking output screen for presenting an image of the calculated target person who is similar to the image of the tracked target person detected from the similarity calculated based on the similarity level, together with a step display indicating the caution level; It is designed to have the following.

One aspect of the person tracking system according to the present invention includes one aspect of the person tracking device according to the present invention, the plurality of surveillance cameras, the surveillance terminal operated by the supervisor, and the similarity calculation section. An analysis section is provided.

One aspect of the person detection method according to the present invention is a person detection method for detecting a specific person from surveillance images acquired by a plurality of surveillance cameras, in which a computer uses one similarity calculation method to A plurality of threshold values for classifying the target image into a plurality of stages are stored in a memory based on the similarity calculated by a similarity calculation unit that calculates the similarity of the target image, and a plurality of surveillance cameras are stored. Obtaining, from the similarity calculation unit, the similarity of the image of the person to be calculated, which is the target image, extracted from each of the regularly acquired monitoring images, with respect to the image of the specific person, which is the specific image; Based on the plurality of threshold values stored in the memory, from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of monitoring cameras, images similar to the image of the specific person are selected. At the same time as detecting an image, it is also detected which of a plurality of caution levels the detected image of the person to be calculated for similarity is in, and the image similar to the detected image of the specific person is classified into the detected caution levels. An output screen is generated to be presented together with a stage display indicating the stage.

One aspect of the person tracking method according to the present invention is a person tracking method for tracking a person to be tracked from surveillance images acquired by a plurality of surveillance cameras, wherein a computer Specifying an image of a tracking target person to be tracked from among a plurality of similarity calculation target images similar to the image of the specific person presented on the output screen generated according to the one aspect along with the step display. , the tracking image is received from a monitoring terminal operated by a supervisor, the specified image is registered in the memory as a tracking image, and the tracking image is an image of the tracking target person who is to be tracked and registered in the memory. as the specific image, causing the similarity calculation unit to extract the similarity of the image of the person to be calculated using the similarity calculation method, and extracting the similarity from the monitoring images of each of the plurality of monitoring cameras. The degree of similarity of the similarity calculation target person to the image of the tracking target person, which is the specific image, is obtained from the similarity calculation unit, and based on the plurality of threshold values stored in the memory, Detecting an image of the similarity calculation target person that is similar to the image of the tracking target person from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras; Detecting which of the plurality of attention stages the image of the person to be tracked is in, and calculating the similarity to be similar to the image of the person to be tracked detected based on the similarity obtained from the similarity calculation unit. A tracking output screen is generated that presents an image of the subject together with a stage display indicating the detected caution stage.

Alternatively, one aspect of the person tracking method according to the present invention is a person tracking method for tracking a person to be tracked from surveillance images obtained by a plurality of surveillance cameras, in which a computer performs one similarity calculation. storing in a memory a plurality of threshold values for classifying the target image into a plurality of stages based on each of the degrees of similarity calculated by a similarity calculation unit that calculates the degree of similarity of the target image to a specific image using a method; Then, the image of the person to be tracked specified by the monitoring terminal operated by the monitor is registered in the memory as the specific image, and the monitoring image is periodically acquired by each of the plurality of monitoring cameras. The degree of similarity of the image of the person to be calculated, which is the target image extracted from each of the target images, to the specific image is obtained from the similarity degree calculation unit, and based on the plurality of threshold values stored in the memory, From among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras, an image similar to the image of the tracking target person is detected, and the detected similarity calculation target detecting which of a plurality of attention stages the image of the person is in, and detecting an image of the target person whose similarity is similar to the image of the target person detected from the similarity obtained from the similarity calculating unit; A tracking output screen is generated to be presented together with a step display indicating the caution step that has been taken.

One aspect of the person detection program according to the present invention is a program that causes a computer to operate as each part of one aspect of the person detection apparatus according to the present invention.

One aspect of the person tracking program according to the present invention is a program that causes a computer to operate as each part of one aspect of the person tracking device according to the present invention.

According to each aspect of the present invention, a supervisor can easily recognize the certainty of an authentication result.

FIG. 1 is a diagram showing an example of the overall configuration of a person tracking system according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the configuration of the video analysis function section shown in FIG. 1. FIG. 3 is a block diagram showing the software configuration of the Web server shown in FIG. 1 as a person detection device and a person tracking device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of the storage contents of the past search data storage section shown in FIG. 3. FIG. 5 is a diagram showing an example of the storage contents of the detection history data storage section shown in FIG. 3. FIG. 6 is a diagram showing an example of the storage contents of the management data storage section shown in FIG. 3. FIG. 7 is a diagram showing an example of the contents written in the camera information table storage section shown in FIG. 3. FIG. 8 is a diagram showing an example of the contents written in the past search data table storage section shown in FIG. 3. FIG. 9 is a diagram showing an example of the contents written in the detection history table storage section shown in FIG. 3. FIG. 10 is a diagram showing an example of the contents written in the management table storage section shown in FIG. 3. FIG. 11 is a diagram showing an example of the contents written in the tracking table storage section shown in FIG. 3. FIG. 12 is a block diagram showing the hardware configuration of the Web server in FIG. 3. FIG. 13 is a flowchart showing the processing procedure of monitoring processing by the monitoring/tracking execution function unit shown in FIG. FIG. 14 is a flowchart showing the processing procedure of the detection result judgment process by the detection/tracking result judgment function section shown in FIG. FIG. 15 is a flowchart showing the processing procedure of detection alert output processing by the detection alert output function section shown in FIG. FIG. 16 is a diagram showing an example of a detection alert screen displayed on the web browser of the monitoring terminal shown in FIG. FIG. 17 is a flowchart showing the processing procedure of image registration processing by the image registration function section shown in FIG. FIG. 18 is a flowchart showing the processing procedure of tracking processing by the monitoring/tracking execution function section shown in FIG. FIG. 19 is a flowchart showing the processing procedure of the tracking result judgment process by the detection/tracking result judgment function section shown in FIG. FIG. 20A is a flowchart showing a first part of the processing procedure of tracking alert output processing by the tracking alert output function unit shown in FIG. 3. FIG. FIG. 20B is a flowchart showing the second part of the processing procedure for tracking alert output processing by the tracking alert output function unit shown in FIG. 3. FIG. FIG. 20C is a flowchart showing the third part of the processing procedure for tracking alert output processing by the tracking alert output function unit shown in FIG. 3. FIG. FIG. 21 is a diagram showing an example of a tracking alert screen displayed on the web browser of the monitoring terminal shown in FIG. FIG. 22 is a diagram showing another example of the tracking alert screen displayed on the web browser of the monitoring terminal shown in FIG.

Embodiments of the present invention will be described below with reference to the drawings.

[One embodiment]
(1) Configuration (1-1) Overall Configuration FIG. 1 is a diagram showing an example of the overall configuration of a person tracking system 1 according to an embodiment of the present invention.

The person tracking system 1 is a system that tracks a specific person in a large facility with a wide monitoring range, such as a building with multiple floors such as an office building or a department store, or a commercial facility with multiple stores.

This person tracking system 1 functions as a plurality of surveillance cameras 10, a video analysis function section 20 provided corresponding to each surveillance camera 10, and a person detection device and a person tracking device according to an embodiment of the present invention. It includes a server device SV including a Web server 30 and a monitoring terminal 40. Each surveillance camera 10, server device SV, and surveillance terminal 40 are connected via a network NET.

The network NET is a local area network, such as a wireless LAN (Local Area Network) or a wired LAN. The network NET may be a wide area network such as the Internet.

The plurality of surveillance cameras 10 are network cameras that are distributed within a large facility so that their photographic ranges do not overlap or partially overlap. The surveillance camera 10 may be a video camera that photographs a moving image within a photographing range, and can acquire each frame image forming the moving image as a monitoring image of the photographing range. Furthermore, the surveillance camera 10 may be a still camera that periodically acquires surveillance images by photographing still images at fixed time intervals.

Each of the plurality of video analysis functional units 20 arranged in the server device SV has various analysis functions for monitoring images acquired from the corresponding monitoring camera 10. Details of the analysis function provided by this video analysis function section 20 will be described later. Note that the video analysis function unit 20 is not placed inside the server device SV in this way, but is placed independently in a dedicated computing device, cloud, etc., and is connected to the corresponding surveillance camera 10 via the network NET. It is also possible to exchange data with the server device SV. Further, the video analysis function unit 20 can be placed inside the surveillance camera 10 if it has a calculation function. In this embodiment, the surveillance camera 10 and the video analysis function section 20 are configured in a one-to-one correspondence, but one video analysis function section 20 may be assigned to a plurality of surveillance cameras 10. Also good.

The Web server 30 of the server device SV is constituted by a server computer, and includes a file server 31, a database (hereinafter abbreviated as DB) server 32, a Web application 35, a search person detection result storage function section 33, and a function section. , a detection/tracking result determination function section 34. Details of each of these functional units will be described later.

The monitoring terminal 40 is configured by a computer such as a personal computer, and can execute a program of a web browser 41. Using the web browser 41, a supervisor such as a security guard who operates the monitoring terminal 40 can view the person tracking results provided by the web application 35 of the web server 30. Although FIG. 1 shows only one monitoring terminal 40, the person tracking system 1 may include a plurality of monitoring terminals 40.

Further, the server device SV may have the function of the monitoring terminal 40.

(1-2) Video analysis function section 20
FIG. 2 is a block diagram showing an example of the configuration of the video analysis function section 20 shown in FIG. 1. As shown in FIG. The video analysis function section 20 includes an image acquisition module 21, a human detection information extraction function section 22, and a monitoring/tracking execution function section 23.

The image acquisition module 21 acquires frame images transmitted from the surveillance camera 10 via the network NET according to a predetermined communication protocol such as RTSP (Real Time Streaming Protocol). The image acquisition module 21 transmits the acquired frame image to the Web server 30 via intra-server communication, Websocket, etc., and causes the file server 31 to accumulate it. Note that when the video analysis function section 20 is arranged independently from the server device SV, the image acquisition module 21 transmits the acquired frame image to the Web server 30 via the network NET. Further, the image acquisition module 21 sends the acquired frame image to the human detection information extraction function section 22.

The human detection information extraction function unit 22 extracts a full-body image of a person from the input image, extracts a whole-body feature amount and a face image of the person from this full-body image, and further extracts the features of the person from this face image. It functions as an extraction unit that extracts facial feature quantities. Specifically, the human detection information extraction function section 22 includes a whole body detection module 221 , a region tracking module 222 , a whole body feature amount extraction module 223 , a face detection module 224 , and a face feature amount extraction module 225 .

The whole body detection module 221 is a whole body image extraction unit that detects the whole body of each person appearing in a frame image that is an input image and extracts a whole body image. The whole body detection module 221 can detect the whole body of a person, for example, based on a whole body image of the person trained in advance by machine learning such as deep learning. The whole body detection module 221 transmits the extracted whole body image to the Web server 30 via intra-server communication, Websocket, etc., and stores it in the file server 31. Note that when the video analysis function section 20 is arranged independently from the server device SV, the whole body detection module 221 transmits the extracted whole body image to the Web server 30 via the network NET. The whole body detection module 221 also sends the extracted whole body image to the whole body feature extraction module 223 and the face detection module 224. Furthermore, the whole body detection module 221 sends the frame image and the extracted whole body image to the region tracking module 222 .

The region tracking module 222 detects the region position of the whole body image in the frame image input from the whole body detection module 221.

The whole body feature extraction module 223 is a whole body feature extraction unit that extracts whole body features from the input whole body image. The whole body feature quantity is an objective numerical representation of the body features of the person included in the whole body image. For example, the whole-body feature amount can be a numerical representation of attribute information about the person, such as body shape, height, gender, age, etc., estimated from the image. The whole body feature extraction module 223 can extract whole body features using a model learned in advance such as deep learning (for example, deep-person-reid). Further, the whole-body feature extraction module 223 can also calculate these whole-body features as local image features such as HOG (Histogram of Oriented Gradients) and SIFT (Scaled Invariance Feature Transform). The whole body feature amount extraction module 223 transmits the extracted whole body feature amount to the Web server 30 via intra-server communication, Websocket, etc., and stores it in the file server 31. Note that when the video analysis function unit 20 is arranged independently from the server device SV, the whole body feature amount extraction module 223 transmits the extracted whole body feature amount to the Web server 30 via the network NET. Furthermore, the whole body feature amount extraction module 223 sends the extracted whole body feature amount to the monitoring/tracking execution function unit 23.

The face detection module 224 is a face image extraction unit that detects a person's face from the input whole body image and extracts a face image. The face detection module 224 can detect a person's face, for example, based on a person's face image trained in advance by machine learning such as deep learning. The face detection module 224 sends the extracted face image to the face feature amount extraction module 225.

The facial feature extraction module 225 is a facial feature extraction unit that extracts facial features from the input facial image. The facial feature amount is an objective numerical representation of the facial features of a person included in the facial image. For example, similar to the whole body feature extraction module 223 that extracts whole body features, the facial feature extraction module 225 can extract facial features using a model learned in advance by deep learning or the like. Further, the face detection module 224 may calculate these facial feature amounts as local image feature amounts such as HOG or SIFT. The face detection module 224 transmits the extracted facial feature amount to the Web server 30 via intra-server communication, Websocket, etc., and stores it in the file server 31. Note that when the video analysis function unit 20 is arranged independently from the server device SV, the facial feature amount extraction module 225 transmits the extracted facial feature amount to the Web server 30 via the network NET. Further, the face detection module 224 sends the extracted facial feature amount to the monitoring/tracking execution function unit 23.

In addition, the monitoring/tracking execution function unit 23 calculates whole body image similarity, which is the similarity between the whole body features of the two input images, and facial image similarity, which is the similarity between the facial features of the two input images. functions as a similarity calculation unit. Specifically, the monitoring/tracking execution function unit 23 includes a whole body matching module 231 and a face matching module 232.

The whole body verification module 231 calculates the degree of similarity between the whole body feature input from the whole body feature extraction module 223 and the whole body feature of the person to be tracked instructed by the monitoring terminal 40 . For example, the whole body feature amount of the person to be tracked is extracted in advance by the whole body feature amount extraction module 223 and registered in the file server 31 of the Web server 30. The whole body verification module 231 transmits the calculated degree of similarity to the web server 30 via intra-server communication, websocket, or the like. Note that when the video analysis function unit 20 is arranged independently from the server device SV, the whole body verification module 231 transmits the calculated similarity to the Web server 30 via the network NET.

The face matching module 232 calculates the degree of similarity between the facial feature amount input from the facial feature amount extraction module 225 and the facial feature amount of the person to be tracked instructed by the monitoring terminal 40 . The facial feature amount of the person to be tracked is, for example, extracted in advance by the facial feature amount extraction module 225 and registered in the file server 31 of the Web server 30. The face matching module 232 transmits the calculated similarity to the web server 30 via intra-server communication, websocket, or the like. Note that when the video analysis function unit 20 is arranged independently from the server device SV, the face matching module 232 transmits the calculated similarity to the Web server 30 via the network NET.

(1-3) Web server 30
FIG. 3 is a block diagram showing the software configuration of the Web server 30. As described above, the Web server 30 includes a file server 31, a DB server 32, a search person detection result storage function section 33, a detection/tracking result determination function section 34, and a Web application 35 as functional sections.

The file server 31 stores various data files. The file server 31 can include a past search data storage section 311, a detection history data storage section 312, and a management data storage section 313.

The past search data storage unit 311 stores past search data that is data acquired by each of the plurality of video analysis function units 20. FIG. 4 is a diagram showing an example of past search data stored in this past search data storage section 311. The past search data can be stored in the past search data storage section 311 for each whole body image detected from the frame image by the whole body detection module 221 of each video analysis function section 20. The past search data includes a frame image 3111 acquired by the image acquisition module 21 of the video analysis function unit 20, a whole body image 3112 extracted by the whole body detection module 221, a whole body feature 3113 extracted by the whole body feature extraction module 223, and The facial feature amount 3114 extracted by the facial feature amount extraction module 225 can be included. These frame images 3111, whole body images 3112, whole body features 3113, and face features 3114 are linked by, for example, storing them in the same path or adding the same character string to part of the file name. be able to.

The detection history data storage unit 312 stores detection history data that is data regarding each person detected as a specific monitoring target, such as a suspicious person, a lost child, or a regular customer. FIG. 5 is a diagram showing an example of detection history data of a detected person stored in the detection history data storage section 312. The detection history data can include a detected face image 3121 that is a face image of a detected person, and a facial feature amount 3122 corresponding to the detected face image 3121. The detected face image 3121 and the facial feature amount 3122 can be linked by, for example, being stored in the same path or by adding the same character string to part of the file name.

Note that, although not particularly shown, data regarding the person to be monitored is stored in the file server 31 in advance. The data regarding the person to be monitored can be stored in the file server 31 by, for example, inputting the face image of the person to be monitored from the monitoring terminal 40 to the Web server 30 via the network NET. Then, the Web server 30 inputs the facial image to the facial feature extraction module 225 of one video analysis function unit 20 through intra-server communication, Websocket, etc., and extracts the facial feature amount from the facial feature extraction module 225. It can be acquired and stored in the file server 31. In addition, the whole body images 3112 of a plurality of people stored in the past search data storage unit 311 are viewed on the web browser 41 of the monitoring terminal 40 via the network NET, and any full body image 3112 is displayed from the monitoring terminal 40. By specifying this, it is also possible to designate that person as a person to be monitored. In this case, the Web server 30 inputs the designated full-body image 3112 to the face detection module 224 of one video analysis function section 20 through intra-server communication, Websocket, etc., and extracts the face image from the face detection module 224. It can be acquired and stored in the file server 31 as a face image of the person to be monitored. Further, the web server 30 acquires the facial feature amount 3114 from the past search data having the specified full body image 3112 stored in the past search data storage unit 311, and stores it in the file server 31. be able to.

Further, when data regarding a person to be monitored is stored in the file server 31, a table of persons to be monitored containing information about the person to be monitored is also stored in the DB server 32. This monitoring target person table includes the suspicious person ID, which is identification information that uniquely identifies the monitoring target, the suspicious person name, which is the name of the monitoring target, and the risk that indicates the reason for the monitoring target. It is possible to record information such as degree type, physical characteristics and behavior patterns of the person to be monitored, and explanatory notes such as precautions.

The detected face image 3121 in the detection history data is stored in the detection history data storage unit when the web server 30 detects a person to be monitored from a surveillance image of any of the surveillance cameras 10 by comparing it with data regarding the person to be monitored. 312. That is, the Web server 30 sends the whole body image acquired from the monitoring image and stored in the past search data storage section 311 to the face detection module 224 of one video analysis function section 20 through intra-server communication, Websocket, etc. By inputting the information, a face image can be acquired from the face detection module 224 and stored in the detection history data storage unit 312 as the detected face image 3121. This one video analysis function section 20 may be a video analysis function section 20 corresponding to the surveillance camera 10 that acquired the surveillance image in which the person to be monitored was detected. If the face detection module 224 of the video analysis function unit 20 is buffering the extracted face image, the web server 30 detects the face image by requesting the face detection module 224 for the buffered face image. A face image 3121 may also be acquired.

The management data storage unit 313 stores management data that is data regarding each tracked person. FIG. 6 is a diagram showing an example of management data for each person to be tracked stored in the management data storage unit 313. Management data for each person to be tracked can be stored in the management data storage unit 313 according to a designation from the monitoring terminal 40. The management data includes a registered face image 3131 which is a face image of a designated person to be tracked, a facial feature amount 3132 which is a feature extracted from the registered face image 3131, and a registered whole body image which is a whole body image of the person to be tracked. 3133 and a whole body feature amount 3134 that is a feature extracted from the registered whole body image 3133. These registered face image 3131, facial feature amount 3132, registered whole body image 3133, and whole body feature amount 3134 can be stored in the same path, or by adding the same character string to part of the file name, for example. Can be linked.

For example, when a person to be monitored is detected, the web server 30 sends the detected face image 3121 in the detection history data for the monitored person stored in the detection history data storage unit 312 to the person being monitored using the web browser 41 of the monitoring terminal 40. to view it. Then, when the monitoring terminal 40 specifies that the person is to be tracked, the Web server 30 stores the detected face image 3121 in the detection history data as the registered face image 3131 in this management data. Furthermore, the Web server 30 can store the facial feature amount 3122 in the detection history data as the facial feature amount 3132 in this management data. Further, the web server 30 acquires the whole body image that is the source of the detected face image 3121 in the specified detection history data from the past search data stored in the past search data storage unit 311, and also acquires the whole body feature amount. are also acquired from the past search data and stored as the registered whole body image 3133 and whole body feature amount 3134 in this management data.

Alternatively, the web server 30 allows the web browser 41 of the monitoring terminal 40 to view the full-body images 3112 of a plurality of people stored in the past search data storage unit 311, and allows the monitoring terminal 40 to specify an arbitrary full-body image. In response to this, the person may be designated as a designated person to be tracked. In this case, the Web server 30 inputs the designated full-body image 3112 to the face detection module 224 of one video analysis function section 20 through intra-server communication, Websocket, etc., and extracts the face image from the face detection module 224. It can be acquired and stored in the management data storage unit 313 as a registered face image 3131. Further, in this case, as for the face feature amount 3132, the registered whole body image 3133, and the whole body feature amount 3134, those stored in the past search data storage unit 311 may be acquired and stored. Alternatively, the face image and full body image of the person to be tracked can also be registered in the management data storage unit 313 by inputting them from the monitoring terminal 40 to the web server 30 via the network NET, in the same way as the face image of the person to be monitored. They may be stored as the image 3131 and the registered whole body image 3133. In this case, the web server 30 may send the input face image and full body image to the video analysis function unit 20 via intra-server communication, Websocket, etc., and obtain the facial feature amount 3132 and the whole body feature amount 3134. .

The DB server 32 stores various data tables. The DB server 32 can include a camera information table storage section 321, a past search data table storage section 322, a detection history table storage section 323, a management table storage section 324, a tracking table storage section 325, and the like.

The camera information table storage unit 321 is provided corresponding to each of the plurality of surveillance cameras 10, and stores in advance a camera information table 3211 in which various information regarding the corresponding surveillance camera 10 is described. FIG. 7 is a diagram showing an example of the contents of the camera information table 3211 for each surveillance camera 10, which is stored in the camera information table storage section 321. The camera information table 3211 includes, for example, camera ID, camera name, aspect ratio, camera position X, camera position Y, camera angle, face matching thresholds 1 to 3, whole body matching thresholds 1 to 3, and the like.

Here, the camera ID is identification information that uniquely identifies the corresponding surveillance camera 10. The camera name is the name of the corresponding surveillance camera 10. This can be a name associated with the installation position of the corresponding surveillance camera 10 in a large facility. The aspect ratio is the aspect ratio of the image acquired by the corresponding surveillance camera 10. Camera position X and camera position Y are XY coordinates within a large facility that indicate the installation position of the corresponding surveillance camera 10. The camera angle indicates the installation direction of the corresponding surveillance camera 10.

Further, the face matching thresholds 1 to 3 and the whole body matching thresholds 1 to 3 are thresholds for comparison with the degrees of similarity calculated by the whole body matching module 231 and the face matching module 232 of the video analysis function section 20. The face matching threshold 1 is a threshold value corresponding to face image similarity for classifying whether a face image of a person extracted from a surveillance image of the corresponding surveillance camera 10 is a face image of a person to be monitored or a person to be tracked. 1 threshold. If the face image similarity calculated by the face matching module 232 is larger than the first threshold, ie, the face matching threshold 1, the web server 30 determines whether the face image of the person extracted from the monitoring image of the corresponding monitoring camera 10 is It can be determined that the face image is of a person to be monitored or a person to be tracked. Face matching thresholds 2 and 3 correspond to face image similarity greater than face matching threshold 1, and when the extracted face image of a person is a face image of a person to be monitored or a person to be tracked, the face image is This is the second threshold for classifying into multiple stages. That is, the magnitudes of these thresholds have a relationship of [face matching threshold 1<face matching threshold 2<face matching threshold 3]. Similarly, the whole body matching threshold 1 is the whole body image similarity for classifying whether the whole body image of the person extracted from the surveillance image of the corresponding surveillance camera 10 is the whole body image of the person to be monitored or the person to be tracked. is the corresponding first threshold. If the whole body image similarity calculated by the whole body matching module 231 is larger than the whole body matching threshold 1, the web server 30 determines whether the whole body image of the person extracted from the monitoring image of the corresponding monitoring camera 10 is a monitoring target or a tracking target. It can be determined that the image is a full-body image of a person. Whole body matching thresholds 2 and 3 correspond to a whole body image similarity greater than whole body matching threshold 1, and when the extracted whole body image of a person is a whole body image of a person to be monitored or a person to be tracked, the whole body image is This is the second threshold for classifying into multiple stages. That is, the magnitudes of these threshold values have a relationship of [whole body verification threshold 1<whole body verification threshold 2<whole body verification threshold 3].

The value of each threshold is generally determined by the matching model used by the face matching module 232 and the matching model used by the whole body matching module 231 in the monitoring/tracking execution function section 23 of the video analysis function section 20 that calculates the degree of similarity. Each threshold value depends on the type, performance, installation conditions, etc. of the surveillance camera 10 itself, so the threshold value is adjusted to an appropriate value between the time the surveillance camera 10 is installed and the actual operation begins. It is written in the information table 3211. Note that the face matching thresholds 1 to 3 and the whole body matching thresholds 1 to 3 may be separately recorded in the camera information table 3211 as different values for the monitoring target person and the tracking target person. Moreover, it goes without saying that each threshold value is not limited to three levels. That is, at least one second threshold value is sufficient.

The past search data table storage unit 322 includes a past search data table 3221 in which various data related to the past search data are described, corresponding to each of the past search data stored in the past search data storage unit 311. Remember. When storing the past search data in the past search data storage section 311, the Web server 30 creates this past search data table 3221 and stores it in the past search data table storage section 322. FIG. 8 is a diagram showing an example of the contents of the past search data table 3221 stored in the past search data table storage section 322. The past search data table 3221 includes, for example, detection ID, detection date and time, detection camera ID, tracking ID, frame ID, detection coordinate information X, detection coordinate information Y, detection coordinate information Width, detection coordinate information Height, etc. be done.

Here, the detection ID is identification information assigned to each whole body image 3112 in the frame image 3111 detected by the whole body detection module 221 of the video analysis function unit 20. The detection date and time is the date and time when the whole body detection module 221 detected the whole body image 3112. The detected camera ID is the camera ID of the surveillance camera 10 that acquired the frame image 3111. The tracking ID is identification information for linking the whole body images 3112 of the same person among multiple frame images. The frame ID is identification information for uniquely identifying the frame image 3111. Detection coordinate information For example, it shows the XY coordinates of the upper left corner in the frame image, and the image width and image height from there.

Note that the Web server 30 can assign the tracking ID of the past search data table 3221 in the following manner, for example. The web server 30 uses the region position of the whole body image provided from the human detection information extraction function unit 22 of the video analysis function unit 20 and the previous frame image of the surveillance camera 10 stored in the past search data table storage unit 322. The identity of the person in the whole-body image is determined by comparing the area positions of each of the whole-body images in the past search data table 3221 and taking into consideration overlapping areas and the like. If the same person is not shown in the previous frame image, the web server 30 assigns a new tracking ID. Furthermore, if the same person appears in the previous frame image, the tracking ID is taken over from the past search data table 3221 corresponding to the whole body image 3112 of the person. Note that the Web server 30 may determine the identity of the person by comparing the whole-body feature amount instead of the position of the whole-body image, and determine whether or not to provide a new tracking ID.

The detection history table storage unit 323 stores a detection history table 3231 in which various data related to the detection history data are described, corresponding to each piece of detection history data stored in the detection history data storage unit 312. When storing detection history data in the detection history data storage section 312, the Web server 30 creates this detection history table 3231 and stores it in the detection history table storage section 323. FIG. 9 is a diagram showing an example of the contents of the detection history table 3231 stored in the detection history table storage section 323. The detection history table 3231 includes, for example, a detection ID, detection date and time, detection camera ID, suspicious person ID, face authentication score, face left/right angle, face up/down angle, tracking ID, and the like.

Here, the detection ID, detection date and time, detection camera ID, and tracking ID are as described for the past search data table 3221. The suspicious person ID is the suspicious person ID of the detected person to be monitored, which is listed in a table of persons to be monitored (not shown). The face recognition score is the degree of similarity with the face image of the person to be monitored, which is calculated by the face matching module 232 of the video analysis function unit 20. This degree of similarity is stored as a real value of the degree of similarity calculated by the face matching module 232, but as described later, in order to display a detection alert screen on the web browser 41, a three-level face image similarity label ID is stored. can be rewritten as . The face left/right angle and the face up/down angle indicate the direction of the face in the detected face image 3121 of the corresponding detection history data.

The management table storage unit 324 stores, corresponding to each piece of management data stored in the management data storage unit 313, a management table 3241 in which various data related to the management data are described. When storing management data in the management data storage section 313, the Web server 30 creates this management table 3241 and stores it in the management table storage section 324. FIG. 10 is a diagram showing an example of the contents of the management table 3241 stored in the management table storage section 324. The management table 3241 includes, for example, a suspicious person ID, suspicious person name, risk type, explanatory text, query face image path, facial feature file path, monitoring status, pinning flag, long stay flag, etc. Ru.

Here, as the suspicious person ID, suspicious person name, risk type, and explanatory text, information about the person to be monitored who is specified as the person to be tracked, which is written in the table of persons to be monitored (not shown), is transferred. The query face image path and face feature amount file path indicate the storage paths of the registered face image 3131 and face feature amount 3132 of management data in the management data storage unit 313. The monitoring status is attribute information other than the risk type that is arbitrarily set for the person to be tracked from the monitoring terminal 40. The pinning flag is a flag that is set to indicate when a person is designated as a tracking target. The long-term stay flag is a flag that is set when a person is not designated as a tracking target but stays at the facility for a specified period of time or longer. That is, when there is a person who is not a monitoring target but has been staying for a long time, the Web server 30 also stores the face image and facial feature amount of the person in the management data storage unit 313, and stores the face image and facial feature amount of the person in the management table storage unit 324. Management tables can be stored. This makes it possible to discover new suspicious persons and new customer candidates.

The tracking table storage unit 325 stores the whole body image or face image of the person to be tracked by the web server 30 based on the whole body image similarity or face image similarity calculated by the whole body matching module 231 or the face matching module 232 of the video analysis function unit 20. A tracking table is stored in which the degree of similarity of the person extracted from the monitoring image of any of the monitoring cameras 10 determined to be described is described. FIG. 11 is a diagram showing an example of the contents of the tracking table 3251 stored in the tracking table storage section 325. The tracking table 3251 includes a record consisting of a detection ID, a whole body image similarity degree, and a face image similarity degree for each suspicious person ID of a person to be tracked, and each record is a record that the web server 30 uses as a whole body image or a face image of the person to be tracked. It is added each time it is determined that it exists. For the whole body image similarity degree and the face image similarity degree, the real values of the similarity calculated by the whole body matching module 231 and the face matching module 232 of the video analysis function unit 20 are stored, but as will be described later, tracking with the Web browser 41 is performed. In order to display the alert screen, it can be rewritten into a three-level whole body image similarity label ID and a three-level facial image similarity label ID.

The search person detection result storage function section 33 and the detection/tracking result determination function section 34 are provided as back-end functions of the Web server. The search person detection result storage function section 33 receives the output from each function section of the person detection information extraction function section 22 of the video analysis function section 20 and creates a search data table to be stored in the past search data table storage section 322. do. The detection/tracking result determination function unit 34 receives the output from each function unit of the monitoring/tracking execution function unit 23 of the video analysis function unit 20, and determines whether the same person as the face image of the person to be monitored is stored in advance. A detection history table is created to be detected in the surveillance image of any of the surveillance cameras 10 and stored in the detection history table storage section 323, and extracted by each functional section of the human detection information extraction function section 22. The detected face image 3121 and facial feature amount 3122 of the person are stored in the detection history data storage section 312. The detection/tracking result determination function unit 34 also receives the output of each functional unit of the monitoring/tracking execution function unit 23 of the video analysis function unit 20 and determines the whole body image and/or the tracking target person specified by the monitor. It is detected that the same person as the face image appears in the monitoring image of any of the monitoring cameras 10, and in response to the detection, a record is created in the tracking table of the tracking target person stored in the tracking table storage unit 325. will be added.

The web application 35 is an application program that performs various processes in response to requests from the web browser 41 of the monitoring terminal 40 and creates data for viewing indicating the processing results. The Web application 35 provides the Web server 30 with functions as a login function section 351, a monitoring function section 352, a tracking function section 353, and a past search function section 354.

The login function unit 351 receives a login from the Web browser 41, performs authentication, and permits the authorized user to use the functions provided by the Web application 35.

When the detection/tracking result determination function unit 34 detects a person identical to the facial image of the person being monitored, the monitoring function unit 352 generates an alert and presents it on the web browser 41. To this end, the monitoring function unit 352 includes a detection alert output function unit 3521 that generates a detection alert screen to be presented on the web browser 41. The detection alert screen is an output screen for presenting an image of a person detected to be similar to the person being monitored, together with a stage display showing a warning stage based on the degree of similarity.

The tracking function unit 353 tracked the person in real time across the plurality of surveillance cameras 10 using the same face image and whole body image of the person as the person to be monitored as the face image detected by the detection/tracking result determination function unit 34. The tracking results are presented to the web browser 41. To this end, the tracking function section 353 includes an image registration function section 3531 and a tracking alert output function section 3532. The image registration function unit 3531 receives a designation of a face image of a person to be tracked to be tracked through a designation operation from a supervisor on the web browser 41, and stores a registered face image 3131 and facial feature amounts in the management data storage unit 313. 3132 is stored, and at the same time, a management table 3241 is created and stored in the management table storage section 324. The tracking alert output function unit 3532 outputs an image of a person detected to be similar to the tracked person when the detection/tracking result determination function unit 34 detects a person who is the same as the whole body image and/or face image of the tracked person. A tracking alert screen is presented on the Web browser 41, which is an output screen for presenting the following along with a step display indicating the caution step based on the degree of similarity.

The past search function section 354 receives a time and place specification from the Web browser 41, displays a list of people who visited the specified time and place, and accepts the specification of a person to be searched from among them. Then, the past search function unit 354 searches for a similar person stored in the past search data storage unit 311 from the face image and full body image of the specified person by the detection/tracking result determination function unit 34, and searches the result on the Web. It is presented on the browser 41.

FIG. 12 is a block diagram showing the hardware configuration of the Web server 30. The web server 30 includes, for example, a hardware processor 301A such as a CPU (Central Processing Unit), a program memory 301B, a storage device 302, and a communication interface device 303. Program memory 301B, storage device 302 and communication interface device 303 are connected to hardware processor 301A via bus 304.

The communication interface device 303 includes, for example, one or more wired or wireless communication interface units, and transmits and receives various information between the video analysis function unit 20 and the monitoring terminal 40 according to the communication protocol used in the network NET. enable.

The program memory 301B includes, as a storage medium, a non-volatile memory that can be written to and read from at any time such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive), and a non-volatile memory such as a ROM (Read Only Memory). It stores programs necessary for executing various control processes according to an embodiment of the present invention by being executed by a hardware processor 301A such as a CPU. That is, the hardware processor 301A reads and executes the program stored in the program memory 301B, thereby functioning as the search person detection result storage function unit 33 and the detection/tracking result determination function unit 34 as shown in FIG. be able to. Note that these processing functional units may be realized by separate hardware processors. That is, the Web server 30 may include multiple hardware processors. In addition, at least a portion of these processing functional units may include various other hardware circuits, including integrated circuits such as ASICs (Application Specific Integrated Circuits), FPGAs (Field-programmable Gate Arrays), and GPUs (Graphics Processing Units). It may be realized in the form of Furthermore, the programs stored in the program memory 301B can include a program of a web application 35 as shown in FIG.

The storage device 302 is a storage medium that uses a combination of a non-volatile memory such as an HDD or an SSD that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory). It is used to store various data acquired and created during processing. Furthermore, the storage device 302 can be configured with a file server 31 and a DB server 32 as shown in FIG.

(2) Operation Next, the operation of the person tracking system configured as above will be explained.

(2-1) Monitoring operation When the web browser 41 is started on the monitoring terminal 40 and an access operation is performed to the web server 30, the web server 30 performs an authentication operation using the login function section 351 provided by the web application 35. do. If it is confirmed that the user is an authorized user, the web server 30 uses the monitoring function unit 352 provided by the web application 35 to generate a monitoring screen for viewing on the web browser 41 and sends the screen to the web browser 41. Send. A monitoring screen can be generated as follows.

Each of the plurality of surveillance cameras 10 periodically acquires surveillance images, and inputs the acquired surveillance images to the corresponding video analysis function section 20. The image acquisition module 21 of the video analysis function section 20 acquires a frame image which is a surveillance image from the corresponding surveillance camera 10, and the human detection information extraction function section 22 acquires a whole body image of the person shown in the frame image, The region position, whole body feature amount, and facial feature amount of the whole body image are extracted. Then, the video analysis function unit 20 transmits the frame image, the whole body image, the region position of the whole body image, the whole body feature amount, and the face feature amount to the Web server 30 via intra-server communication, Websocket, or the like.

The web server 30 stores the frame image, whole body image, whole body feature amount, and face feature amount transmitted from each of the plurality of video analysis function units 20 as past search data in the past search data storage unit of the file server 31. Accumulate in 311. At this time, the search person detection result storage function section 33 of the Web server 30 creates a past search data table including the region positions of the whole body images transmitted from each of the plurality of video analysis function sections 20, and stores the data table in the DB. It is stored in the past search data table storage section 322 of the server 32. In this way, each time a surveillance image is acquired by each surveillance camera 10, past search data is accumulated in the past search data storage section 311, and linked to the past search data in the past search data table storage section 322. Past search data tables are accumulated.

The monitoring function unit 352 can generate a monitoring screen by arranging the frame images from each monitoring camera 10 accumulated in the past search data storage unit 311 on one screen. By viewing this monitoring screen displayed on the web browser 41 of the monitoring terminal 40, a supervisor such as a security guard can grasp the real-time situation of each part within the facility.

In addition, in the person tracking system, in parallel with the real-time situation monitoring of each part, the operation of detecting the arrival of the person to be monitored is performed. This monitoring target person visit detection operation includes a monitoring process by the monitoring/tracking execution function unit 23 of each video analysis function unit 20, a detection result judgment process by the detection/tracking result judgment function unit 34, and a detection alert by the monitoring function unit 352. Detection alert output processing by the output function unit 3521.

FIG. 13 is a flowchart showing the processing procedure of the monitoring process by the monitoring/tracking execution function unit 23. As shown in FIG. The monitoring/tracking execution function unit 23 executes the process shown in this flowchart each time an update of a person to be monitored is reported from the monitoring function unit 352 of the Web server 30 through, for example, intra-server communication or Websocket.

The monitoring/tracking execution function unit 23 first accesses the file server 31 of the web server 30 through intra-server communication, Websocket, etc., and obtains the facial feature values of each of the multiple monitoring subjects stored in advance in the file server 31. Obtain the facial feature amount for monitoring (step S101).

Thereafter, the monitoring/tracking execution function unit 23 acquires the face image at time t and its feature amount from the human detection information extraction function unit 22 as a matching face image and matching face feature amount (step S102). When the monitoring images of the corresponding monitoring cameras 10 include full-body images of multiple persons, the monitoring/tracking execution function unit 23 acquires a matching facial image and matching facial feature amounts for each person.

Then, the monitoring/tracking execution function unit 23 uses the face matching module 232 to determine the face image similarity, which is the degree of similarity with the monitoring facial feature amount, for each of the matching facial features from the human detection information extraction function unit 22. is calculated (step S103).

Thereafter, the monitoring/tracking execution function unit 23 transmits the set of the calculated face image similarity of each person to be monitored and the matching face image to the detection/tracking result determination function of the web server 30 via intra-server communication, Websocket, etc. 34 (step S104).

Then, after adding the time interval t1 to the time t, that is, updating to the next processing time (step S105), the monitoring/tracking execution function unit 23 repeats the above processing from step S102. Here, the time interval t1 can be any arbitrary time interval, for example, several seconds. Alternatively, the time interval t1 may be a time interval that is an integral multiple of the acquisition interval of surveillance images of the surveillance camera 10.

FIG. 14 is a flowchart showing the processing procedure of detection result determination processing by the detection/tracking result determination function unit 34 of the Web server 30. The detection/tracking result determination function unit 34 can periodically execute the process shown in this flowchart, for example, at every time interval t1 in the monitoring/tracking execution function unit 23 of the video analysis function unit 20.

The detection/tracking result determination function unit 34 first determines whether or not a set of similarity and matching face images has been received from the monitoring/tracking execution function unit 23 (step S201). If it is determined that the set of similarity and matching images has not been received, the detection/tracking result determination function unit 34 ends the detection/tracking result determination process.

On the other hand, if it is determined that the set of similarity and matching face images has been received, the detection/tracking result determination function unit 34 acquires the face matching threshold 1 (step S202). That is, the detection/tracking result determination function unit 34 selects the surveillance camera 10 corresponding to the video analysis function unit 20 that is the source of the set of similarity and matching face images stored in the camera information table storage unit 321. Face matching threshold 1 is acquired from the camera information table 3211 for .

Then, the detection/tracking result determination function unit 34 compares each of the received face image similarities with the acquired face matching threshold 1 and determines whether there is a face image similarity greater than the face matching threshold 1. (Step S203). If it is determined that there is no face image similarity greater than the face matching threshold 1, the detection/tracking result determination function unit 34 ends this detection/tracking result determination process.

On the other hand, if it is determined that there is a facial image similarity greater than the face matching threshold 1, the detection/tracking result determination function unit 34 determines that the person having the facial feature amount is the person to be monitored. do. Therefore, in this case, the detection/tracking result determination function section 34 causes the detection history data storage section 312 to store the detected face image 3121 and the facial feature amount 3122 as detection history data, and also generates the detection history table 3231. It is stored in the detection history table storage unit 323 (step S204). In the face authentication score of the generated detection history table 3231, a face image similarity having a value larger than the face matching threshold 1 will be written. After that, the detection/tracking result determination function section 34 ends this detection/tracking result determination process.

FIG. 15 is a flowchart showing the processing procedure of detection alert output processing by the detection alert output function unit 3521 of the monitoring function unit 352. The detection alert output function section 3521 periodically executes the process shown in this flowchart, for example, at every time interval t1 in the monitoring/tracking execution function section 23 of the video analysis function section 20.

The detection alert output function unit 3521 first obtains face matching thresholds 1 to 3 from the camera information table 3211 for each surveillance camera 10 stored in the camera information table storage unit 321 (step S301).

The detection alert output function unit 3521 also acquires information on a new detection history table 3231 from the detection history table storage unit 323 (step S302). That is, the detection alert output function unit 3521 extracts and obtains the face image score in the detection history table 3231 in which the real number of face similarities is recorded instead of the face image similarity label ID.

Then, the detection alert output function unit 3521 determines whether the face image similarity value described as the face authentication score in the acquired new detection history table 3231 is greater than the face matching threshold 3 (step S303). . If it is determined that the face image similarity value is larger than the face matching threshold 3, the detection alert output function unit 3521 converts the face image similarity value in the detection history table 3231 of the detection history table storage unit 323 from the real number. The face image similarity label ID, here, is rewritten to ID=3 (step S304).

On the other hand, if it is determined that the face image similarity value is not larger than the face matching threshold 3, the detection alert output function unit 3521 further determines whether the face image similarity value is larger than the face matching threshold 2. It is determined whether or not (step S305). If it is determined that the face image similarity value is larger than the face matching threshold 2, the detection alert output function unit 3521 converts the face image similarity value in the detection history table 3231 of the detection history table storage unit 323 from the real number. The face image similarity label ID, here, is rewritten to ID=2 (step S306).

Further, when it is determined that the value of the face image similarity is not larger than the face matching threshold 2, the detection alert output function unit 3521 outputs the face image similarity value in the detection history table 3231 of the detection history table storage unit 323. The value is rewritten from a real number to face image similarity label ID, here ID=1 (step S307). This is because, in the first place, the detection history table 3231 is not created unless the face image similarity value is greater than the face matching threshold 1, so if the face image similarity value is not greater than the face matching threshold 2, the face image similarity This is because it is determined that the value of degree is larger than the face matching threshold value 1.

After rewriting the face image similarity value in the detection history table 3231 to the face image similarity label ID as described above, the detection alert output function unit 3521 creates a new detected person card that is a screen image. (Step S308). This detected person card includes a detected face image 3121 stored as detection history data in the detection history data storage unit 312, and a caution level display indicating a caution level based on the face image similarity label ID in the corresponding detection history table 3231; This is a screen image containing. Details of this detected person card will be described later.

The detection alert output function unit 3521 updates the detection alert screen to be displayed on the Web browser 41 using the created new detected person card (step S309). Then, the detection alert output function unit 3521 transmits this updated detection alert screen to the Web browser 41 via the network NET, and displays it there (step S310). The web browser 41 displays this detection alert screen in an alert screen display area provided in a part of the monitoring screen, or by opening a window separate from the monitoring screen. Then, the detection alert output function unit 3521 ends this detection alert output processing.

FIG. 16 is a diagram showing an example of the detection alert screen 42 displayed on the Web browser 41. The detection alert screen 42 includes a detected person card 421 for each detected person. This detected person card 421 includes, as its display contents, a monitoring target person face image 4211, a detected face image 4212, a warning message 4213, a suspicious person ID 4214, a warning stage display 4215, detection information 4216, and a tracking start button 4217. be able to.

Here, the monitored person face image 4211 is a face image of the monitored person stored in advance in the file server 31, and the detected face image 4212 is the detected face image of the person detected to be the monitored person. This is a detected face image 3121 stored in the history data storage unit 312. The warning message 4213 is a message to call the monitor's attention, and may be displayed with a flashing indicator light for identification. The suspicious person ID 4214 is identification information transcribed from the detection history table 3231.

The caution stage display 4215 indicates the degree of similarity between the face image of the person to be monitored and the face image of the detected person. That is, the degree of similarity is shown not as a numerical value but as a warning level display 4215 of multiple levels, here three levels, based on the face image similarity degree label ID in the detection history table 3231. In this caution stage display 4215, the number of identified squares represented by hatching in FIG. 16 corresponds to the face image similarity label ID. For example, in the detected person card 421 of a person whose suspicious person ID 4214 is "00098842", the face image similarity label ID in the detection history table 3231 is "3", so in the caution stage display 4215, three rectangles are displayed. In the detected person card 421 of the person who is identified and whose suspicious person ID 4214 is "00059820," the face image similarity label ID in the detection history table 3231 is "2," so two rectangles are identified and displayed. There is.

The detection information 4216 includes detection camera location information based on the detection camera ID in the past search data table 3221 or the detection history table 3231, and detection date and time information also based on the detection date and time. The tracking start button 4217 is a button that is pressed when tracking the person.

Note that each time a new person is detected as a person to be monitored, the old detected person card 421 is shifted downward so that it is displayed at the top of the detection alert screen 42. A new detected person card 421 is not added for the same person. By selecting one detected person card 421, a list of the history of the person stored in the detection history data storage section 312 and the detection history table storage section 323 is displayed.

(2-2) Tracking operation When the monitor presses the tracking start button 4217 on the detection alert screen 42, the web server 30 uses the tracking function unit 353 provided by the web application 35 to perform the video analysis function unit 20 and the detection /Using the tracking result determination function section 34, the Web browser 41 is caused to present the tracking results of tracking the person. This tracking operation includes image registration processing by the image registration function unit 3531 of the tracking function unit 353, tracking processing by the monitoring/tracking execution function unit 23 of the video analysis function unit 20, and tracking results by the detection/tracking result determination function unit 34. This includes determination processing and tracking alert output processing by the tracking alert output function section 3532 of the tracking function section 353.

FIG. 17 is a flowchart showing the processing procedure of image registration processing by the image registration function section 3531 of the tracking function section 353. The image registration function unit 3531 periodically performs the processing shown in this flowchart, for example, in synchronization with the acquisition interval of monitoring images of the monitoring camera 10.

The image registration function unit 3531 first determines whether or not the monitor presses the tracking start button 4217 on the detection alert screen 42 displayed on the Web browser 41 on the monitoring terminal 40 (step S401). If it is determined that the tracking start button 4217 has not been pressed, the image registration function unit 3531 ends this image registration process.

On the other hand, if it is determined that the tracking start button 4217 has been pressed, the image registration function unit 3531 acquires the face image of the person as a tracking face image (step S402). That is, the image registration function unit 3531 acquires the detected face image 3121 as a tracking face image from the detection history data storage unit 312 based on the tracking ID of the detection history table storage unit 323 having the suspicious person ID of the detection alert screen 42. do. For example, the image registration function unit 3531 acquires the detected face image 3121 specified by the detection ID listed in the detection history table 3231 having the tracking ID, whose detection date and time are in a specific period, for example, the latest one. be able to. The specific period is not limited to the latest, but may be the oldest, the latest or oldest within a specified period, or the season (spring, summer, fall, winter, etc.). Alternatively, the image registration function unit 3531 generates a selection screen that includes the detected face images 3121 specified by the detection IDs listed in each of the detection history tables 3231 having the tracking ID in a list format, and displays the selection screen on the web browser 41. may be displayed so that the observer can select the detected face image 3121 to be used as the tracking face image. Then, the image registration function unit 3531 stores the acquired tracking face image in the management data storage unit 313 as a registered face image 3131. Further, the image registration function unit 3531 creates a management table 3241 based on information regarding the relevant monitoring target person described in a monitoring target person table (not shown) stored in advance in the DB server 32, and stores it in the management table 3241. The information is stored in the storage unit 324. The management table 3241 includes a query facial feature path indicating the storage path of the registered facial image 3131 in the management data storage unit 313.

Then, the image registration function unit 3531 obtains, as a tracking full-body image, a whole-body image corresponding to the detected face image 3121 obtained from the detection history data storage unit 312 as a tracking-use full-body image (step S403). That is, the image registration function unit 3531 tracks the whole body image 3112 from the past search data storage unit 311 based on the detection ID listed in the detection history table 3231 for the detected face image 3121 acquired as the tracking face image. Acquire as a full-body image. The image registration function unit 3531 stores this acquired tracking whole body image in the management data storage unit 313 as a registered whole body image 3133. Further, the image registration function unit 3531 adds a query whole body feature path indicating the storage path of the registered whole body image 3133 in the management data storage unit 313 to the management table 3241 stored in the management table storage unit 324.

Furthermore, the image registration function unit 3531 acquires a facial image feature amount for tracking and a whole body image feature amount for tracking (step S404). That is, the image registration function section 3531 acquires the facial feature amount 3122 from the detection history data storage section 312 as the tracking facial feature amount based on the detection ID, and also acquires the whole body feature amount from the past search data storage section 311. 3113 is acquired as the whole body feature amount for tracking. The image registration function unit 3531 stores the acquired tracking facial feature amount and tracking whole body feature amount in the management data storage unit 313 as a facial feature amount 3132 and a whole body feature amount 3134. Further, the image registration function unit 3531 adds the face feature file path and the whole body feature file path indicating the feature storage paths to the management table 3241 stored in the management table storage unit 324.

Then, the image registration function unit 3531 uses the registered face image 3131 and the registered whole body image 3133 stored in the management data storage unit 313 as tracking images, and also uses the facial feature amount 3132 and the whole body feature amount 3134 as tracking feature amounts. , is transmitted to each of the plurality of video analysis function units 20 via intra-server communication, Websocket, etc. (step S405). Thereafter, the image registration function unit 3531 ends this image registration process.

FIG. 18 is a flowchart showing the processing procedure of the tracking process by the monitoring/tracking execution function unit 23 of the video analysis function unit 20. The monitoring/tracking execution function unit 23 performs the processing shown in this flowchart every time it receives a tracking image and a feature amount from the tracking function unit 353 and image registration function unit 3531 of the web server 30, for example, through intra-server communication, Websocket, etc. implement.

The monitoring/tracking execution function unit 23 first acquires the tracking image and feature quantity received from the image registration function unit 3531 of the Web server 30 (step S111).

After that, the monitoring/tracking execution function unit 23 acquires the whole body image at time T and its feature amount from the human detection information extraction function unit 22 as the comparison whole body image and the comparison whole body feature amount (step S112). When the monitoring images of the corresponding monitoring camera 10 include full-body images of multiple people, the monitoring/tracking execution function unit 23 acquires the full-body image for comparison and the full-body feature for comparison for each person.

Then, the monitoring/tracking execution function unit 23 uses the whole body matching module 231 to determine whether each of the whole body features for matching from the human detection information extraction function unit 22 is similar to the whole body features for tracking from the image registration function unit 3531. The whole body image similarity is calculated (step S113).

Further, the monitoring/tracking execution function unit 23 acquires the face image at time T and its feature amount from the human detection information extraction function unit 22 as a matching face image and matching face feature amount (step S114). When the monitoring image of the corresponding monitoring camera 10 includes facial images of a plurality of people, the monitoring/tracking execution function unit 23 acquires a matching facial image and a matching facial feature amount for each person. Note that depending on the orientation of the person, a full body image may be extracted but a face image may not be extracted. Therefore, the number of full body images for comparison acquired in step S112 and the number of facial images for comparison acquired in step S114 do not necessarily match. Furthermore, there may be a case in which no face image can be extracted and no matching face image (and matching face feature amount) can be obtained.

Then, the monitoring/tracking execution function unit 23 uses the face matching module 232 to determine whether each of the matching face features from the human detection information extraction function unit 22 is similar to the tracking face feature from the image registration function unit 3531. The face image similarity is calculated (step S115).

Thereafter, the monitoring/tracking execution function unit 23 transmits the set of similarity and matching images for the whole body image and face image to the detection/tracking result determination function unit 34 of the web server 30 via intra-server communication, Websocket, etc. (Step S116). Note that the set of similarity and matching images may include both a set of full-body image similarity and matching full-body images and a set of face image similarity and matching facial images, There may be a case where only a set of whole body image similarity and a comparison whole body image is included without a set of similarity and a comparison face image.

Then, the monitoring/tracking execution function unit 23 determines whether or not the monitor presses the tracking stop button on the tracking alert screen displayed on the web browser 41 on the monitoring terminal 40 (step S117). Details of the tracking alert screen having a tracking stop button will be described later. If it is determined that the tracking stop button has been pressed, the monitoring/tracking execution function unit 23 ends this monitoring/tracking process.

Further, if it is determined that the tracking stop button has not been pressed, the monitoring/tracking execution function unit 23 adds the time interval t1 to the time T, that is, after updating to the next processing time ( Step S118), repeat the above processing from step S112. Here, the time interval t1 is as described above. Note that it goes without saying that a time interval different from this time interval t1 may be used.

FIG. 19 is a flowchart showing the processing procedure of the tracking result judgment process by the detection/tracking result judgment function unit 34 of the Web server 30. The detection/tracking result determination function unit 34 periodically executes the process shown in this flowchart, for example, in synchronization with the processing interval of the tracking process of the monitoring/tracking execution function unit 23 of the video analysis function unit 20.

The detection/tracking result determination function unit 34 first determines whether or not a set of similarity and matching images has been received from the monitoring/tracking execution function unit 23 (step S221). If it is determined that the set of similarity and matching images has not been received, the detection/tracking result determination function unit 34 ends this tracking result determination process.

On the other hand, if it is determined that the set of similarity and matching images has been received, the detection/tracking result determination function unit 34 obtains the face matching threshold 1 and the whole body matching threshold 1 (step S222). That is, the detection/tracking result determination function unit 34 determines the surveillance camera 10 corresponding to the video analysis function unit 20 that is the source of the set of similarity and matching images stored in the camera information table storage unit 321. The face matching threshold 1 and the whole body matching threshold 1 are obtained from the camera information table 3211 of .

Then, the detection/tracking result determination function unit 34 compares each of the received whole-body image similarities with the acquired whole-body matching threshold 1, and determines whether it is larger than the whole-body matching threshold 1 or whether there is a whole-body image similarity. (Step S223). If it is determined that the whole body image similarity is greater than the whole body matching threshold 1, the detection/tracking result determination function unit 34 determines that the person having the whole body feature amount is the person to be tracked. Therefore, in this case, the detection/tracking result determination function unit 34 stores the whole body image similarity determined to be large in a predetermined storage area of the storage device 302 together with the corresponding whole body image for comparison received in step S221. At the same time, the whole body image similarity is registered in the tracking table 3251 (step S224). When registering this whole body image similarity, if the tracking table 3251 of the relevant suspicious person ID is not yet stored in the tracking table storage unit 325, the tracking table 3251 is generated, and the generated tracking table 251 is stored in the tracking table storage unit 325. The information is stored in the storage unit 325. Further, if the tracking table 3251 of the relevant suspicious person ID is already stored in the tracking table storage unit 325, a record is added instead of generating a new tracking table 3251. The whole body image similarity registered in the tracking table 3251 is a whole body image similarity having a value larger than the whole body matching threshold 1.

Thereafter, or if it is determined in step S223 that there is no whole body image similarity greater than the whole body matching threshold 1, the detection/tracking result determination function unit 34 compares each of the received face image similarities with the acquired face matching threshold 1 and determines whether there is a face image similarity greater than the face matching threshold 1 (step S225). If it is determined that there is a face image similarity greater than the face matching threshold 1, the detection/tracking result determination function unit 34 determines that the person having that face feature is a tracking target. Therefore, in this case, the detection/tracking result determination function unit 34 stores the face image similarity determined to be large together with the corresponding matching face image received in step S221 in a predetermined memory area of the storage device 302, and registers the face image similarity in the tracking table 3251 (step S226). When registering this facial image similarity, if a tracing table 3251 for the corresponding suspicious person ID has not yet been stored in the tracing table storage unit 325, a tracing table 3251 is generated, and the generated tracing table 251 is stored in the tracing table storage unit 325. Also, if a tracing table 3251 for the corresponding suspicious person ID has already been stored in the tracing table storage unit 325, a record is added rather than a new tracing table 3251 being generated. The facial image similarity registered in the tracing table 3251 is a whole-body image similarity with a value greater than the facial matching threshold 1.

Thereafter, or if it is determined in step S225 that there is no face image similarity greater than the face matching threshold 1, the detection/tracking result determination function unit 34 ends this tracking result determination process.

20A to 20C are flowcharts showing the processing procedure of the tracking alert output processing by the tracking alert output function section 3532 of the tracking function section 353. The tracking alert output function unit 3532 periodically executes the process shown in this flowchart, for example, at every time interval t1 in the monitoring/tracking execution function unit 23 of the video analysis function unit 20.

The tracking alert output function unit 3532 first determines whether a narrowing designation operation has been performed on the tracking alert screen displayed on the Web browser 41 by the monitor on the monitoring terminal 40 (step S501). If it is determined that the narrowing down designation operation has been performed, the tracking alert output function unit 3532 proceeds to step S519, which will be described later.

On the other hand, if it is determined that the narrowing down specification operation has not been performed, the tracking alert output function section 3532 outputs the information from the camera information table 3211 for each surveillance camera 10 stored in the camera information table storage section 321. Face matching thresholds 1 to 3 and whole body matching thresholds 1 to 3 are acquired (step S502).

After that, the tracking alert output function unit 3532 obtains new tracking results from the tracking table 3251 stored in the tracking table storage unit 325 (step S503). That is, the tracking alert output function unit 3532 extracts and obtains the whole body image similarity and face image similarity in the tracking table 3251 in which real numbers of similarities are recorded instead of image similarity IDs.

Then, the tracking alert output function unit 3532 determines whether the face image similarity value in the tracking table 3251 is larger than the face matching threshold 3 (step S504). When it is determined that the face image similarity value is larger than the face matching threshold 3, the tracking alert output function unit 3532 converts the face image similarity value in the tracking table 3251 of the tracking table storage unit 325 into a face image based on the real number. The similarity label ID, here, is rewritten to ID=3 (step S505).

On the other hand, if it is determined that the face image similarity value is not greater than the face matching threshold 3, the tracking alert output function unit 3532 further determines whether the face image similarity value is greater than the face matching threshold 2. It is determined whether or not (step S506). If it is determined that the face image similarity value is larger than the face matching threshold 2, the tracking alert output function unit 3532 converts the face image similarity value in the tracking table 3251 of the tracking table storage unit 325 from the real number to the face image similarity value. The similarity label ID, here, is rewritten to ID=2 (step S507).

Further, when determining that the value of face image similarity is not greater than face matching threshold 2, the tracking alert output function unit 3532 further determines whether the value of face image similarity is greater than face matching threshold 1. (Step S508). If it is determined that the face image similarity value is larger than the face matching threshold 1, the tracking alert output function unit 3532 converts the face image similarity value in the tracking table 3251 of the tracking table storage unit 325 into the face image from the real number. The similarity label ID, here, is rewritten to ID=1 (step S509).

In the process related to the detection operation in the monitoring function unit 352, only the face image similarity is used. Therefore, if the value of the face image similarity is not larger than the face matching threshold 2, the value of the face image similarity is used for face matching. It is determined that the value is larger than the threshold value 1. On the other hand, in the processing related to the tracking operation of the tracking function unit 353, both the face image similarity and the whole body similarity are used, and the tracking table 3251 contains either the face image similarity or the whole body similarity. There are cases where only one side is registered. In other words, when the value of face image similarity is not larger than face matching threshold 2, there is a case where a face image similarity value larger than face matching threshold 1 is registered, and a case where the face image similarity value is less than face matching threshold 1. There are cases in which a similarity value is registered (or a face image similarity value is not registered). Therefore, just because the face image similarity value is not greater than the face matching threshold 2, it cannot be determined that the face image similarity value is greater than the face matching threshold 1.

If the face image similarity value in the tracking table 3251 is rewritten to the face image similarity label ID as described above, or the face image similarity value is not larger than the face matching threshold 1 in step S508 above. If it is determined that this is the case, the tracking alert output function unit 3532 moves to a whole body image similarity determination operation.

That is, the tracking alert output function unit 3532 determines whether the value of the whole body image similarity in the tracking table 3251 is larger than the whole body matching threshold 3 (step S510). If it is determined that the whole body image similarity value is larger than the whole body matching threshold 3, the tracking alert output function unit 3532 converts the whole body image similarity value in the tracking table 3251 of the tracking table storage unit 325 into the whole body image from the real number. The similarity label ID, here, is rewritten to ID=3 (step S511).

On the other hand, if it is determined that the whole body image similarity value is not greater than the whole body matching threshold 3, the tracking alert output function unit 3532 further determines whether the whole body image similarity value is greater than the whole body matching threshold 2. It is determined whether or not (step S512). If it is determined that the whole body image similarity value is larger than the whole body matching threshold 2, the tracking alert output function unit 3532 converts the whole body image similarity value in the tracking table 3251 of the tracking table storage unit 325 into the whole body image from the real number. The similarity label ID, here, is rewritten to ID=2 (step S513).

Further, when determining that the value of the whole body image similarity is not greater than the whole body matching threshold 2, the tracking alert output function unit 3532 further determines whether the value of the whole body image similarity is greater than the whole body matching threshold 1. (Step S514). If it is determined that the whole body image similarity value is larger than the whole body matching threshold 1, the tracking alert output function unit 3532 converts the whole body image similarity value in the tracking table 3251 of the tracking table storage unit 325 into the whole body image from the real number. The similarity label ID, here, is rewritten to ID=1 (step S515).

If the whole body image similarity value in the tracking table 3251 is rewritten to the whole body image similarity label ID as described above, or the whole body image similarity value is not larger than the whole body matching threshold 1 in step S514 above. If it is determined that this is the case, the tracking alert output function unit 3532 creates a new tracking information card that is a screen image (step S308). This tracking information card includes the matching full-body image and matching face image stored in the predetermined storage area of the storage device 302 in steps S224 and S226, the corresponding full-body image similarity label ID in the tracking table 3251, and the tracking information card. It is a screen image including a caution level display indicating a caution level based on a face image similarity degree label ID. Details of this tracking information card will be described later.

The tracking alert output function unit 3532 updates the tracking alert screen to be displayed on the web browser 41 using the created new tracking information card (step S517). Then, the tracking alert output function unit 3532 transmits this updated tracking alert screen to the Web browser 41 via the network NET, and displays it there (step S518). The web browser 41 displays this tracking alert screen in an alert screen display area provided in a part of the monitoring screen, or by opening a window separate from the monitoring screen. The tracking alert output function unit 3532 then ends this tracking alert output processing.

On the other hand, if it is determined in step S501 that a narrowing down designation operation has been performed, the tracking alert output function unit 3532 acquires the specified narrowing down level (step S519). This narrowing down level is an index for narrowing down the tracking information cards to be displayed on the tracking alert screen.

The tracking alert output function unit 3532 extracts a suspicious person ID according to the specified narrowing down level from the tracking table 3251 stored in the tracking table storage unit 325 (step S520).

Then, the tracking alert output function unit 3532 uses the whole body image 3112 stored as past search data in the past search data storage unit 311 and the video analysis function unit 20 based on the detection ID of the extracted suspicious person ID. The number of tracking information cards that can be displayed on the tracking alert screen, including the face image extracted from the whole body image 3112 by the human detection information extraction function unit 22, is created (step S521).

Thereafter, the tracking alert output function unit 3532 creates a new tracking information card (step S522) in the same manner as in step S516 above, and updates the tracking alert screen with the created tracking information card in the same way as in step S517 (step S523). ). The tracking alert output function unit 3532 then proceeds to the process of step S518, and transmits the updated tracking alert screen to the Web browser 41 to display it there.

FIG. 21 is a diagram showing an example of the tracking alert screen 43 displayed on the web browser 41. The tracking alert screen 43 includes a tracking target person information area 431, a tracking information card 432, and a narrowing down instruction area 433.

The tracking target information area 431 is an area for displaying information regarding the tracking target, and is additionally displayed at the top of the tracking alert screen 43 each time a tracking target is added. The old tracked person information area 431 is sequentially shifted downward. Here, the display contents are different between the highest level, that is, the latest tracking target person information area 431 and the other tracking target person information areas 431. That is, the highest tracking target person information area 431 includes an alert message 4311, a suspicious person ID 4312, a registered image 4313, tracking information 4314, and a tracking stop button 4315. On the other hand, the tracking target person information area 431 other than the top level only includes a registered image 4313 and tracking information 4314.

Here, the alert message 4311 is a message for calling the monitor's attention, and can display, for example, the risk level of the person to be monitored who is designated as the person to be tracked. This risk type can be transferred from the management table 3241. Suspicious person ID 4312 is identification information transferred from management table 3241. The registered images 4313 are a registered face image 3131 and a registered whole body image 3133 stored in the management data storage unit 313 of the tracked person. The tracking information 4314 is information regarding the tracking target person, and the tracking information 4314 in the highest tracking target person information area 431 and the tracking information 4314 in the other tracking target person information areas 431 have different contents. That is, the tracking information 4314 in the top-level tracking target person information area 431 includes the suspicious person ID, risk type, and explanatory text transferred from the management table 3241. The tracking information 4314 in the tracking target person information area 431 other than the highest level includes a tracking status instead of an explanatory text. The tracking stop button 4315 is a button that is pressed to end tracking of the person.

Further, the tracking information card 432 is additionally displayed at the top of the tracking alert screen 43 every time the tracking alert output function unit 3532 creates a new card. Old tracking information cards 432 are sequentially shifted downward. This tracking information card 432 includes, as its display contents, detection date and time information 4321, detected camera location information 4322, detected face image 4323, face caution stage display 4324, detected whole body image 4325, whole body caution stage display 4326, and detected background image. Contains 4327.

Here, the detection date and time information 4321 is date and time information when the tracking target person was detected from the monitoring image of any of the monitoring cameras 10. The detected camera location information 4322 is camera location information based on the detected surveillance camera ID. The detected face image 4323 is a comparison face image of the detected and saved tracking target person. The face attention level display 4324 indicates the level of similarity between the face image of the person to be tracked and the face image of the detected person. That is, the degree of similarity is shown not as a real number but as a warning level display of multiple levels, here three levels, based on the face image similarity label ID in the tracking table 3251. In this face attention level display 4324, the number of identified squares represented by hatching in FIG. 21 corresponds to the face image similarity label ID. For example, in the top tracking information card 432, the face image similarity label ID in the tracking table 3251 is "2", so in the face attention stage display 4324, two rectangles are identified and displayed, and two squares are displayed from the top. In the th tracking information card 432, the face image similarity label ID in the tracking table 3251 is "3", so three rectangles are identified and displayed in the face attention stage display 4324.

Similarly, the detected full-body image 4325 is a detected and saved full-body image of the tracked person for comparison. The whole body caution level display 4326 indicates the level of similarity between the whole body image of the tracked person and the whole body image of the detected person. That is, the similarity is shown not as a real number but as a warning level display of multiple levels, here three levels, based on the whole body image similarity label ID in the tracking table 3251. Similar to the face caution stage display 4324, this whole body caution level display 4326 also has the number of identified rectangles represented by hatching in FIG. 21 corresponding to the whole body image similarity degree label ID. For example, in the top tracking information card 432, the whole body image similarity label ID in the tracking table 3251 is "3", so in the whole body caution stage display 4326, three rectangles are identified and displayed, and two squares from the top are displayed. In the th tracking information card 432, the whole body image similarity label ID in the tracking table 3251 is "2", so two squares are identified and displayed in the whole body caution stage display 4326.

The detected background image 4327 is a frame image that is the original monitoring image from which the face image for comparison and/or the whole body image for comparison are extracted. As this frame image, a frame image 3111 stored in the past search data storage unit 311 is used.

Note that the tracking information card 432 does not necessarily include both the detected face image 4323 and the face caution stage display 4324, and the detected whole body image 4325 and the whole body caution stage display 4326, and may include only one of them. Possible.

The narrowing down instruction area 433 is an area for accepting instructions for narrowing down the tracking information cards 432 to be displayed on the tracking alert screen 43. This narrowing down instruction area 433 includes a face image similarity selection button 4331, a face image similarity selection reset button 4332, a whole body image similarity selection button 4333, and a whole body image similarity selection reset button 4334.

Here, the face image similarity selection buttons 4331 include three buttons corresponding to the number of stages of the face attention level display 4324. Only the tracking information cards 432 that include the face caution level display 4324 of a specified number of levels or more in accordance with the pushing operation of the face image similarity selection button 4331 by the observer are displayed on the tracking alert screen 43. In the example of FIG. 21, the number of stages is not specified with the face image similarity selection button 4331, so the tracking alert screen 43 displays a tracking information card 432 that includes a face attention stage display 4324 of one or more stages. . The face image similarity selection reset button 4332 is a button that is pressed when resetting the number of stages specified by the face image similarity selection button 4331.

Similarly, the whole body image similarity selection button 4333 includes three buttons corresponding to the number of stages of the whole body caution level display 4326. Only the tracking information cards 432 that include the whole body caution level display 4326 of a specified number or more in accordance with the press operation of the whole body image similarity selection button 4333 by the observer are displayed on the tracking alert screen 43. In the example of FIG. 21, since the number of stages is not specified with the whole body image similarity selection button 4333, the tracking information card 432 including one or more stages of whole body caution stage display 4326 is displayed on the tracking alert screen 43. . The whole body image similarity selection reset button 4334 is a button that is pressed when resetting the number of stages specified by the whole body image similarity selection button 4333.

FIG. 22 is a diagram showing another example of the tracking alert screen 43 displayed on the web browser 41. This example shows a case where three stages are designated by the face image similarity selection button 4331 and one stage is designated by the whole body image similarity selection button 4333. Here, the pressed button is identified and displayed as indicated by hatching in FIG. 21. The specification by the face image similarity selection button 4331 and the specification by the whole body image similarity selection button 4333 become an AND condition, so that only the tracking information cards 432 that match the number of stages specified in both are displayed on the tracking alert screen 43. become.

(2-3) Past Search Operation The tracking operation by the tracking function unit 353 is a function of tracking a person to be tracked in real time. On the other hand, there are cases where it is desired to investigate whether a specific person has visited the facility in the past.

Therefore, the past search function unit 354 receives the time and place specification from the web browser 41 and searches the past search data based on the past search data table 3221 stored in the past search data table storage unit 322. A full-body image 3112 of the person who visited the specified time and place is extracted from the storage unit 311. Then, a selection screen displaying a list of the extracted whole body images 3112 is created and displayed on the Web browser 41.

Upon receiving the selection and designation of the person to be searched from the web browser 41, the past search function unit 354 uses the selected and designated full-body image 3112 as the full-body image for tracking, and also replaces it with the monitoring image from the real-time monitoring camera 10. The same operation as the tracking function section 353 is performed for the frame image 3111 stored in the past search data storage section 311 for the specified time and place.

Thereby, the past search function unit 354 searches for a person similar to the selected person from among the past visitors stored in the past search data storage unit 311, and displays the results on the tracking alert screen 43. It can be displayed on the web browser 41 as .

(3) Effects As detailed above, the Web server 30 as a person detection device according to one embodiment uses two or more similarity calculation methods having mutually different judgment criteria to determine the target image for a specific image. A plurality of similarity calculation units for classifying a target image into a plurality of stages based on each similarity calculated by the monitoring/tracking execution function unit 23 of the video analysis function unit 20, which is a similarity calculation unit that calculates two or more degrees of similarity. The threshold values for each similarity calculation method are stored in the camera information table storage unit 321 as a threshold storage unit, and the detection/tracking result determination function unit 34 as a detection unit calculates the threshold values for each of the plurality of surveillance cameras 10. Obtain from the monitoring/tracking execution function unit 23 at least one degree of similarity of the image of the person to be calculated, which is the target image extracted from each of the periodically acquired monitoring images, to the image of the specific person, which is the specific image. , based on a plurality of threshold values stored in the camera information table storage unit 321 corresponding to the similarity calculation method used to calculate at least one similarity in the monitoring/tracking execution function unit 23. An image similar to an image of a specific person is detected from among the images of the person to be calculated for similarity extracted from the monitoring images of each of the surveillance cameras 10, and the detected image of the person to be calculated for similarity is placed in a plurality of caution levels. Detect which one is. In this way, by classifying similarities that are difficult to compare with each other due to different judgment criteria of similarity calculation methods into multiple levels that do not depend on the similarity calculation method, it is possible to It is also possible to apply the degree of similarity to the same evaluation criteria.
More specifically, the camera information table storage unit 321 stores, for example, a plurality of threshold values for determining which of a plurality of caution levels the degree of similarity is for each of at least one matching model, and the detection /The tracking result determination function unit 34 calculates the degree of similarity between the image of the specific person and the image of the specific person extracted from each of the surveillance images periodically acquired by each of the plurality of surveillance cameras 10, for example. The degree of similarity is acquired from the monitoring/tracking execution function unit 23 that is calculated using the matching model, and based on a plurality of threshold values stored in the camera information table storage unit 321, the degree of similarity is determined in which of the caution stages. Detect if there is.
Such a detected attention step allows the observer to easily recognize the certainty of the authentication result. Further, since the threshold value for each surveillance camera 10 is stored in the camera information table storage unit 321, it is possible to reduce variations in the acquired similarity between the surveillance cameras 10.

Further, the web server 30 presents an image similar to the image of the specific person detected by the detection/tracking result determination function unit 34 together with a stage display indicating the caution level detected by the detection/tracking result determination function unit 34. It is possible to further include a detection alert output function section 3521 as an output screen generation section that generates a detection alert screen 42 which is an output screen of the above.
More specifically, the detection alert output function unit 3521 generates an image of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras 10 based on the determination result of the detection/tracking result determination function unit 34. Among these, a detection alert screen 42 is generated for presenting an image of a person to be calculated for similarity similar to an image of a specific person along with a caution level.
In this way, since the degree of similarity which is the authentication result is presented as the caution stage display 4215, it becomes easy for even non-skilled observers to intuitively and accurately recognize the degree of image similarity. Therefore, the observer can easily recognize the reliability of the authentication result.

Note that the detection/tracking result determination function unit 34 receives from the monitoring/tracking execution function unit 23 a face image of the person to be calculated for similarity and a face image of the specific person extracted from each of the monitoring images of the plurality of monitoring cameras 10. The detection alert output function unit 3521 may generate a detection alert screen 42 that presents a face image of the person to be calculated for similarity that is similar to the face image of the specific person together with a caution level display 4215. .
This allows the supervisor to easily recognize the certainty of the facial image authentication result. Note that instead of a face image, a whole body image of a specific person and a face image of a person to be calculated for similarity may be used.

Here, the plurality of threshold values are a first threshold for classifying whether the target image of the person to be calculated for similarity is an image of a specific person, and a first threshold for classifying whether or not the image of the person to be calculated for similarity is an image of a specific person. If the image is an image, it can include at least one second threshold for classifying the image of the person to be calculated for similarity into a plurality of stages.
More specifically, the camera information table storage unit 321 sets a face matching threshold value 1 corresponding to the degree of similarity at which the image of the person to be calculated for similarity is estimated to be an image of a specific person, as a first threshold value; A face matching threshold 2 and a face matching threshold 3 each corresponding to a degree of similarity greater than the face matching threshold 1 are stored as second thresholds. The detection/tracking result determination function unit 34 detects an image of the similarity calculation target person whose similarity is greater than the face matching threshold 1, and the detection alert output function unit 3521 detects an image of the person to be calculated whose face is higher than the face matching threshold 1. The image of the similarity calculation target person detected based on the matching threshold 1 is presented together with an attention level display 4215 indicating the attention level determined by the detection/tracking result determination function unit 34 based on the face matching threshold 2 and the face matching threshold 3. A detection alert screen 42 can be generated.
As a result, it is possible to easily detect the image of the person to be calculated based on the similarity calculation target to be presented on the detection alert screen 42 using the face matching threshold 1, and for each detected image, what kind of image is to be displayed based on the face matching threshold 2 and face matching threshold 3? It becomes possible to decide whether to display the caution level display 4215.

In addition to the configuration of the web server 30 as a person detection device, the web server 30 as a person tracking device according to an embodiment has an image similar to the image of a specific person presented with a staged display on the detection alert screen 42. From among the images of the plurality of similarity calculation targets, the designation of the image of the tracking target to be tracked is received from the monitoring terminal 40 operated by the supervisor, and the designated image is registered as a tracking image; The tracking image, which is the registered image of the person to be tracked who should be tracked, is used as a specific image in the monitoring/tracking execution function unit 23 to calculate the similarity of the image of the person to be tracked using at least two similarity calculation methods. The image registration function unit 3531, which is a registration unit that extracts at least two degrees of similarity between Tracking as a tracking output screen for presenting two or more similarity calculation target person images similar to the tracking target person images together with a stage display indicating the caution level detected by the detection/tracking result determination function unit 34. A tracking alert output function section 3532 as a tracking output screen generation section that generates the alert screen 43 can be provided.
More specifically, the image registration function unit 3531 selects a tracking target to be tracked from among a plurality of similarity calculation target person images similar to the specific person image presented on the detection alert screen 42 along with a step-by-step display. The specified face image and the whole body image of the person to be tracked corresponding to the face image are used as a tracking face image and a tracking whole body image. Register as. In addition, the camera information table storage unit 321 has a first threshold value corresponding to the degree of similarity estimated to indicate that the face image of the person to be calculated is the tracking face image registered by the image registration function unit 3531. A face matching threshold 1 and a whole body matching threshold 1 corresponding to the degree of similarity estimated to indicate that the whole body image of the person to be calculated for similarity is a tracking whole body image registered by the image registration function unit 3531 are stored; At least one second threshold, a face matching threshold 2 and a face matching threshold 3 each corresponding to a degree of similarity of the face image that is greater than the face matching threshold 1, and a degree of similarity of the whole body image that is greater than the whole body matching threshold 1, respectively. A whole body matching threshold 2 and a whole body matching threshold 3 corresponding to are stored. Then, the detection/tracking result determination function unit 34 receives the degree of similarity extracted from each of the surveillance images of the plurality of surveillance cameras 10, which is calculated using, for example, the first matching model, from the monitoring/tracking execution function unit 23. The similarity between the face image of the person to be calculated and the tracking face image registered by the image registration function unit 3531, and the whole body image of the person to be calculated using the second matching model, for example, and the image registration. The degree of similarity with the full body image for tracking registered by the functional unit 3531 is acquired, and from among the degrees of similarity of the acquired face images, the face of the target person for similarity calculation having a degree of similarity greater than face matching threshold 1. The tracking alert output function unit 3532 detects at least one of the image and the whole body image of the person to be calculated for similarity having a degree of similarity greater than the whole body matching threshold 1 from among the degrees of similarity of the acquired whole body images, and the tracking alert output function unit 3532 The detection/tracking result determination function unit 34 converts at least one of the face image and the whole body image of the similarity calculation target person detected based on at least one of the face matching threshold 1 and the whole body matching threshold 1 to the corresponding face matching threshold 2 and the face. A tracking alert screen 43 is generated that is presented along with a stage display indicating the caution stage determined based on the verification threshold 3 and at least one of the whole body verification threshold 2 and the whole body verification threshold 3.
As a result, two or more similarity calculation methods having different judgment criteria, for example, a first matching model for face images and a second matching model for whole body images, can be used to create similarities with different ranges or scales. Even in the case of outputting the degree value, the degree of similarity is displayed in the same stepwise manner on the tracking alert screen 43, so that the observer can easily recognize the same degree of similarity.

Further, the web server 30 serving as a person tracking device extracts a whole-body image of a person from each of the monitoring images, extracts a whole-body feature quantity and a face image of the person from the whole-body image, and extracts the features of the person from the face image. an image storage section that accumulates the whole body image, the whole body feature amount, and the face feature amount extracted by the human detection information extraction function section 22 of the video analysis function section 20, which is an extraction section that extracts the facial feature amount, which is the amount of the image; The image registration function unit 3531 may further include a file server 31 as a file server 31, and the image registration function unit 3531 stores the whole body image, whole body feature amount, and facial feature stored in the file server 31 corresponding to the face image of the tracking target person specified from the monitoring terminal 40. The detection/tracking result determination function unit 34 registers the quantity and the face image from which the facial feature quantity is extracted as a tracking whole body image, a tracking whole body feature quantity, a tracking facial feature quantity, and a tracking face image. Monitoring images accumulated in the file server 31 from the time of registration by the image registration function unit 3531 are used as matching images, and the whole body feature amount and the tracking whole body extracted by the human detection information extraction function unit 22 for each matching image are Monitoring/tracking that calculates the whole body image similarity, which is the similarity with the feature amount, and the face image similarity, which is the similarity between the facial feature amount extracted by the human detection information extraction function unit 22 and the tracking facial feature amount. A set of the calculated whole body image similarity and face image similarity, the tracking face image, and the tracking whole body image is received from the execution function unit 23, and at least one of the face image similarity and the whole body image similarity corresponds to the corresponding face. It is determined whether the degree of similarity is greater than at least one of the matching threshold 1 and the whole body matching threshold 1, and the face image and the whole body image of the target person for similarity calculation have a similarity that is larger than at least one of the face matching threshold 1 and the whole body matching threshold 1. The detection alert output function unit 3521 detects at least one of the face image and the whole body image of the similarity calculation target person detected by the detection/tracking result determination function unit 34, and the corresponding face matching threshold 2 and face. The stage display indicating the caution stage determined based on at least one of the verification threshold 3, the whole body verification threshold 2, and the whole body verification threshold 3, and the position information of the surveillance camera 10 that acquired the surveillance image as the verification image are displayed as an image. A tracking alert screen 43 is generated as a tracking output screen to be presented together with the tracking whole body image and tracking face image registered by the registration function unit 3531.
As a result, even if a face image of the person to be tracked cannot be acquired by the surveillance camera 10, the person to be tracked can be tracked using the whole body image.

In addition, upon receiving the stage number designation from the monitoring terminal 40, the detection alert output function unit 3521 outputs a designated number of at least one of a face image and a whole body image of the person to be calculated for similarity detected by the detection/tracking result determination function unit 34. A tracking alert as a narrowing output screen that presents, along with a stage display, those that have a degree of similarity of at least one of face matching threshold 2 and face matching threshold 3 and whole body matching threshold 2 and whole body matching threshold 3 corresponding to the selected stage. Screen 43 can be generated. If the number of false positives increases during tracking, the number of displayed results will increase, and results with low accuracy will also be displayed, making it difficult to identify the person to be tracked.
The web server 30 as a person tracking device according to an embodiment can encourage viewing of only highly accurate tracking results by making it possible to set the criteria for the display stage of the stage display indicating the caution stage.

Alternatively, in the web server 30 as a person tracking device according to an embodiment, the camera information table storage unit 321 as a threshold value storage unit uses two or more similarity calculation methods having mutually different judgment criteria to perform identification. The target image is divided into multiple stages based on each of the similarities calculated by the monitoring/tracking execution function unit 23 of the video analysis function unit 20, which is a similarity calculation unit that calculates two or more degrees of similarity between the target image and the image. A plurality of threshold values for classification are stored for each similarity calculation method, and an image registration function unit 3531 serving as a registration unit selects a person to be tracked who is specified from a monitoring terminal 40 operated by a supervisor. The image is registered as a specific image. Then, the detection/tracking result determination function unit 34 as a detection unit identifies the image of the person to be calculated for similarity, which is the target image extracted from each of the surveillance images periodically acquired by each of the plurality of surveillance cameras 10. At least one degree of similarity for the image is acquired from the monitoring/tracking execution function unit 23, and a camera information table is created corresponding to the similarity calculation method used to calculate the at least one degree of similarity in the monitoring/tracking execution function unit 23. Based on a plurality of threshold values stored in the storage unit 321, an image similar to the image of the tracking target person is selected from among the images of the target person for similarity calculation extracted from the monitoring images of the plurality of surveillance cameras 10. At the same time, the tracking alert output function unit 3532, which serves as a tracking output screen generation unit, detects which of a plurality of caution levels the image of the detected similarity calculation target person is in, and the detection/tracking result determination function unit 34 The monitoring/tracking execution function unit calculates two or more similarity calculation target person images that are similar to the images of the tracking target person detected based on the at least two similarities obtained from the monitoring/tracking execution function unit 23. A tracking alert screen 43 is generated as a tracking output screen to be presented together with a step display indicating the attention step detected by the controller 23.
More specifically, the camera information table storage unit 321 stores, for example, a plurality of threshold values for determining which of a plurality of caution levels the degree of similarity is for each of two or more matching models, The image registration function unit 3531 registers the image of the person to be tracked who is designated by the monitoring terminal 40 operated by the monitor. Then, the detection/tracking result determination function unit 34 calculates the degree of similarity extracted from the image of the tracked person registered by the image registration function unit 3531 and each of the surveillance images periodically acquired by each of the plurality of surveillance cameras 10. Two or more degrees of similarity are acquired from the monitoring/tracking execution function unit 23 that calculates the degree of similarity with the target person's image using two or more matching models, and are stored in the camera information table storage unit 321. The tracking alert output function section 3532 determines which of the caution stages each of the two or more acquired degrees of similarity is based on a plurality of threshold values, and the tracking alert output function section 3532 determines whether the obtained two or more degrees of similarity are in the caution stage based on the determination result of the detection/tracking result determination function section 34. Among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras 10, images of the similarity calculation target person that are similar to the image of the tracking target person are displayed together with a stage display indicating each caution level. A tracking alert screen 43 is generated for presentation.
As a result, even if two or more similarity calculation methods having different judgment criteria, for example, two or more matching models output similarity values with different ranges or scales, the tracking alert screen 43 By displaying the degree of similarity using a similar stage display, the observer can easily recognize the same degree of similarity.

Here, upon receiving the stage number designation from the monitoring terminal 40, the tracking alert output function unit 3532 outputs a signal corresponding to the designated stage among the images of the similarity calculation target person detected by the detection/tracking result determination function unit 34. It is possible to generate a tracking alert screen 43 as a narrowing down output screen that presents those having a degree of similarity equal to or higher than a threshold value together with a step-by-step display.
In this way, by making it possible to set the criteria for the display stage of the stage display indicating the caution stage, viewing only highly accurate tracking results can be encouraged.

The camera information table storage unit 321 also stores a face matching threshold 1 or a whole body matching threshold 1, which is a first threshold corresponding to the degree of similarity in which the image of the person to be calculated for similarity is estimated to be the image of the person to be tracked. , face matching threshold 2 and face matching threshold 3, or whole body matching threshold 2 and whole body matching threshold 3, which are at least one second threshold corresponding to a degree of similarity greater than face matching threshold 1 or whole body matching threshold 1, respectively. , and the detection/tracking result determination function unit 34 detects an image of the similarity calculation target person having a face similarity or whole body similarity greater than the face matching threshold 1 or the whole body matching threshold 1, and issues a tracking alert. The output function unit 3532 performs face matching on the image of the similarity calculation target person detected by the detection/tracking result judgment function unit 34 based on the face matching threshold 1 or the whole body matching threshold 1. It is possible to generate a tracking alert screen 43 that is presented together with a stage display indicating the caution stage determined based on the threshold values 2 and 3 or the whole body comparison threshold values 2 and 3.
Thereby, it is possible to easily detect the image of the similarity calculation target person to be presented on the tracking alert screen 43 using the face matching threshold 1 or the whole body matching threshold 1, and for each detected image, the face matching threshold 2, the face matching threshold 3, Alternatively, it becomes possible to determine what kind of attention level display 4215 should be displayed based on the whole body matching threshold 2 and the whole body matching threshold 3.

Further, a person tracking system according to an embodiment includes a person tracking device according to an embodiment, a plurality of surveillance cameras 10, a surveillance terminal 40 operated by a supervisor, and an analysis function section 22 for extracting human detection information. A video analysis function section 20 as a section can be provided.

Further, the person detection method according to one embodiment is a person detection method for detecting a specific person from surveillance images acquired by a plurality of surveillance cameras 10, in which the web server 30, which is a computer, has mutually different judgment criteria. Similarity calculated by the monitoring/tracking execution function unit 23 of the video analysis function unit 20, which is a similarity calculation unit that calculates two or more degrees of similarity between a target image and a specific image using two or more similarity calculation methods. A plurality of threshold values for classifying a target image into a plurality of stages based on each similarity calculation method are stored in the camera information table storage unit 321 of the DB server 32 provided in the storage device 302 which is a memory. At least one of the images of the specific person, which is the specific image, of the images of the subject of similarity calculation, which are the target images extracted from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras 10, is stored. The degree of similarity is acquired from the monitoring/tracking execution function unit 23 and stored in the camera information table storage unit 321 in accordance with the similarity calculation method used to calculate at least one degree of similarity in the monitoring/tracking execution function unit 23. Based on a plurality of threshold values, an image similar to an image of a specific person is detected from among the images of the subject person extracted from the surveillance images of the plurality of surveillance cameras 10, and the detected image is It is attempted to detect which of a plurality of attention stages the image of the person to be calculated for similarity is in.
More specifically, for example, for each of at least one matching model, the camera information table storage unit 321 stores a plurality of threshold values for determining which of the plurality of caution levels the degree of similarity is, and A monitoring/tracking execution function that calculates the similarity between an image of a specific person and an image of a specific person extracted from each of the surveillance images periodically acquired by each of the surveillance cameras 10 of The similarity degree is acquired from the camera information table storage unit 321, and based on a plurality of threshold values stored in the camera information table storage unit 321, it is detected whether the acquired similarity degree is in a caution level.
Such a detected attention step allows the observer to easily recognize the certainty of the authentication result. Further, since the threshold value for each surveillance camera 10 is stored in the camera information table storage unit 321, it is possible to reduce variations in the acquired similarity between the surveillance cameras 10.

Further, the person tracking method according to one embodiment is a person tracking method for tracking a person to be tracked from surveillance images acquired by a plurality of surveillance cameras 10, and the Web server 30, which is a computer, A detection alert screen 42 is generated, which is an output screen for presenting an image similar to an image of a specific person detected by the person detection method according to an embodiment, together with a step display indicating the detected caution step. A monitoring terminal 40 operated by a monitor accepts designation of an image of a tracking target to be tracked from among a plurality of images of similarity calculation targets who are similar to images of a specific person presented with a staged display. , the specified image is registered as a tracking image in the management data storage section 313 of the file server 31 provided in the storage device 302 which is a memory. Then, using the tracking image, which is the image of the person to be tracked who is to be tracked and is registered in the management data storage unit 313, as a specific image, the monitoring/tracking execution function unit 23 applies at least two similarity calculation methods. to extract at least two similarities between the images of the person to be tracked, which is a specific image of the images of the person to be tracked, extracted from the monitoring images of each of the plurality of surveillance cameras 10. At least two degrees of similarity for the image are acquired from the monitoring/tracking execution function unit 23, and management data is stored in accordance with the similarity calculation method used to calculate each similarity in the monitoring/tracking execution function unit 23. Based on the plurality of threshold values stored in the unit 313, two or more images of the person to be tracked that are similar to the image of the person to be tracked are selected from among the images of the person to be tracked, which are extracted from the monitoring images of the plurality of surveillance cameras 10. At least two or more of the images obtained from the monitoring/tracking execution function unit 23 are obtained by detecting an image of the person to be calculated for similarity, and detecting which of a plurality of attention stages the detected image of the person to be calculated for similarity is in. A tracking alert screen 43 as a tracking output screen for presenting each of two or more similarity calculated target person images that are similar to the image of the tracked target person detected from the similarity of , together with a step display indicating the detected caution level. I am trying to generate .
More specifically, from among a plurality of images of persons to be subjected to similarity calculation extracted from each of the monitoring images of the plurality of monitoring cameras 10 presented on the detection alert screen 42 together with the caution level, the tracking to be performed is selected. A designation of a target person's face image is received from the monitoring terminal 40 operated by a supervisor, and the designated face image and a whole body image of the tracking target person corresponding to the face image are used as a tracking face image and a tracking whole body image. It is registered in the management data storage unit 313 as . Then, the monitoring/tracking execution function unit 23 registers the facial image of the person to be calculated for similarity extracted from each of the monitoring images of the plurality of monitoring cameras 10, which is calculated using, for example, the first matching model. The degree of similarity with the face image for tracking and the degree of similarity between the whole body image of the person to be calculated using the second matching model and the registered whole body image for tracking are obtained. In addition, as a first threshold in the camera information table storage unit 321, a face matching threshold 1 corresponding to the similarity estimated to be a tracking face image in which the face image of the person to be calculated for similarity is registered and a similarity calculation A whole-body matching threshold 1 corresponding to the degree of similarity estimated to indicate that the subject's full-body image is a registered tracking full-body image, and at least one second threshold for a face image that is larger than the face matching threshold 1. A face matching threshold 2 and a face matching threshold 3 corresponding to the degree of similarity, and a whole body matching threshold 2 and a whole body matching threshold 3 corresponding to the degree of similarity of the whole body images each larger than the whole body matching threshold 1 are stored. Then, from among the obtained similarities, a face image of the similarity calculation target person with a similarity higher than the face matching threshold 1 and a whole body image of the similarity calculation target with a similarity higher than the whole body matching threshold 1 , and at least one of the face image and the whole body image of the similarity calculation target person detected based on at least one of the face matching threshold 1 and the whole body matching threshold 1 is subjected to the corresponding face matching threshold 2 and face matching. A tracking alert screen 43 is generated as a tracking output screen to be presented along with a stage display indicating the caution level determined based on the threshold 3 and at least one of the whole body matching threshold 2 and the whole body matching threshold 3.
As a result, two or more similarity calculation methods having different judgment criteria, for example, a first matching model for face images and a second matching model for whole body images, can be used to create similarities with different ranges or scales. Even in the case of outputting the degree value, the degree of similarity is displayed in the same stepwise manner on the tracking alert screen 43, so that the observer can easily recognize the same degree of similarity.

Alternatively, the person tracking method according to one embodiment is a person tracking method for tracking a person to be tracked from surveillance images acquired by a plurality of surveillance cameras 10, and in which the web servers 30, which are computers, communicate with each other. A monitoring/tracking execution function unit of the video analysis function unit 20, which is a similarity calculation unit that calculates two or more degrees of similarity of a target image to a specific image using two or more similarity calculation methods having different judgment criteria. The camera information table of the DB server 32 is provided with a plurality of threshold values for classifying the target image into a plurality of stages based on each of the similarities calculated by 23 in the storage device 302, which is a memory, for each similarity calculation method. Management data of the file server 31 that is stored in the storage unit 321 and provided in the storage device 302, which is a memory, as a specific image of the image of the person to be tracked who is designated from the monitoring terminal 40 operated by the supervisor. Monitors at least one degree of similarity to a specific image of the image of the person to be calculated, which is a target image registered in the storage unit 313 and extracted from each of the monitoring images periodically acquired by each of the plurality of monitoring cameras 10. A plurality of information obtained from the /tracking execution function unit 23 and stored in the camera information table storage unit 321 corresponding to the similarity calculation method used to calculate at least one similarity in the monitoring/tracking execution function unit 23. Based on the threshold value, an image similar to the image of the tracking target person is detected from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras 10, and the detected similarity is calculated from among the images of the target person. Detect which of multiple attention stages the image of the target person is in, and calculate two or more images similar to the detected image of the target person based on the degree of similarity of at least two or more obtained from the monitoring/tracking execution function unit 23. A tracking alert screen 43 is generated as a tracking output screen for presenting each image of the person to be subjected to similarity calculation together with a stage display indicating the detected caution level.
More specifically, for example, for each of two or more matching models, the camera information table storage unit 321 stores a plurality of threshold values for determining which of the plurality of caution levels the degree of similarity is in, and the monitoring An image of the tracking target to be tracked designated by the monitoring terminal 40 operated by the person is registered in the management data storage unit 313, and images of the registered tracking target and each of the plurality of monitoring cameras 10 are periodically acquired. Two or more degrees of similarity are obtained from the monitoring/tracking execution function unit 23, which calculates the degree of similarity with the image of the person to be calculated using two or more matching models, respectively, extracted from each of the surveillance images obtained. Based on a plurality of stored threshold values, it is determined whether the obtained similarity is in the caution stage, and based on the result of the determination, the similarity extracted from the surveillance images of each of the plurality of surveillance cameras 10 is determined. A tracking alert screen 43 is generated to present an image of the similarity calculation target person who is similar to the image of the tracking target person among the images of the calculation target person, together with a stage display indicating each caution level.
As a result, even if two or more similarity calculation methods having different judgment criteria, for example, two or more matching models output similarity values with different ranges or scales, the tracking alert screen 43 By displaying the degree of similarity using a similar stage display, the observer can easily recognize the same degree of similarity.

[Other embodiments]
In the above embodiment, the whole body image of the person is extracted from the surveillance image acquired by the surveillance camera 10, and the face image of the person is extracted from the extracted whole body image, but the present invention is not limited to this. If a person's face is included in the monitoring image, the face image may be extracted without extracting the whole body image.

Further, in one embodiment, the attribute information of the person itself is digitized as the whole body image feature amount and the face image feature amount, but the present invention is not limited to this. The whole body feature amount may include, for example, a numerical representation of incidental information associated with a person, such as clothing, belongings (type and color of bag, stroller, etc.), accessories (sunglasses, mask, etc.). Further, the whole body feature amount may include numerical information such as fever information, pulse rate, etc., which is obtained from an image of the person or from other sensors. Further, the facial feature amount may include numerical values of incidental information attached to the face, such as accessories (glasses, hats, etc.).

In addition, the whole body image feature amount and face image feature amount are only those that quantify the attribute information of the person itself, and those that quantify the accompanying information and the person's internal information are treated as other feature quantities. Another feature amount may be used as one of the detection criteria similar to the whole body image feature amount and the face image feature amount.

Further, in the above embodiment, the past search data storage unit 311 of the file server 31 stores a frame image 3111, a whole body image 3112, a whole body feature amount 3113, and a face feature amount 3114 as past search data. However, the facial image from which the facial feature amount 3114 is extracted may also be stored.

Further, the order of the processing steps shown in the flowcharts of FIGS. 13 to 15 and FIGS. 17 to 20C is an example, and the order is not limited to this order. For example, in FIG. 18, the processing in steps S112 and S113 and the processing in steps S114 and S115 may be performed in reverse order or may be performed in parallel. Furthermore, the processing in steps S504 to S509 in FIG. 20A and the processing in steps S510 to S515 in FIG. 20B may be performed in reverse order or in parallel. In this way, the processing order and the like of each processing step may be changed as long as there is no discrepancy with the preceding or subsequent processing steps.

Furthermore, the detection alert screen 42 shown in FIG. 16 and the tracking alert screen shown in FIG. 21 (and FIG. 22) are not limited to this layout and display content. For example, the caution stage display 4215, the face caution stage display 4324, and the whole body caution stage display 4326 are expressed by the number of rectangles that are identified and displayed, but the color of one square is changed, for example, yellow, orange, or red. The caution level may be indicated by an identification display. Alternatively, instead of the number or color of rectangles, the caution levels may be expressed in letters, such as A to C or 1 to 3.

In short, the present invention is not limited to the above-described embodiments as they are, but can be implemented by modifying the constituent elements within the scope of the invention at the implementation stage. Moreover, various inventions can be formed by appropriately combining the plurality of components disclosed in the above embodiments. For example, some components may be deleted from all the components shown in the embodiments. Furthermore, components from different embodiments may be combined as appropriate.

1... Person tracking system 10... Surveillance camera 20... Video analysis function unit 21... Image acquisition module 22... Person detection information extraction function unit 23... Monitoring/tracking execution function unit 30... Web server 31... File server 32... Database (DB) Server 33...Search human detection result storage function unit 34...Detection/tracking result determination function unit 35...Web application 40...Monitoring terminal 41...Web browser 42...Detection alert screen 43...Tracking alert screen 221...Full body detection module 222...Area tracking Module 223...Whole body feature extraction module 224...Face detection module 225...Face feature extraction module 231...Whole body matching module 232...Face matching module 251...Tracking table 301A...Hardware processor 301B...Program memory 302...Storage device 303...Communication Interface device 304... Bus 311... Data storage section for past search 312... Detection history data storage section 313... Management data storage section 321... Camera information table storage section 322... Data table storage section for past search 323... Detection history table storage section 324 ... Management table storage section 325 ... Tracking table storage section 351 ... Login function section 352 ... Monitoring function section 353 ... Tracking function section 354 ... Past search function section 421 ... Detected person card 431 ... Tracking target person information area 432 ... Tracking information card 433 ...Narrowing down instruction area 3111...Frame image 3112...Whole body image 3113, 3134...Whole body feature amount 3114, 3122, 3132...Face feature amount 3121, 4212...Detected face image 3131...Registered face image 3133...Registered whole body image 3211...Camera information table 3221...Data table for past search 3231...Detection history table 3241...Management table 3251...Tracking table 3521...Detection alert output function section 3531...Image registration function section 3532...Tracking alert output function section 4211...Monitoring target person face image 4213...Caution Alert message 4214, 4312...Suspicious person ID
4215... Caution stage display 4216... Detection information 4217... Tracking start button 4311... Alert message 4313... Registered image 4314... Tracking information 4315... Tracking stop button 4321... Detection date and time information 4322... Detected camera location information 4323... Detected face image 4324... Face caution stage display 4325...Detected whole body image 4326...Whole body caution stage display 4327...Detected background image 4331...Face image similarity selection button 4332...Face image similarity selection reset button 4333...Whole body image similarity selection button 4334...Full body image similarity Degree selection reset button NET...Network SV...Server device

Claims (11)

Based on the similarity calculated by a similarity calculation unit that calculates the similarity of the target image to a specific image using one similarity calculation method, the target image is classified into a plurality of stages with respect to one matching model. a threshold storage unit storing a plurality of threshold values for
One similarity of the image of the target person, which is the target image extracted from each of the surveillance images periodically acquired by each of a plurality of surveillance cameras, to the image of the specific person, which is the specific image, is calculated as the similarity. Based on the plurality of threshold values acquired from the degree calculation unit and stored in the threshold value storage unit, from among the images of the subject for similarity calculation extracted from the monitoring images of each of the plurality of monitoring cameras. , a detection unit that detects an image similar to the image of the specific person and detects which of a plurality of caution levels the detected image of the person to be calculated for similarity is in;
an output screen generation unit that generates an output screen for presenting the image similar to the image of the specific person detected by the detection unit together with a step display indicating the caution level detected by the detection unit;
A person detection device comprising: The plurality of threshold values include a first threshold value for classifying whether the image of the person to be calculated for similarity, which is the target image, is an image of the specific person; The person detection device according to claim 1, further comprising at least one second threshold for classifying the image of the person to be calculated for similarity into the plurality of stages when the image is of a specific person. A person detection device according to claim 1;
Specifying an image of a tracking target person to be tracked from among a plurality of images of the similarity calculation target person who are similar to the image of the specific person presented on the output screen by the person detection device together with the stepwise display. , the designated image is received from a monitoring terminal operated by a supervisor, the specified image is registered as a tracking image, and the tracking image, which is an image of the registered tracking target person to be tracked, is used as the specific image. , a registration unit that causes the similarity calculation unit to extract the similarity of the image of the person to be calculated using the similarity calculation method;
a stage display indicating the caution stage in which the detection unit detects an image of the person to be tracked who is similar to the image of the person to be tracked detected from the similarity obtained from the similarity calculation unit; a tracking output screen generation unit that generates a tracking output screen for presentation together with the tracking output screen;
A person tracking device equipped with A plurality of thresholds for classifying the target image into a plurality of stages are set based on the similarity calculated by a similarity calculation unit that calculates the similarity of the target image to a specific image using one similarity calculation method. a stored threshold storage unit;
a registration unit that registers, as the specific image, an image of a person to be tracked who is designated by a monitoring terminal operated by a monitor;
One degree of similarity with respect to the specific image of the image of the person to be calculated, which is the target image extracted from each of the surveillance images periodically acquired by each of the plurality of surveillance cameras, is acquired from the similarity degree calculation unit. , based on the plurality of threshold values stored in the threshold value storage unit, the tracking target person is selected from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras. a detection unit that detects an image similar to the image and detects which of a plurality of attention levels the detected image of the person to be calculated for similarity is in;
Tracking for the detection unit to present an image of the similarity calculation target person that is similar to the image of the tracking target person detected from the similarity obtained from the similarity calculation unit, together with a stage display indicating the caution stage. a tracking output screen generation unit that generates an output screen;
A person tracking device equipped with The tracking output screen generation unit receives the stage number designation from the monitoring terminal, and selects a similarity level equal to or higher than a threshold value corresponding to the designated stage among the images of the similarity calculation target person detected by the detection unit. The person tracking device according to claim 3 or 4, wherein the person tracking device generates a narrowing down output screen that presents an image having the step display together with the step display. A person tracking device according to any one of claims 3 to 5,
the plurality of surveillance cameras;
the monitoring terminal operated by the monitoring person;
an analysis unit including the similarity calculation unit;
A person tracking system equipped with A person detection method for detecting a specific person from surveillance images obtained by multiple surveillance cameras,
The computer is
A plurality of thresholds for classifying the target image into a plurality of stages are set based on the similarity calculated by a similarity calculation unit that calculates the similarity of the target image to a specific image using one similarity calculation method. Store it in memory,
The similarity calculation is performed by calculating the similarity of the image of the target person, which is the target image extracted from each of the surveillance images periodically acquired by each of a plurality of surveillance cameras, to the image of the specific person, which is the specific image. obtained from the department,
Based on the plurality of threshold values stored in the memory, from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of monitoring cameras, images similar to the image of the specific person are selected. detecting an image, and detecting which of a plurality of attention stages the detected image of the similarity calculation target person is in;
generating an output screen for presenting the image similar to the detected image of the specific person together with a step display indicating the detected caution step;
Person detection method. A person tracking method for tracking a person to be tracked from surveillance images obtained by multiple surveillance cameras, the method comprising:
The computer is
A tracking target to be tracked from among a plurality of images of similarity calculation targets who are similar to the image of the specific person presented on the output screen together with the step display generated by the person detection method according to claim 7. The system accepts the designation of the person's image from the monitoring terminal operated by the monitoring person,
registering the specified image in the memory as a tracking image;
Using the tracking image, which is an image of the tracking target person who is to be tracked and is registered in the memory, as the specific image, the similarity calculation unit uses the similarity calculation method to calculate the similarity calculation target person. The similarity of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras to the image of the tracking target person, which is the specific image, is extracted. Obtained from the similarity calculation unit,
Based on the plurality of threshold values stored in the memory, from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras, images similar to the image of the tracking target person are selected. detecting an image of the similarity calculation target person, detecting which of the plurality of attention stages the detected image of the similarity calculation target person is in, and based on the similarity obtained from the similarity calculation unit; generating a tracking output screen that presents an image of the similarity calculation target person who is similar to the detected image of the tracking target person together with a step display indicating the detected caution level;
How to track people. A person tracking method for tracking a person to be tracked from surveillance images obtained by multiple surveillance cameras, the method comprising:
The computer is
A plurality of thresholds for classifying the target image into a plurality of stages are set based on the similarity calculated by a similarity calculation unit that calculates the similarity of the target image to a specific image using one similarity calculation method. Store it in memory,
registering an image of a person to be tracked designated by a monitoring terminal operated by a supervisor in the memory as the specific image;
obtaining from the similarity calculation unit the similarity of the image of the person to be calculated, which is the target image extracted from each of the surveillance images periodically acquired by each of a plurality of surveillance cameras, with respect to the specific image;
Based on the plurality of threshold values stored in the memory, from among the images of the similarity calculation target person extracted from the monitoring images of each of the plurality of surveillance cameras, images similar to the image of the tracking target person are selected. At the same time, it detects which of a plurality of attention stages the detected image of the similarity calculation target person is in, and detects the image of the tracking target person detected based on the similarity obtained from the similarity calculation unit. generating a tracking output screen for presenting an image of the similarity calculation target person similar to the image together with a stage display indicating the detected caution stage;
How to track people. A person detection program that causes a computer to operate as each part of the person detection device according to claim 1 or 2. A person tracking program that causes a computer to operate as each part of the person tracking device according to any one of claims 3 to 5.