JP4945477B2 - Surveillance system, person search method - Google Patents

Description

  The present invention relates to a monitoring system and a person search method in the monitoring system, and more particularly to a monitoring system and a person search method capable of performing a person search with high accuracy.

Conventionally, video surveillance systems have been installed in facilities visited by an unspecified number of people such as hotels, buildings, convenience stores, financial institutions, dams, and roads for the purpose of crime prevention and accident prevention.
In such a video monitoring system, a person or the like to be monitored is photographed by an imaging device such as a camera, and the video is transmitted to a monitoring center such as a management office or a security room.
A monitor resident in the monitoring center monitors the video, and pays attention or records / stores the video according to the purpose and necessity.

As a recording medium for recording and storing video in this video surveillance system, a sequential access storage medium such as a video tape has been used in the past.
However, in recent years, an example in which a randomly accessible storage medium represented by a hard disk drive (HDD) is used in a video surveillance system is increasing.
Randomly accessible storage media are becoming popular because they are easy to organize videos and are resistant to repeated use.

Furthermore, the capacity of this randomly accessible recording medium is increasing year by year.
With this increase in capacity, the amount of video data that can be recorded has increased dramatically. This made it possible to record for a long time.
However, an increase in the burden on the observer when visually checking a large amount of recorded video is becoming a problem.

Therefore, video surveillance systems having a search function for finding a desired video more easily are becoming widespread.
A video monitoring system having this search function may have a function of storing time and external sensor value information together with video.
An alarm such as a human sensor is often used as an external sensor. Also, the value output by this alarm is often a binary value of 0 or 1. Actually, when the alarm output is 1, the fact and the time are often stored as text information.
In such a conventional search function, a video is searched by using text information when an alarm output value of a human sensor becomes 1 as a search key.
In recent years, there are monitoring systems having a search function using image information as a search key.
As a kind of monitoring system using the image information as a search key, there is a recording apparatus having a person search function.

The person search function is a function that, when a user designates a person shown in a certain image, searches for other videos showing the same person as that person in the recording apparatus and presents the list to the user.
In this case, a person's “face” image is often used as a clue for determining the same person.
As a conventional system for searching for a person from such a face image, for example, referring to Patent Document 1, a person search system, a person tracking system, a person search method, and a person tracking that search for faces of a large number of people A method is disclosed (hereinafter referred to as Prior Art 1).
The person search function of the person search system as in the prior art 1 can be used for investigating the behavior of a specific person at other times or places, or for examining the trajectory of movement. For this reason, it is very effective.

An example of a system having a conventional person search function using a face image, such as the prior art 1, will be described with reference to FIG. 9 and FIG.
Hereinafter, for ease of explanation, an image designated by the user is referred to as a search input image, and a recorded image to be searched is referred to as a search target image.
An image presented as the same person image as a result of the search is referred to as a search output image.
Further, a list of thumbnails of search output images is referred to as a search output image list.

First, a control configuration of a conventional video surveillance system X including a recording device 201 having a person search function will be described with reference to FIG.
The network 200 is an IP network such as a dedicated line connecting each device, an intranet, or the Internet.
The recording device 201 is a recording device that stores image data in an HDD or the like for recording. In addition, the recording apparatus 201 has a person search function.
The imaging device 202 performs digital conversion processing on an image (moving image or still image) captured by a CCD or CMOS element, and outputs the converted image data via a network, such as a network camera or a surveillance camera. Device.
The monitoring terminal 203 is a device that acquires image data recorded in the recording device 201 via a network and displays the image data on a display unit such as a liquid crystal display or a CRT master monitor. In addition, the monitoring terminal 203 transmits the search input image to the recording device 201 via the network 200 by a built-in CPU or program. The monitoring terminal 203 can also display the search output image list transmitted from the recording apparatus 201 on the screen. Furthermore, the monitoring terminal 203 includes an OS (operating system) for this purpose and a user interface that is a program operating on the OS.

In the recording apparatus 201, the network unit 211, the storage unit 212, the face area detection unit 213, the face feature amount extraction unit 214, the face feature amount recording unit 215, and the face determination unit 216 are connected by, for example, a common bus. Configured.
The network unit 211 is a LAN interface or the like, and is a processing unit that performs input / output from the network 200. The network unit 211 receives image data input from the imaging device 202, receives a video distribution request from the monitoring terminal 203, receives a search input image, and transmits a video and search output image to the monitoring terminal 203.
The storage unit 212 includes a storage medium capable of random access, such as a RAM, a ROM, and an HDD, and a controller thereof, and reads and writes image data and other data on the recording medium. At the time of video recording, in addition to the image data, an ID (Identification, ID data) for retrieving the image data again is written to the storage medium.

The face area detection unit 213 is a part including a DSP (Digital Signal Processor), a program for the DSP, and the like. The face area detection unit 213 performs face detection on the input image data using a conventional image recognition process for general face detection. The face area detection unit 213 determines whether or not there is a face in the video, and when there is a face, calculates a coordinate of the area.
The face feature amount extraction unit 214 is a part including a DSP and the like, and performs face feature amount calculation using image recognition processing on the face region detected by the face region detection unit 213. Here, the facial feature amount indicates, for example, a numerical value of the size, shape, arrangement relationship, and the like of main components such as the shape and direction of the face outline, skin color, eyes, nose, and mouth.
The face feature amount recording unit 215 is a controller or the like for performing storage in the storage unit 212. The face feature amount recording unit 215 reads / writes the face feature amount calculated by the face feature amount extraction unit 214 to / from the recording medium of the storage unit 212.

The face determination unit 216 is a part that includes a DSP or the like, and is a part that performs the same person determination between the person in the search input image and the person in the recorded video during the search.
In addition, the face determination unit 216 collects recorded images determined to be the same person, and generates a search output image list. In this determination, a value called “face similarity” is calculated from the face feature amount obtained in each image, and the magnitude relationship between the calculated face similarity and a predetermined threshold stored in the storage unit 212 is calculated. Judgment based on.

With reference to FIG. 10, an operation when the user actually uses the person search function of the recording apparatus 201 in the conventional video monitoring system X will be described. FIG. 10 is an example of a user interface screen displayed on the display unit of the monitoring terminal 203.
In this user interface, the control unit of the monitoring terminal 203 detects an input signal of an input unit including a keyboard, a dedicated switch, a jog dial, a pointing device such as a mouse of the monitoring terminal 203, and moves and displays the pointer. Furthermore, the control unit of the monitoring terminal 203 detects that the user has pressed a button or the like that is an input key drawn on the screen of the display unit, and performs each process.

The user interface screen will be described more specifically below.
The playback video display unit 301 is an area for displaying the video being played back. In the reproduced video display unit 301, the monitoring terminal 203 decodes and displays the image data transmitted from the recording apparatus 201 via the network 200.

The reproduction operation unit 302 is an area where buttons for performing a reproduction operation are displayed. Each button on the playback operation unit 302 is assigned a unique playback type.
When it is desired to give a new playback command to the video displayed on the playback video display unit 301, the user selects a button corresponding to the playback command on the playback operation unit 302 with a pointing device or the like, and presses the button. Press. The control unit of the monitoring terminal 203 detects button presses in the user interface and performs operations corresponding to various playback types.
For example, a “two left triangles” button indicates fast rewind of the video, “one left triangle” indicates normal video rewind playback, and a “square” button stops the video. / "Pause", "Right triangle" means playback of video at normal speed, "Right triangle" button indicates fast forward of video.
The camera switching operation unit 303 is an area for displaying a camera switching operation button. Each button on the camera switching operation unit 303 is assigned with a camera to be recorded, and is pressed when switching the video displayed on the playback video display unit 301 to the video of another camera. To do.

The search input image specifying unit 304 is an area for displaying and specifying the search input image. This designation is performed by detecting that the “capture” button displayed in the search input image designation unit 304 is pressed.
When the control unit of the monitoring terminal 203 detects that the user has pressed the “capture” button, the image displayed on the playback video display unit 301 is displayed as a thumbnail as a search input image.

The search operation unit 305 is an area for performing a search on an image displayed on the search input image specifying unit 304 by pressing a “search” button on the search operation unit. Although not shown in this example, the search operation unit 305 may be provided with buttons for specifying the date and time and the camera.
When the control unit of the monitoring terminal 203 detects that the user has pressed the “search” button, if there is an image (search input image) displayed in the search input image specifying unit 304, the search unit 201 performs this search. The frame number related to the image for use is transmitted via the network 200.
The control unit of the recording apparatus 201 receives the frame number related to the search input image, reads the search image related to the frame number from the storage unit 212, and executes the person search function using each unit as described above. Then, a search output image (and its frame number) that is an image including the searched person is transmitted to the monitoring terminal 203 via the network 200.

The search output image list display unit 306 is an area for displaying a list of search output images as thumbnails.
When one of the search output images displayed here is pressed, the display video of the playback video display unit 301 is switched to the pressed image. This switching is called cueing.

  As described above, the person search function using “face” as a clue is very convenient because it is possible to easily find a target person image from a large number of search target images.

JP 2003-281157 A

However, although the image recognition technology such as the prior art 1 has excellent recognition accuracy for the face image projected from the front direction, the recognition accuracy for the face image projected from the other direction is low. In addition, the size (resolution) of the face in the image greatly affects the recognition accuracy, and when the face size is small, the recognition accuracy is low.
Therefore, in a video surveillance system that is equipped with a wide-angle lens and often projects the subject small from diagonally above, an erroneous search, i.e., wrongly recognizing different people as the same person or wrongly recognizing the same person as a different person. In other words, there is a problem that the accuracy is low.

  This invention is made | formed in view of such a condition, and makes it a subject to eliminate the above-mentioned subject.

People search method of the present invention, the person searching method in a monitoring system for recording an image obtained from each of the image data of a plurality of images recorded, photographing time information, the photographing position information, the face feature amount, and randomly clothes information 4 items corresponding to four combinations of conditions , each of which is stored in an accessible storage medium and has the proximity of whether the shooting time information is near or not, and the identity of whether the clothing information is the same or not time + cLOTHING weight setting value group of numbers of, corresponding to the combination of the photographing time information and the vicinity of or near or non vicinity, condition 4 kinds of the photographing position information or non-neighboring or Locality vicinity four of numbers time + position weight setting value group, and neighborhood of whether the photographing position information near or non-neighboring, the clothes information conditions are four of the same or non-identical or identity combinations that Receiving at least one input of the corresponding four numbers consisting of positions + clothes weight setting value group in advance, when receiving a search instruction with the specified search input image, for each of the plurality of images the recording, One of four numerical values of the input weight setting value group is selected according to the relationship between the recorded image and the search input image, and the recorded image and the facial feature amount of the search input image the difference with the overall similarity obtained by weighted numerical values the selection, characterized in that to determine the identity of the recorded person.
In the person search method of the present invention, the time + clothing weight setting value group, the time + position weight setting value group, and the position + clothing weight setting value group have the four numeric input fields arranged in a matrix. The time + clothing weight setting table, the time + position weight setting table, and the position + clothing setting table are input or changed by the user through the weight setting screen, and the search instruction is the time + clothing weight setting value group, Including time + position weight setting value group and position + clothing weight setting value group, arbitrary plural designations, the proximity of the shooting time information, the identity of the clothing information and the proximity of the shooting position information are The total similarity is weighted by the difference between the facial feature values by multiplying the selected values from the plurality of designated weight setting value groups. Also And characterized in that.
According to the person search method of the present invention, the time + clothing weight setting value group, the time + position weight setting value group, and the position + clothing weight setting value group have the following seven conditions: (2) The person with the same clothes may be the same person even at a time that is not near (3) The person with non-identical clothes may not be the same person at a time near (4) It is highly likely that the person in the vicinity and the person in the vicinity are the same person. (5) The person in the vicinity and the person in the non-neighbor position are highly likely not to be the same person. (6) 3. The person according to claim 1 or 2, wherein the person at the position is set based on the possibility that the person at the position is the same person (7) the position near the person and the person at the same clothes may be the same person. characterized in that it is a search method
The monitoring system of the present invention is a monitoring system that includes an imaging device, a recording device, and a monitoring terminal to record an image, and the recording device extracts a facial feature amount from an image captured by the imaging device. A quantity extraction unit; a clothing feature quantity extraction unit that extracts a clothing feature quantity from an image captured by the imaging device; and image data of a plurality of recorded images, shooting time information, shooting position information, and the face feature quantity extraction unit. For each of the recorded face image and a plurality of recorded images , storage means for storing at least one or more of the clothes information extracted by the clothing feature value extraction means in a randomly accessible storage medium ; A time consisting of four numerical values corresponding to a combination of four conditions: proximity of whether the shooting time information is near or non-neighbor and identity of whether the clothing information is the same or non-identical + A set of weights, consisting of four numerical values corresponding to four combinations of conditions: proximity of whether the photographing time information is near or non-neighbor and proximity of whether the photographing position information is near or non-neighbor. Four numerical values corresponding to combinations of four conditions: time + position weight setting value group, proximity of whether the shooting position information is near or non-near, and identity of whether the clothing information is the same or non-identical The position + clothing weight set value group consisting of: is selected according to the relationship between the recorded image and the search input image, and the difference between the recorded image and the face feature amount of the search input image is And a comprehensive judgment means for judging the identity of the recorded person using the total similarity obtained by weighting with the selected numerical value .

  According to the present invention, it is possible to provide a video surveillance system and a person search method with improved accuracy.

<First Embodiment>
[Control configuration]
With reference to FIG. 1, a control configuration of a video surveillance system Y including a recording device 101 having a person search function according to a person search method according to an embodiment of the present invention will be described.
The network 100 includes a LAN, an optical fiber, c. A line capable of data communication such as a link, a wireless LAN, and a mesh network. The network 100 may be an IP network such as a dedicated line, an intranet, or the Internet.
The recording device 101 is a recording device that stores image data in an HDD or the like for recording. In addition, the recording apparatus 101 has a face search function as in the conventional recording apparatus 201, and further has a clothing detection function. By combining this clothing detection function and the face detection function, a person search method with improved accuracy according to the embodiment of the present invention can be provided. This person search method will be described later.
The imaging device 102 is an image capturing device having the same function as the imaging device 202.
The monitoring terminal 103 is a device having the same function as the monitoring terminal 203, and may be realized by a program stored in a storage unit of a PC (personal computer) such as a PC / AT compatible machine or a MAC, or a dedicated terminal You may implement | achieve as a monitoring terminal device. The monitoring terminal 103 also has a user interface for using the person search method of the recording apparatus 101.

  The recording apparatus 101 includes a network unit 111, a storage unit 112 (storage unit), a face region detection unit 113, a face feature amount extraction unit 114 (face feature amount extraction unit), a face feature amount recording unit 115, a clothing region detection unit 116, The clothing feature value extraction unit 117 (clothing feature value extraction unit), the clothing feature value recording unit 118, the shooting time recording unit 119, the shooting position recording unit 120, and the overall determination unit 121 (overall determination unit) are connected by, for example, a common bus. Configured to do.

The network unit 111 is a processing unit such as a LAN interface that performs input / output from the outside of the apparatus. The network unit 111 receives image data transmitted from the imaging apparatus 102. Further, the network unit 111 receives a video distribution request and a search input image from the monitoring terminal 103. Further, the network unit 111 transmits image data and search output image data to the monitoring terminal 103.
The storage unit 112 includes a storage medium such as a RAM, a ROM, a flash memory, an optical disk, a magnetic tape, and an HDD, an intelligent controller, a CPU, an MPU, and the like. The storage unit 112 reads and writes image (video) data and other data to and from a recording medium. When recording video, in addition to image data, ID (identification) for retrieving image data again, time code, frame number, various tag data, video ID such as hash data, database, etc. Write using. In addition, the storage unit 112 can retrieve a video ID from a database by using a controller, a CPU, or an MPU, and read / write from / to a storage medium.
The face area detection unit 113 includes a DSP having the same function as the face area detection unit 213, a program for the DSP, and the like. The face area detection unit 113 performs face detection using image recognition technology on the input image data. Furthermore, the face area detection unit 113 determines whether or not there is a face in the video, and if a face exists, calculates the coordinates of the area.
The face feature amount extraction unit 114 includes a DSP having the same function as the face feature amount extraction unit 214. Further, the facial feature amount extracted by the facial feature amount extraction unit 114 is, for example, from the shape and direction of the face outline, the color of the skin, the size, shape, arrangement relationship, etc. of main components such as eyes, nose and mouth. It is possible to use a vector component or a statistic that statistically shows a difference for each individual. Further, in the facial feature quantity extraction unit 114 according to the embodiment of the present invention, the type and number of facial feature quantities to be used are arbitrary. Furthermore, it is possible to determine the sexes with a high probability from the characteristics of the hair length and the estimated skeleton (such as the size of the jaw).
The face feature amount recording unit 115 includes a controller or the like for performing storage in the storage unit 112. The face feature amount recording unit 115 reads / writes the face feature amount calculated by the face feature amount extraction unit 114 to / from the recording medium of the storage unit 112.

The clothing region detection unit 116 includes a DSP, a program for the DSP, and the like, and performs coordinate calculation of the clothing region using image recognition technology on the input image data. As this image recognition technique, a known technique that can be used for calculating the amount of clothes can be used. For example, a person's contour is extracted using dynamic programming, etc., and clothes (clothes) are derived from the color distribution within the contour and the frequency characteristics of the texture (frequency distribution when FFT, wavelet transform, etc. are performed). It is possible to use a technique for detecting that it is. Furthermore, the height of the back can be estimated from the coordinates of the clothing area, and it can be determined whether the child is an adult or a child.
The clothing feature value extraction unit 117 calculates the clothing feature value using the image recognition technique for the clothing region detected by the clothing region detection unit 116. Here, the clothing feature amount includes, for example, the color distribution and frequency characteristics of the clothing described above. In the present invention, the type and number of clothing feature values used are arbitrary. In addition, the clothing feature quantity extraction unit 117 can also determine a feature such as right front or left front, a feature such as a curved outfit, and a feature such as a one-piece. This makes it possible to determine with high probability whether the person wearing the clothes is male or female.
The clothing feature value recording unit 118 includes a controller for storing data in the storage unit 112. The clothing feature value recording unit 118 reads / writes the clothing feature value calculated by the clothing feature value extraction unit 117 from / to the recording medium of the storage unit 112.

The shooting time recording unit 119 reads / writes the shooting time information, which is the time when the image is shot, to the recording medium in the storage unit 112.
As this photographing time information, it is preferable to store an accurate time by using, for example, GPS (Global Positioning System), NTP (Network Time Protocol) or the like via the network unit 111.
Further, when there is an error in the time of the reference clock in each imaging apparatus 102, the error is measured and corrected and stored in accordance with the time of the recording apparatus 101 as the “master clock”.

  The photographing position recording unit 120 is configured to include a controller or the like for performing storage in the storage unit 112. The shooting position recording unit 120 reads / writes video shooting position information from / to a recording medium in the storage unit 112. The shooting position information may be physical information such as GPS position information of a camera, for example, or may be reference information for a separately provided position calculation unit such as a camera installation number.

The overall determination unit 121 performs the same person determination between the person in the search input image and the person in the recorded video during the search, collects the recorded images determined as the same person, and generates a search output image list.
In this determination, a value called “face similarity” is calculated from the facial feature amount obtained for each image, and the weight value obtained from shooting time information and shooting position information, and clothing information to be described later, for the calculated face similarity. Is determined based on a magnitude relationship such as a value obtained by adding (hereinafter referred to as total similarity) and a predetermined threshold value set in advance. Details of the face similarity and other processes will be described later.
In addition, the comprehensive determination unit 121 includes temporary storage means such as a RAM, a register, a flash memory, and an HDD for such determination.

The temporary storage unit of the comprehensive determination unit 121 may be provided in a specific area of the storage unit 112 or a storage medium of the storage unit 112.
In addition, the comprehensive determination unit 121 may be realized as a program executed by the control unit. In this case, the temporary storage unit holds the value of the program variable or a file.
Furthermore, the face area detection unit 113, the face feature quantity extraction unit 114, the face feature quantity recording unit 115, the clothing area detection unit 116, the clothing feature quantity extraction unit 117, the clothing feature quantity recording unit 118, the shooting time recording unit 119, the shooting position. The recording unit 120 may also be realized as a program executed by the control unit.
In this case, computation processing that requires high-speed responsiveness such as feature quantity extraction and large computation performance in SIMD (single instruction multiple data) format is used for other various DSPs and GPUs (graphic processing). -You may make it process with the chip | tip or board provided with the unit.

  In addition, the video monitoring system Y includes a control unit such as a CPU that controls each of the processes described above. The program executed by the control unit may be used to implement each unit using hardware resources, and may be stored in the storage unit 112 or stored in a ROM or flash memory in the control unit. May be.

[Method for improving the accuracy of person search methods]
As a result of repeated experiments and examinations, the inventor of the present invention is effective to add shooting time information, shooting position information, clothing information, etc. to the determination of the same person in order to improve the performance of the person search function. We found something and came to complete the present invention.
Specifically, the same person can be reliably determined by comparing shooting time information, shooting position information, clothing information, and the like between the search input image and the search target image.
Here, for the shooting time information and the shooting position information, the proximity (temporal and spatial proximity) of whether or not the person exists in the near field is determined.
The clothing information is a vector of clothing feature values and the like, and the identity of whether or not the person is the same is determined based on the clothing features.
The determination result is determined by applying the weight value calculated using the following conditions (1) to (7) to shooting time information, shooting position information, clothing information, and the like.
(1) People in the same clothes are likely to be the same person at a nearby time (2) People in the same clothes may be the same person even at a time not nearby (3) Non-identical at a nearby time There is a high possibility that the person in the clothes is not the same person (4) The time in the vicinity and the person in the vicinity are likely to be the same person (5) The time in the vicinity and the person in the non-neighbor position may not be the same person High (6) Non-near time, person in neighborhood position may be the same person (7) Person in neighborhood position, same clothes may be the same person The calculated weight is used as the facial feature value By multiplying the judgment result of the same person used, it has become possible to dramatically improve the accuracy of the person search result.
In addition, even if it judges a part of (1)-(7), a person search precision can be improved.

[Database]
Next, the feature quantity database 801 according to the embodiment of the present invention will be described with reference to FIG.
In order to realize the above-described technique for improving accuracy, the recording apparatus 101 according to the embodiment of the present invention obtains a video ID, facial feature amount, clothing information, shooting time information, and shooting position information during recording. Remember.
In order to store such information, the recording apparatus 101 includes a feature amount database 801 in the storage medium of the storage unit of the storage unit 112.
The feature amount database 801 can be configured in a table format as shown in FIG. 2, for example, but may be any type.

Among the stored contents of the feature database 801, the video ID column 811 stores the video ID. As this video ID, a time code indicating a recording time or a frame number can be used. The frame number is, for example, a number uniquely assigned from the beginning of one stream, and a value such as a continuous value that monotonously increases may be used.
The face feature value column 812 stores the face feature value calculated by the face feature value extracting unit 114. When there are a plurality of facial feature quantities to be stored, this is a plurality of columns.
The clothing feature value column 813 stores the clothing feature value calculated by the clothing feature value extraction unit 117. When there are a plurality of clothing feature values to be stored, this becomes a plurality of columns.
The shooting time column 814 stores shooting time information.
The shooting position column 815 stores shooting position information. GPS position information and the like are also stored here.

  For writing or reading of the feature amount database 801, a database daemon (background execution program or the like) or service such as SQL is executed by the CPU of the storage unit 112 or the like. This daemon or service can read, write (store), add information, etc., with respect to the storage contents of the feature amount database 801 by commands from each component.

The face feature amount recording unit 115 stores the video ID and the face feature amount by using the next row of the last feature row of the feature amount database 801, that is, the unstored row, as a write row when input from the face feature amount extraction unit 114.
The clothing feature value recording unit 118, the shooting time recording unit 119, or the shooting position recording unit 120 stores the clothing feature value, the shooting time, and the shooting position for the row that is the writing line.
Further, even using the program executed by the comprehensive determination unit 121, it is possible to read (reference) or write (store) each row of the feature amount database 801.

[Processing during video recording]
Next, with reference to FIG. 3, the flow of processing when video recording is actually performed by the recording apparatus 101 according to the embodiment of the present invention will be described.
At the time of this video recording, the shooting time information, the shooting position information, and the clothing information are also stored as described above.
In addition, the processing for recording the video is performed in cooperation by each unit according to a program stored in a storage medium such as a ROM, flash memory, or HDD in the storage unit 112.

(Step S701)
In step S701, the network unit 111 performs video reception processing.
Specifically, the network unit 111 receives image data input from the imaging apparatus 102 via the network 100 and outputs the image data to the storage unit 112.

(Step S702)
In step S702, the storage unit 112 performs image data and video ID storage processing.
That is, the storage unit 112 records the input image data on a randomly accessible recording medium such as an HDD.
The storage unit 112 also generates and records a video ID for taking out the video again. Then, the image data is output to the face area detection unit 113.
In the following description of the flowchart, it is assumed that a video ID is attached to an image even if there is no explicit description.

(Step S703)
In step S703, the face area detection unit 113 performs face detection processing on the input image data.
In this face detection process, a known value is used to detect a face (image) using a color such as brightness and skin color in the video, and other feature quantities, and a detected value (detectability, p Outputability).

(Step S704)
In step S704, the comprehensive determination unit 121 determines whether a face has been detected.
Specifically, the overall determination unit 121 determines whether a face has been detected using a predetermined threshold. That is, when the above-described detection value exceeds a predetermined threshold, it is determined as Yes, and otherwise it is determined as No.
In the case of Yes, the comprehensive determination unit 121 advances the process to step S705.
In No, the comprehensive determination part 121 advances a process to step S717.

(Step S705)
In step S705, the face area detection unit 113 performs face area coordinate calculation processing.
Specifically, when a face is detected, the face area detection unit 113 calculates the coordinates of the area based on the coordinate value of the pixel rectangle.
After that, the face area detecting unit 113 outputs the calculated coordinate values to the face feature amount extracting unit 114 together with the image data.

(Step S706)
In step S706, the face feature amount extraction unit 114 performs face feature amount calculation processing.
Specifically, the face feature quantity extraction unit 114 calculates a face feature quantity for a face area image obtained from the input video and face area coordinates.
The face feature amount extraction unit 114 outputs the calculated face feature amount together with the image data to the face feature amount recording unit 115.

(Step S707)
In step S707, the face feature amount recording unit 115 performs a face feature amount and video ID storage process.
Specifically, the face feature amount recording unit 115 records the input face feature amount in the face feature amount column 812 of the feature amount database 801 of the storage unit 112. At that time, the video ID is also recorded in the video ID column 811.
Then, the face feature amount recording unit 115 outputs the image data to the clothing region detection unit 116.

(Step S708)
In step S708, the clothing region detection unit 116 performs clothing detection processing on the input image data.
In this clothing detection process, the clothing region detection unit 116 outputs a clothing detection value using various clothing detection algorithms as described above. At that time, it is preferable to use after adjusting the detection parameters and the like according to the characteristics such as the white balance and resolution of the imaging device 102 stored in advance in the storage unit 112.

(Step S709)
In step S709, the overall determination unit 121 determines whether clothing has been detected.
Specifically, the overall determination unit 121 determines Yes when the detected value of clothing is greater than a predetermined threshold stored in the storage unit 112, and determines No otherwise.
If Yes, that is, if clothing is detected, the overall determination unit 121 proceeds to step S710.
If No, that is, if clothes are not detected, the overall determination unit 121 proceeds to step S713.

(Step S710)
In step S710, the clothing region detection unit 116 performs clothing region coordinate calculation processing.
Specifically, the clothing area detection unit 116 calculates the coordinate value of the clothing area from the input image data using the rectangular coordinates of the video pixels, the shape of a specific block, the vector value, the coordinates of the curve, and the like.
The clothing region detection unit 116 outputs the calculated coordinate values to the clothing feature value extraction unit 117 together with the image data.

(Step S711)
In step S711, the clothing feature value extraction unit 117 performs a clothing feature value calculation process.
Specifically, the clothing feature value extraction unit 117 calculates a clothing feature value for the region image data of the clothing using the various algorithms described above.
The clothing feature value extraction unit 117 outputs the calculated clothing feature value to the clothing feature value recording unit 118 together with the image data.

(Step S712)
In step S712, the clothing feature value recording unit 118 performs a clothing feature value storage process.
Specifically, the clothing feature value recording unit 118 stores the input clothing feature value in the cell of the row with the matching video ID in the clothing feature value column 813 of the feature value database 801 of the storage unit 112.
After performing this process, clothing feature value recording unit 118 advances the process to step S715.

(Step S713)
In step S713, the clothing area detection unit 116 performs information output processing without clothing detection.
Specifically, the clothing area detection unit 116 outputs “no detection” information (clothing detection no information) to the clothing feature value recording unit 118 when no clothing is detected.

(Step S714)
In step S714, the clothing feature value recording unit 118 performs information storage processing without clothing detection.
Specifically, when the information “no detection” is input from the clothing area detection unit 116, the clothing feature value recording unit 118 includes the clothing feature value column 813 of the feature value database 801 as undetected clothing. Record in the cell of the row with the matching video ID.

(Step S715)
In step S715, the shooting time recording unit 119 performs shooting time storage processing.
Specifically, the shooting time recording unit 119 stores the shooting time of the video in the shooting time column 814 of the feature amount database 801 of the storage unit 112.
The photographing time may be a time based on a time series unified in the system.
That is, the time at which the image is generated by the imaging apparatus 102 may be used, the time at which the video is received by the recording apparatus may be used, or the time at which the process reaches step S715 is used. Also good.
Further, as described above, accurate time can be stored using GPS or the like.

(Step S716)
In step S716, the shooting position recording unit 120 performs shooting position storage processing.
Specifically, the shooting position recording unit 120 records the shooting position of the video in the shooting position column 815 of the feature amount database 801 of the storage unit 112.
Any information can be used as the photographing position used in the photographing position recording unit 120 as long as the information indicates the degree of relative distance between the image capturing apparatuses 102. For example, the X coordinate, Y coordinate, Z coordinate (altitude), etc. on the ground plane can be recorded.
In addition, as described above, an accurate position using the GPS or the like can be stored as the shooting position of the video.
Further, a code (or number) indicating the relative position of the camera is recorded instead of the shooting position, and the actual camera position is obtained from information indicating the actual physical position of the camera. You may make it calculate.

(Step S717)
In step S717, the comprehensive determination unit 121 performs reception waiting state processing.
Here, the comprehensive determination unit 121 enters a state of waiting for reception of the next video. Specifically, when the network unit 111 has not received the stored image data, the comprehensive determination unit 121 stands by until it is received.
Also, when the comprehensive judgment unit 121 receives a recording end processing command from the monitoring terminal 103 or detects that a power button of an input unit (not shown) is pressed, the overall judgment unit 121 ends all processing and turns on the power. Shut down.

In steps S <b> 706 and S <b> 707, the clothing area detection unit 116 can also input the face area coordinates from the face feature amount recording unit 115.
In this case, it is possible to improve the accuracy of detection of the clothing area using the condition that “there is often clothing under the face”.

In the person search method according to the embodiment of the present invention, the case where the number of faces detected in step S703 is one is shown.
However, when a plurality of persons are shown in the same video and a plurality of faces are detected, steps S704 to S716 may be repeated as many times as the number of detections.

[User interface]
Next, with reference to FIG. 4, a user interface screen for the user to execute the person search method according to the embodiment of the present invention will be described.
As described above, the recording apparatus 101 according to the embodiment of the present invention stores the video ID, the face feature value, the clothing feature value, the shooting time information, and the shooting position information.
From the stored information, the control unit of the monitoring terminal 103 detects a user instruction using the user interface of the monitoring terminal 103, and transmits this instruction to the recording apparatus 101 via the network 100 as a command. The recording apparatus 101 that has received this command can execute the person search method.
FIG. 4 is a user interface screen displayed on the display unit of the monitoring terminal 103.

In this user interface, like the above-described conventional monitoring terminal 203, the input unit of the monitoring terminal 103 detects a user input and performs display on the display unit and various processes.
Here, the area | region with the same code | symbol is a display area | region for implement | achieving the function equivalent to the user interface of the conventional person search function of FIG.
The user interface according to the embodiment of the present invention is different from the conventional user interface in that the search operation unit 505 is used.
The search operation unit 505 will be described in more detail below.

The search operation unit 505 is an area for instructing to perform a search using an image displayed on the search input image specifying unit 304 when the user presses a search button on the search operation unit 505. In addition, the search operation unit 505 may be provided with means for specifying a search range such as date and time or a camera (not shown).
The weight type selection buttons 511 to 513 are buttons for giving selection of the type of weight used for determination of the same person.
Three types of weights are used: a weight based on the relationship between shooting time information and clothing information, a weight based on the relationship between shooting time information and shooting position information, and a weight based on the relationship between shooting position information and clothing information. be able to.
A plurality of types of weights can be selected, and the selection status when the search button is pressed is reflected in the search process described below.
The weight setting button 514 activates a weight setting screen when the button is pressed. The weight setting screen will be described later.

  Next, referring to FIG. 5, an example of a weight setting screen in the user interface for executing the person search method according to the embodiment of the present invention is shown. FIG. 5 is a screen that is displayed on the display unit by the control unit of the monitoring terminal 103 when the pressing of the weight setting button 514 shown in FIG. 4 is detected.

The time + clothing weight setting table 610 is an area for giving a weight setting based on the relationship between the shooting time information and the clothing information.
Setting value input fields 611 to 614 are fields for inputting weight setting values.
The weight set value is a weight value used in a weight calculation process described later.
As the set value of each weight, for example, a ratio value from 0.0 to 2.0 can be set. When the value is close to 0.0, the calculated value becomes small. That is, no weighting is performed at 0.0.
Conversely, when the value is close to 2.0, the calculated value increases.
The value calculated by adjusting the weight set value is applied to determine the above-described proximity, that is, “identity of whether or not the person is the same”.

The set value input field 611 is used to input a set value of weight when the shooting time of the input image and the shooting time of the target image are close, and the clothes of the person of the input image and the clothes of the person of the target image are the same. It is a column to do.
The set value input field 612 indicates the set value of the weight when the shooting time of the input image and the shooting time of the target image are not close, and the clothes of the person of the input image and the clothes of the person of the target image are the same. It is a column to input.
The set value input field 613 indicates a weight setting value when the shooting time of the input image and the shooting time of the target image are close, and the clothes of the person of the input image and the clothes of the person of the target image are not the same. It is a column to input.
The setting value input field 614 is a weight setting value when the shooting time of the input image and the shooting time of the target image are not close, and the clothes of the person of the input image and the clothes of the person of the target image are not the same. Is a column for inputting.
Hereinafter, the set value group of weights input to the time + clothing weight setting table 610 is referred to as time + clothing weight setting value.

Similarly, the time + position weight setting table 620 is an area for giving a weight setting based on the relationship between the shooting time information and the shooting position information.
Setting value input columns 621 to 624 are columns for inputting weight setting values. As for the set value of the weight, a value from 0.0 to 2.0 can be set similarly to the above-described set value input fields 611 to 614.
The setting value input field 621 is used to input a weight setting value when the shooting time of the input image and the shooting time of the target image are close, and the shooting position of the input image and the shooting position of the target image are close. It is a column.
The setting value input field 622 is used to input a weight setting value when the shooting time of the input image and the shooting time of the target image are not close and the shooting position of the input image and the shooting position of the target image are close. It is a column to do.
The setting value input field 623 is used to input a weight setting value when the shooting time of the input image and the shooting time of the target image are close, and the shooting position of the input image and the shooting position of the target image are not close. It is a column to do.
The set value input field 624 is a weight setting value when the shooting time of the input image and the shooting time of the target image are not close, and the shooting position of the input image and the shooting position of the target image are not close. It is a column to input.
Hereinafter, the weight set value group input to the time + position weight setting table 620 is referred to as time + position weight set value.

Similarly, the position + clothing weight setting table 630 is an area for inputting a weight setting value based on the relationship between the shooting position information and the clothing information.
As for the set values, values from 0.0 to 2.0 can be set in the same manner as the set value input fields 611 to 614 and the set value input fields 621 to 624 described above.
Among the set value input fields 631 to 634, 631 is a weight when the shooting position of the input image and the shooting position of the target image are close, and the clothes of the person of the input image and the clothes of the person of the target image are the same. This is a column for inputting the set value.
The set value input field 632 indicates the set value of the weight when the shooting position of the input image and the shooting position of the target image are not close and the clothes of the person of the input image and the clothes of the person of the target image are the same. This is a field for input.
The set value input field 633 indicates the set value of the weight when the shooting position of the input image and the shooting position of the target image are close, and the clothes of the person of the input image and the clothes of the person of the target image are not the same. This is a field for input.
The set value input field 634 is a setting value of weight when the shooting position of the input image and the shooting position of the target image are not close, and the clothes of the person of the input image and the clothes of the person of the target image are not the same. Is a column for inputting.
Hereinafter, the weight set value group input to the position + clothing weight setting table 630 is referred to as a position + clothing weight setting value.

The apply button 641 is a button for reflecting the contents input to each setting table when pressed by the user.
The close button 642 is a button for ending this setting screen and returning the processing to the operation screen when pressed by the user.
The control unit of the monitoring terminal 103 detects that the user has pressed each button described above, and performs each process.

Note that in the weight setting tables of the time + clothing weight setting table 610, the time + position weight setting table 620, and the position + clothing weight setting table 630, the vicinity 2 and the non-neighbor 2 for the shooting time information and the shooting position information. Shown that there is an input field for the category. However, the number of divisions may be increased so that it can be set more finely.
Further, in this example, the description has been made such that the threshold value for dividing each section is set to a predetermined value. However, the user may be able to freely set the threshold for dividing this division.

[Person search processing]
Next, referring to the timing chart of FIG. 6, information transmission / reception via the network 100 of the person search processing between the recording apparatus 101 and the monitoring terminal 103 to which the person search method according to the embodiment of the present invention is applied. The flow will be described. The following information transmission / reception processing is executed in cooperation with the control unit and the like of the recording device 101 or the monitoring terminal 103.
It is assumed that the monitoring terminal 103 has input the image data from the recording apparatus 101 before this processing and has completed the display processing shown in FIG. That is, the search input image designation unit 304 is in a state where a search input image is displayed. At this time, the monitoring terminal 103 also receives the search input image and the video ID that is the frame number related thereto.
Here, the monitoring terminal 103 detects that the user has pressed the search button of the search operation unit 505 in the above-described user interface, and starts the following person search process.

(Timing T101)
At timing T101, the monitoring terminal 103 performs search input image transmission processing. That is, when the monitoring terminal 103 detects that the transmission button is pressed, the monitoring terminal 103 transmits the video ID of the search input image displayed on the search input image specifying unit 304 to the recording apparatus 101. Further, associated information for search such as the state of a check box is also transmitted to the recording apparatus 101.
In step S201, the recording apparatus 101 detects a face image from the search input image as pre-search processing.

(Timing T102)
Next, at timing T <b> 102, the recording apparatus 101 performs face detection-free information transmission processing.
Here, the recording apparatus 101 determines in step S202 whether a face image has been detected in the search preprocessing.
When Yes, that is, when a face image is detected, this face detection-less information transmission processing is not performed, and actual face feature amount search and clothing feature amount search processing are performed.
If No, that is, if no face image is detected, information on no face detection that is “no detection” information is transmitted to the monitoring terminal 103.

  The monitoring terminal 103 displays “no face can be detected” or the like when the no face detection information is “no detection” (for the face).

When the recording device 101 detects a face (Yes in step S202), the recording device 101 performs a process of searching for a facial feature value and a search for a clothing feature value according to the search input image and the settings of the other search operation unit 505 (step S203). ).
Further, the recording apparatus 101 performs an actual search process using the set weight value or the calculated weight value, and stores the search output image, which is an image found by searching from the search target image, in the storage unit 112. (Step S204).

(Timing T103)
Next, at timing T103, the recording apparatus 101 performs search output image list transmission processing.
Here, the recording apparatus 101 transmits a search output image list that is a list of search output images to be output to the monitoring terminal 103.

The monitoring terminal 103 displays a list of the search output images on the search output image list display unit 306 of the user interface displayed on the display unit.
Thereby, the person search process is completed.

Next, the actual processing in the recording device 101 and the monitoring terminal 103 will be described more specifically with reference to the flowcharts of FIG. 7 and FIGS.
First, the process of the monitoring terminal 103 will be described with reference to FIG.

[Processing of the monitoring terminal 103]
(Step S901)
In step S901, the control unit of the monitoring terminal 103 performs a search button press detection process.
Specifically, in the user interface of FIG. 4 displayed on the display unit of the monitoring terminal 103, it is detected that the search button of the search operation unit 505 has been pressed by the user using a pointing device or the like.

(Step S902)
In step S902, the control unit of the monitoring terminal 103 performs a search input image transmission process.
Specifically, the video ID related to the image data of the search input image of the search input image specifying unit 304 is transmitted to the recording apparatus 101 via the network 100. This transmission timing is T101.
By transmitting the image data, the recording apparatus 101 is requested to execute a person search. This is a search execution request.
In the case of a search execution request, the selection status of the weight type selection buttons 511 to 513 when the search button is pressed is further transmitted. These pieces of information are used as search accompanying information.
Thereafter, the control unit of the monitoring terminal 103 waits until receiving no face detection information or a search output image list from the recording apparatus 101.
Note that the control unit of the monitoring terminal 103 can also display that the connection with the recording apparatus 101 is not possible when neither the face detection-free information nor the search image list is received for a predetermined time.

(Step S903)
In step S903, the control unit of the monitoring terminal 103 receives information from the recording apparatus 101, and determines whether this information is information without face detection.
Specifically, it is determined whether the face detection-free information transmitted from the recording apparatus 101 is received at the timing of transmitting the face-detection-free information in T102, or whether the search output image list is received (not the information without face detection). To do.
As described above, the no face detection information is information indicating “no detection” in which no face is detected by the recording apparatus 101. The search output image list is image data output from the recording apparatus 101 after a face is detected and various processes are performed.
If Yes, that is, “no detection”, the control unit of the monitoring terminal 103 advances the process to step S906.
If the result is No, that is, the search output image, the control unit of the monitoring terminal 103 advances the process to step S904.

(Step S904)
In step S904, the control unit of the monitoring terminal 103 performs a search output image list reception process.
Specifically, the monitoring terminal 103 receives the search output image list transmitted from the recording apparatus 101 at the timing T103 via the network 100.

(Step S905)
In step S905, the control unit of the monitoring terminal 103 performs thumbnail list display processing.
Specifically, the received contents are displayed as thumbnails on the search output image list display unit 306 shown in FIG.
In the person search method according to the embodiment of the present invention, conditions can be narrowed down using clothing information and weighting, so that it is expected that the number of search images can be reduced (false positives and false positives are less) than the conventional person search method. it can.
Thereby, the effect that the monitoring burden of a user can be reduced is acquired.

(Step S906)
In step S <b> 906, the control unit of the monitoring terminal 103 performs “no detection” display processing when the information that has received no face detection information is received.
Specifically, the control unit of the monitoring terminal 103 displays a message on the search output image list display unit 306 in FIG. 4 and the like indicating that no face has been detected in the search input image.

(Step S907)
In step S907, the control unit of the monitoring terminal 103 performs a user operation waiting process.
Specifically, the control unit of the monitoring terminal 103 performs processing such as display / update of the user interface of the display unit, and enters the waiting state for the next user operation with respect to the search operation unit 505.
Above, the process of the monitoring terminal 103 is complete | finished.

  Next, the person search process performed by the recording apparatus 101 will be described in more detail with reference to FIGS. 8A to 8D.

[Person Search Processing of Recording Apparatus 101]
(Step S911)
First, in step S911, the network unit 111 of the recording apparatus 101 performs a search input image reception process.
Specifically, the video ID related to the image data of the search input image (image of the search input image specifying unit 304) transmitted from the monitoring terminal 103 at timing T101 is received by the network unit 111 of the recording apparatus 101. That is, a search execution request is received.
Here, the network unit 111 also receives search-accompanying information such as selection statuses of the weight type selection buttons 511 to 513.
The network unit 111 outputs the image data and accompanying information of the received search input image to the comprehensive determination unit 121.

(Step S912)
In step S912, the comprehensive determination unit 121 performs a search associated information storage process.
Specifically, the comprehensive determination unit 121 stores the selection status of the weight type selection buttons 511 to 513 and the video ID of the search input image in the search accompanying information in the temporary storage unit of the comprehensive determination unit 121, and The data is output to the storage unit 112.

(Step S913)
In step S913, the storage unit 112 performs image data search processing.
Specifically, the storage unit 112 reads the corresponding image from the input video ID, that is, the image data of the search input image from the recording medium, and outputs it to the comprehensive determination unit 121.

(Step S914)
In step S <b> 914, the comprehensive determination unit 121 performs image data output processing.
Specifically, the comprehensive determination unit 121 stores the image data of the search input image in a temporary storage unit and outputs it to the face area detection unit 113.

(Step S915)
In step S915, the face area detection unit 113 performs face detection processing.
Specifically, the face area detection unit 113 performs face detection processing on the image data of the input search input image.

(Step S916)
In step S916, the comprehensive determination unit 121 determines whether a face has been detected.
Here, the determination is made using the threshold value as in step S704.
In the case of Yes, that is, when a face is detected, the overall determination unit 121 advances the process to step S919.
In the case of No, that is, when the face is not detected, the comprehensive determination unit 121 advances the process to step S917.

(Step S917)
In step S <b> 917, the comprehensive determination unit 121 outputs “no detection” information to the comprehensive determination unit 121.

(Step S918)
In step S918, the comprehensive determination unit 121 performs “no detection” information transmission processing.
Specifically, the comprehensive determination unit 121 transmits no face detection information from the network unit 111 at timing T102.
This face detection absence information is “no detection” information as described above, and is received by the monitoring terminal 103 via the network 100.
Thereafter, the overall determination unit 121 proceeds with the process to step S969.

(Step S919)
In step S919, the comprehensive determination unit 121 performs face area coordinate calculation processing.
That is, as in step S705, the face area coordinates are calculated, and the calculated coordinate values are output to the face feature quantity extraction unit 114 together with the image data.

Note that “detected” information may be transmitted from the network unit 111 instead of transmitting information without face detection at timing T102.
Thereby, when the monitoring terminal 103 receives the “with detection” information, the control unit of the monitoring terminal 103 can draw a display such as “searching” on the search output image list display unit 306 of the user interface. Therefore, it is possible to reduce annoyance when the user waits for the search result.

(Step S920)
In step S920, the face feature amount extraction unit 114 performs a face feature amount calculation process.
That is, as in step S706, the face feature quantity extraction unit 114 calculates a face feature quantity for a face area image obtained from the input image and face area coordinates. Then, the calculated face feature amount is output to the comprehensive determination unit 121.

(Step S921)
In step S921, the comprehensive determination unit 121 performs a face feature amount storage process.
Specifically, the comprehensive determination unit 121 stores the input face feature amount in a storage medium of the storage unit 112.

(Step S922)
In step S922, the overall determination unit 121 determines whether the weight type selection button 511 and / or 513 is selected.
Specifically, the overall determination unit 121 determines whether the weight type selection buttons 511 and / or 513 are selected in the selection status of the weight type selection buttons 511 to 513 input in step S911.
In the case of Yes, the comprehensive determination unit 121 proceeds with the process to step S923.
In No, the comprehensive determination part 121 advances a process to step S931.

(Step S923)
In step S <b> 923, the overall determination unit 121 outputs the image data of the search input image to the clothing region detection unit 116.

(Step S924)
In step S924, the clothing region detection unit 116 performs clothing detection processing on the image data of the search input image. This clothing detection process is performed in the same manner as in step S708.

(Step S925)
In step S925, the overall determination unit 121 determines whether clothing has been detected. This determination is performed in the same manner as in step S709.
In the case of Yes, the comprehensive determination unit 121 proceeds with the process to step S926.
In No, the comprehensive determination part 121 advances a process to step S929.

(Step S926)
In step S926, the clothing region detection unit 116 performs a clothing region coordinate calculation process on the image data of the search input image.
This process is performed similarly to step S710. That is, the clothing region coordinates are calculated, and the calculated coordinate values are output to the clothing feature value extraction unit 117 together with the image.

(Step S927)
In step S927, the clothing feature value extraction unit 117 performs a clothing feature value calculation process on the image data of the search input image.
This process is performed similarly to step S711. That is, the clothing feature amount extraction unit 117 calculates a clothing feature amount for the clothing region image obtained from the image data of the search input image and the clothing region coordinates, and outputs the calculated clothing feature amount to the comprehensive determination unit 121. To do.
Thereby, the comprehensive determination part 121 memorize | stores the clothing detection information which is the information about whether the clothing was detected in a primary memory | storage means as "with detection."

(Step S928)
In step S928, the overall determination unit 121 performs a clothing feature amount storage process.
Specifically, the comprehensive determination unit 121 stores the input clothing feature value in a temporary storage unit.
After this process, the overall determination unit 121 advances the process to step S931.

(Step S929)
In step S <b> 929, the clothing area detection unit 116 outputs “no detection” information to the overall determination unit 121 when no clothing is detected.

(Step S930)
In step S930, when the information of “no detection” is input from the clothing region detection unit 116, the overall determination unit 121 stores the clothing information in the temporary storage unit as no clothing detected.

(Step S931)
In step S931, the overall determination unit 121 determines whether the weight type selection button 511 and / or 512 is selected.
Specifically, it is determined whether the weight type selection buttons 511 and / or 512 are selected in the selection status of the weight type selection buttons 500 to 513 input in step S911.
In Yes, comprehensive judgment part 121 advances processing to Step S932.
In No, the comprehensive determination part 121 advances a process to step S935.

(Step S932)
In step S932, the comprehensive determination unit 121 outputs the video ID of the search input image to the shooting time recording unit 119.

(Step S933)
In step S933, the shooting time recording unit 119 performs shooting time output processing.
Specifically, the shooting time recording unit 119 reads the corresponding time from the input video ID, that is, the shooting time of the search input image from the recording medium of the storage unit 112 and outputs it to the comprehensive determination unit 121.

(Step S934)
In step S934, the comprehensive determination unit 121 stores the input time in a temporary storage unit.

(Step S935)
In step S935, the overall determination unit 121 determines whether the weight type selection button 512 and / or 513 is selected.
Specifically, it is determined whether the weight type selection buttons 512 and / or 513 are selected in the selection status of the weight type selection buttons 511 to 513 input in step S911.
In Yes, comprehensive judgment part 121 advances processing to Step S936.
In No, the comprehensive determination part 121 advances a process to step S939.

(Step S936)
In step S936, the comprehensive determination unit 121 outputs the video ID of the search input image to the shooting position recording unit 120.

(Step S937)
In step S937, shooting position output processing is performed.
Specifically, the shooting position recording unit 120 reads the corresponding position from the input video ID, that is, the shooting position of the search input image from the recording medium of the storage unit 112, and outputs it to the comprehensive determination unit 121.

(Step S938)
In step S938, the comprehensive determination unit 121 stores the input position in a temporary storage unit.

(Step S939)
In step S939, the comprehensive determination unit 121 performs a database read preparation process for reading the feature amount database 801 from the recording medium of the storage unit 112.
Specifically, the comprehensive determination unit 121 accesses a database daemon or service, and transmits various preparation commands for using the feature amount database 801.

(Step S940)
In step S940, the comprehensive determination unit 121 sets the first row of the feature amount database 801 as a reference row.
The reference row is a variable or the like stored in a temporary storage unit of the comprehensive determination unit 121, and is a counter indicating a row number for reading the table of the feature amount database 801 one by one.

(Step S941)
Next, in step S941, the comprehensive determination unit 121 reads and acquires the video ID, the facial feature value, the clothing feature value, the shooting time, and the shooting position from the reference row.

(Step S942)
Next, in step S942, the comprehensive determination unit 121 performs face similarity calculation processing.
Specifically, the comprehensive determination unit 121 determines the search target image having the video ID and the search input image from the face feature value acquired in step S941 and the face feature value of the search input image stored in the storage unit. The face similarity is calculated.

  The calculation of the face similarity is obtained, for example, by the following equation when the number of face feature amounts is m.

(Σ represents the total).
Here, FWi is a coefficient indicating the importance of the facial feature amount,

FW _i = {fw ₁ , fw ₂ ,..., Fw _m }

It is represented by
FFI _i is the facial feature value of the search input image,

FFI _i = {ffi ₁ , fi ₂ ,..., Fi _m }

It is represented by
FFT _i is the facial feature quantity of the search target image,

FFT _i = {fft ₁ , fft ₂ ,..., Fft _m }

It is represented by
That is, in this step, the face similarity is calculated by calculating the distance between the face feature amounts.
In addition to such a method, it is possible to calculate the face similarity using a statistical method using a distribution function of feature amount or an artificial intelligence method such as support vector machine. .

(Step S943)
Next, in step S943, the overall determination unit 121 determines whether the weight type selection button 511 is selected.
Specifically, the overall determination unit 121 determines whether the weight type selection button 511 is selected in the selection status of the weight type selection buttons 511 to 513 input in step S911. In addition, it is also determined whether the clothing information is in a state where clothing is not detected.
In the case of Yes, that is, when the weight type selection button 511 is selected and the clothing is not detected, the comprehensive determination unit 121 advances the process to step S944.
In No, ie, other than that, the comprehensive determination part 121 advances a process to step S948.

(Step S944)
In steps S944 to S947, the overall determination unit 121 calculates a weight based on the relationship between the shooting time information and the clothing information.
First, in step S944, the comprehensive determination unit 121 performs a clothing similarity calculation process. Specifically, the comprehensive determination unit 121 is stored in the clothing feature amount acquired in step S941 and a temporary storage unit. The clothing feature quantity of the search input image is compared, and the search target image having this video ID, the search input image, and the clothing similarity are calculated.
The calculation of clothing similarity is obtained by the following equation when the number of clothing feature values is n.

Here, CWi is a coefficient indicating the importance of the clothing feature value,
CW _i = {cw ₁ , cw ₂ ,..., Cw _n }
It is represented by
CFI _i is the clothing feature of the search input image,
CFI _i = {cfi ₁ , cfi ₂ ,..., Cfi _n }
It is represented by
CFT _i is the clothing feature of the search target image,
CFT _i = {cft ₁ , cft ₂ ,..., Cft _n }
It is represented by
Note that the distance between feature amounts is also calculated for this clothing similarity.
This clothing similarity can also be calculated using a distribution function of feature quantities, an artificial intelligence technique such as a support vector machine, or a statistical technique.

(Step S945)
In step S945, the overall determination unit 121 compares the calculated clothing similarity with a predetermined threshold value and determines the clothing identity.
Since this threshold value varies depending on the clothing feature to be used, an optimum parameter can be designated in advance.
The determination result is stored in a temporary storage means.

(Step S946)
In step S946, the comprehensive determination unit 121 determines the time proximity by comparing it with a predetermined threshold.
Specifically, the comprehensive determination unit 121 determines the absolute value of the difference between the shooting time acquired in step S941 and the shooting time of the search input image stored in the storage unit and the shooting time of the search target image in the vicinity of the time. Judgment is made by comparison with a predetermined threshold related to sex.
The predetermined threshold can be set based on the speed at which a person walks or moves by taxi, for example.
The determination result is stored in the temporary storage unit by the comprehensive determination unit 121.

(Step S947)
In step S947, the comprehensive determination unit 121 performs a weight calculation process based on the relationship between the shooting time information and the clothing information.
This calculation process is performed using values input by the user in the setting value input fields 611 to 614 of the time + clothing weight setting table 610.
Specifically, the overall determination unit 121 calculates a weight by performing a process such as multiplying the determination result of the proximity of time by the value input by the user in the time + clothing weight setting table 610 in FIG. .
After this process, the overall determination unit 121 advances the process to step S950.

(Step S948)
In step S948, when the weight type selection button 511 is not selected in the selection status of the weight type selection buttons 511 to 513 input in step S911, the comprehensive determination unit 121 has a relationship between the shooting time information and the clothing information. The weight value based on is set to 1.
The overall determination unit 121 similarly sets the weight value to 1 even when the clothing information is in a state in which clothing is not detected.

(Step S949)
In step S949, the overall determination unit 121 determines whether the weight type selection button 512 has been selected.
Specifically, it is determined whether or not the weight type selection button 512 has been selected by the user in the selection status of the weight type selection buttons 511 to 513 input in step S911.
In Yes, comprehensive judgment part 121 advances processing to Step S950.
In No, the comprehensive determination part 121 advances a process to step S953.

(Step S950)
Hereinafter, in steps S950 to S952, the comprehensive determination unit 121 first calculates a weight based on the relationship between the shooting time information and the shooting position information.
First, in step S950, the comprehensive determination unit 121 performs time proximity determination processing.
Specifically, the comprehensive determination unit 121 compares the time of proximity with a predetermined threshold from the shooting time acquired in step S941 and the shooting time of the search input image stored in the temporary storage unit. Judging.
That is, the proximity of the time is determined by comparing the value of the difference between the search target image and the shooting time of the search input image with a predetermined threshold.
The determination result is stored in the primary storage unit of the comprehensive determination unit 121.

(Step S951)
In step S951, the comprehensive determination unit 121 performs position proximity determination processing.
In step S943, the comprehensive determination unit 121 determines the proximity of the position from the shooting position acquired in step S941 and the shooting position of the search input image stored in the storage unit by comparing with a preset threshold value.
That is, the comprehensive determination unit 121 determines the proximity of the position by comparing the difference between the shooting positions of the search target image and the search input image with a predetermined threshold.
The result of the determination is similarly stored in the primary storage unit of the comprehensive determination unit 121.

In addition, as the predetermined threshold value in the above-described steps S950 and S951, a threshold value considering the conditions (1) to (7) in the above case can be used.
As this threshold value, for example, a value relating to whether or not it is the same day or a non-identical day, whether or not it is within several hours, or the like can be used.
As the position threshold value, a value such that the difference between the photographing positions is within an actual distance (such as a predetermined meter) or the premises of the same building (such as a station) can be used.
Furthermore, these threshold values can be adjusted and used according to the actual application.

  In addition, when detected at a plurality of distant positions at a close time, it can be further stated that “the possibility of the same person is high”.

(Step S952)
In step S952, the overall determination unit 121 performs weight calculation processing based on the relationship between the shooting time information and the shooting position information using the determined value.
The calculation of the weight is performed using a value input to the time + position weight setting table 620 by the user.
Specifically, the overall determination unit 121 applies the determination results of steps S950 and S951 described above to the time + position weight setting table 620 in FIG. 5 to calculate weights based on the shooting time information and the shooting position information. I do.
After performing this process, the overall determination unit 121 advances the process to step S954.

(Step S953)
In step S953, the overall determination unit 121 sets the weight value based on the relationship between the shooting time information and the shooting position information to 1.
This is a case where the weight type selection button 512 is not selected in the selection status of the weight type selection buttons 511 to 513 input in step S911.

(Step S954)
In step S954, it is determined whether the weight type selection button 513 is selected.
Specifically, the overall determination unit 121 determines whether or not the weight type selection button 513 is selected in the selection status of the weight type selection buttons 511 to 513 input in step S911.
In Yes, comprehensive judgment part 121 advances processing to Step S955.
In No, the comprehensive determination part 121 advances a process to step S959.

(Step S955)
Thereafter, in steps S955 to S958, the comprehensive determination unit 121 calculates a weight based on the relationship between the shooting position information and the clothing information.
First, in step S955, clothing similarity calculation processing is performed.
Specifically, based on the clothing feature value acquired in step S941 and the clothing feature value of the search input image stored in the storage unit, the clothing between the search target image having this video ID and the search input image. Calculate similarity.
The calculation of the clothing similarity is performed in the same manner as the method in step S944.

(Step S956)
In step S956, the overall determination unit 121 performs clothing identity determination processing.
Specifically, as in step S945, the overall determination unit 121 compares the calculated clothing similarity with a predetermined threshold value to determine the identity of the clothing.

(Step S957)
In step S957, position identity determination processing is performed.
Specifically, the comprehensive determination unit 121 determines the proximity of the position from the shooting position acquired in step S941 and the shooting position of the search input image stored in the storage unit by comparing with a predetermined threshold.

(Step S958)
In step S958, the overall determination unit 121 applies the value of the position + clothing weight setting table 630 in FIG. 5 to the determination result in step S958, and calculates the weight based on the relationship between the shooting position information and the clothing information. I do.

(Step S959)
In step S959, the overall determination unit 121 sets the weight value based on the relationship between the shooting position information and the clothing information to 1 (one). That is, in the selection status of the weight type selection buttons 511 to 513 input in step S911, the weight value is 1 when the weight type selection button 513 is not selected.
It should be noted that the weight value is set to 1 in the same manner even when the clothing information is in a state where clothing is not detected.

(Step S960)
In step S960, the overall determination unit 121 calculates the overall similarity.
Specifically, the face similarity calculated in step S942 is multiplied by the weight values calculated in steps S942 to S948, S949 to S953, and S954 to S959, respectively, thereby calculating the total similarity value. .
That is, the overall similarity value is:

Total similarity = face similarity × (time + clothing weight setting value) × (time + position weight setting value) × (position + clothing weight setting value)

It is calculated by the following formula.
Here, for the set value group of each weight, a value based on the above-described condition based on the threshold is applied and calculated. As this condition, whether the proximity of the position or time in the table of each weight setting is equal to or less than the threshold value or exceeds the threshold value is determined using the above conditions (1) to (7).
As described above, the clothing feature amount is not directly reflected in the overall similarity.

(Step S961)
In step S961, the overall determination unit 121 performs a search output image determination process.
Specifically, the overall determination unit 121 compares the calculated overall similarity with a predetermined threshold value to determine whether to output a search output image.

(Step S962)
In step S962, the comprehensive determination unit 121 determines whether the value determined in step S961 is a value that exceeds the threshold value.
In Yes, comprehensive judgment part 121 advances processing to Step S963.
In No, the comprehensive determination part 121 advances a process to step S964.

(Step S963)
In step S963, the overall determination unit 121 determines that the search target image is one of the search output images, and stores the search target image in a temporary storage unit together with the video ID.

(Step S964)
In step S964, the comprehensive determination unit 121 shifts the reference row down by one row.
Thereby, the next line can be referred to until the end line (final line) of the feature amount database 801 is exceeded. The last row is the last row of the table of the feature amount database 801.

(Step S965)
The comprehensive determination unit 121 determines whether the reference line exceeds the end line.
In Yes, comprehensive judgment part 121 advances processing to Step S966.
In No, the comprehensive determination part 121 returns a process to step S941, and repeats a series of processes similarly until a reference line exceeds an end line.

(Step S966)
In step S966, the comprehensive determination unit 121 performs a search output image list listing process.
Specifically, when it is determined in step S965 that the reference row exceeds the end row, the overall determination unit 121 extracts all the overall similarities stored in the temporary storage unit together with the video ID. Then, the overall determination unit 121 lists the image data of the video ID as a search output image list and stores it in a temporary storage unit.
At the time of the listing, the comprehensive determination unit 121 may execute the sort process so that the total similarity, the time order, the camera order, and the like are in order.

(Step S967)
In step S967, the overall determination unit 121 outputs the search output image list to the network unit 111.

(Step S968)
In step S968, the network unit 111 transmits the input search output image list to the monitoring terminal 103 via the network. The timing of this process is timing T103.

(Step S969)
In step S969, the recording apparatus 101 waits for reception of the next search execution request.
If the network unit 111 receives the next search execution request, the network unit 111 returns the process to step S911. Up to here is the processing in the recording apparatus 101.
The recording apparatus 101 ends the process when performing a shutdown or the like in this state.
Thus, the person search process is completed.

By configuring in this way, the following effects can be obtained.
First, the prior art 1 has a problem that recognition accuracy for a face image projected from a direction other than the front direction is low. Further, when the face size is small, there is a problem that the recognition accuracy is low.
As a result, there is a problem that it is difficult to use the person search method practically. For this reason, the person search function has a problem that it is not used in earnest.

However, the person search method according to the embodiment of the present invention can provide a highly accurate person search method even if the accuracy of face recognition is not so high.
This can be realized by using a clothing feature value which is a feature of the clothing in addition to the face feature value, further using a time / position / clothing rule, and further calculating and applying a weighting value.
Thereby, it becomes possible for the user to find a video of a desired person more efficiently and easily.
Therefore, the person search method can be practically used in the monitoring system.

Thus, by improving the accuracy of the person search method, it is also possible to search for an image including the person for all persons shown in all images recorded at that time.
By using the video ID of the searched image, position and time information can be obtained.
This makes it possible to conduct research on the flow of people at specific locations and times.
For example, the movement of a person in a certain building can be searched and used for more efficient store arrangements.
Also, for example, in an amusement facility such as an amusement park, it is possible to grasp the order in which the visitors go around the attraction, so it is possible to set the price of each attraction appropriately and optimize the passage in the amusement facility. is there.
When searching for a large number of persons, it is desirable to statistically process only the values such as movement information of each video ID without leaving a record for each search result in consideration of privacy.

Conventionally, a method of performing a person search using a continuous correlation model or the like is sometimes used. However, there are various situations in which person search is used, and there is a problem that the continuous correlation model cannot cope with these various situations.
On the other hand, in the present invention, the position and time are the same / non-identical in terms of the relationship with the clothing feature value in the person search, and the weights can be specified directly according to the conditions. .
As a result, it is possible to deal with various situations, and the search intended by the searcher is possible, so that an effect of improving usability can be obtained.

In the embodiment of the present invention, the search target image has been described for all images recorded at that time, but of course, the target is an image with a limited time, camera, etc. It is also possible.
Furthermore, it is possible to search for an object of a specific gender and age (adult or child). This gender and age can be determined from the face area, the feature amount of the clothes, the height of the back, and the like.

  In the embodiment of the present invention, the search output image list is completely obtained in the recording apparatus 101 and then transmitted to the monitoring terminal 103. However, every time it is determined as a search output image in step S962. You may make it transmit to the monitoring terminal 103 sequentially.

Further, in the embodiment of the present invention, the case where one person exists in the search input image is shown. However, when there are a plurality of persons in the search input image, the search for each person is performed. The result of ORing the results may be returned to the user.
In this case, the control unit of the monitoring terminal 103 notifies the user that there are a plurality of people, and the user interface is configured so that one person to be searched is designated and selected by the pointing device or the like of the monitoring terminal 103. The selection may be performed by the user, and may be performed for one person selected by the user.

For simplification of description, the imaging apparatus 102 and the monitoring terminal 103 are shown as a single unit in the drawing. However, a plurality of these devices can be connected to the network 100.
Further, in this example, the configuration is shown in which only the search image is designated by the terminal, and the recording and the search are executed by the same device. However, recording and searching may be executed by separate devices.

Further, in the embodiment of the present invention, in accordance with the user's instruction, only the search accompanying information including the video ID is transmitted from the monitoring terminal 103 to the recording apparatus 101, and the image data itself is not transmitted. However, the image data itself can be transmitted from the monitoring terminal 103, and the recording apparatus 101 can perform various search processes based on the image data.
Further, in the embodiment of the present invention, it has been described that the search target image is selected from the recorded images. However, for example, an image captured by a device different from the imaging device 102 can be used as a search target image.

Further, as the above-described video ID, an elapsed time from the start of recording, or a unique numerical ID assigned when a specific signal (such as a facial feature amount) is detected may be used. Also, search tag data, hash data, and other various information for facilitating the search can be used.
Further, although the above-described input weight setting value is shown as a ratio value, an adjustment value such as “weight weight” in the form of an addition value (or subtraction value) may be used. . Further, an adjustment value of “weight of weight” may be added in addition to the set value of weight.

  Note that the configuration and operation of the above-described embodiment are examples, and it is needless to say that the configuration and operation can be appropriately changed and executed without departing from the gist of the present invention.

It is a system configuration figure of monitoring system Y concerning an embodiment of the invention. It is a conceptual diagram which shows the structure of the feature-value database 801 which shows the face feature-value, clothing feature-value, imaging | photography time, and imaging | photography position which concern on embodiment of this invention. It is a flowchart which shows the flow of the process at the time of video recording in the video recording apparatus 101 which concerns on embodiment of this invention. It is a figure which shows the example of the user interface for the person search in the monitoring terminal 103 which concerns on embodiment of this invention. It is a figure which shows the example of the weight setting screen in the monitoring terminal 103 which concerns on embodiment of this invention. It is a timing chart of person search method execution concerning an embodiment of the invention. It is a flowchart which shows the flow of a process in the monitoring terminal 103 at the time of the person search method execution which concerns on embodiment of this invention. It is a flowchart which shows the flow of a process in the video recording apparatus 101 at the time of the person search method execution which concerns on embodiment of this invention. It is a flowchart which shows the flow of a process in the video recording apparatus 101 at the time of the person search method execution which concerns on embodiment of this invention. It is a flowchart which shows the flow of a process in the video recording apparatus 101 at the time of the person search method execution which concerns on embodiment of this invention. It is a flowchart which shows the flow of a process in the video recording apparatus 101 at the time of the person search method execution which concerns on embodiment of this invention. It is a system block diagram of the monitoring system X provided with the conventional person search function. It is a figure which shows the example of the user interface for the person search in the conventional monitoring terminal 203. FIG.

Explanation of symbols

100, 200 Network 101, 201 Recording device 102, 202 Imaging device 103, 203 Monitoring terminal 111, 211 Network unit 112, 212 Storage unit 113, 213 Face area detection unit 114, 214 Face feature amount extraction unit 115, 215 Face feature amount Recording unit 116 Clothing area detection unit 117 Clothing feature amount extraction unit 118 Clothing feature amount recording unit 119 Shooting time recording unit 120 Shooting position recording unit 121 Overall determination unit 216 Face determination unit 301 Playback video display unit 302 Playback operation unit 303 Camera switching operation Unit 304 Search input image designation unit 305, 505 Search operation unit 306 Search output image list display unit 511-513 Weight type selection button 514 Weight setting button 610 Time + clothing weight setting table 611-614, 621-624, 631-634 Setting Value input field 620 Time + position weight setting Table 630 position + clothes weight setting table 641 Apply button 642 close button 801 feature database 811 video ID column 812 facial feature column 813 clothing feature value column 814 shooting time column 815 shooting position row X, Y video surveillance system

Claims (4)

In a person search method in a surveillance system for recording images,
The shooting time information, the shooting position information, the face feature amount, and the clothing information respectively obtained from the image data of a plurality of recorded images are respectively stored in a randomly accessible storage medium ,
The photographing time information and the vicinity of or near or non vicinity, the clothes information of four values corresponding to the combination of conditions of four kinds of identical or non-identical or identity Time + Clothing weight setting value group,
The photographing time information and the vicinity of or near or non vicinity, time + position weight set of four values corresponding to the combination of conditions of four kinds of the photographing position information near or non-neighboring or Locality Value group,
Position + clothing weight setting value consisting of four numerical values corresponding to combinations of four conditions of proximity of whether the photographing position information is near or non-near and identity of the clothing information being the same or non-identical Receiving at least one input of the group in advance,
When a search instruction is received along with the search input image ,
For each of the plurality of recorded images, one of four numerical values of the input weight setting value group is selected according to the relationship between the recorded image and the search input image, and the recorded image and the difference between the face feature amount of the search input image using the general similarity obtained by weighted numerical values the selected person search method characterized by determining the identity of the recorded person. The time + clothing weight setting value group, the time + position weight setting value group, and the position + clothing weight setting value group include a time + clothing weight setting table in which the four numeric input fields are arranged in a matrix, It is input or changed by the user through a weight setting screen having a time + position weight setting table and a position + clothing setting table,
The search instruction includes an arbitrary plurality of designations among the time + clothing weight setting value group, the time + position weight setting value group, and the position + clothing weight setting value group,
The proximity of the shooting time information, the identity of the clothing information, and the proximity of the shooting position information are determined by comparison with corresponding predetermined threshold values, respectively.
The total similarity is obtained by multiplying a numerical value selected from each of the specified plurality of weight setting value groups and weighting the difference of the face feature amount.
The person search method according to claim 1. The time + clothing weight setting value group, the time + position weight setting value group, and the position + clothing weight setting value group include the following seven conditions:
(1) A person with the same clothes is likely to be the same person at a nearby time
(2) The person with the same clothes may be the same person even at a time that is not near
(3) Persons with non-identical clothes are likely not to be the same person at nearby times
(4) It is highly possible that the person in the vicinity and the position in the vicinity are the same person
(5) It is highly possible that people in the vicinity time and non-neighboring positions are not the same person
(6) Non-near time, people at nearby positions may be the same person
(7) Persons in the vicinity and the same clothes may be the same person
The person search method according to claim 1, wherein the person search method is set based on the above. A monitoring system that records an image with an imaging device, a recording device, and a monitoring terminal,
The recording device comprises:
Facial feature amount extraction means for extracting a facial feature amount from an image captured by the imaging device;
Clothing feature value extraction means for extracting a clothing feature value from an image captured by the imaging device;
At least one or more of image data of a plurality of recorded images, photographing time information, photographing position information, facial feature amount extracted by the facial feature amount extraction unit, and clothing information extracted by the clothing feature amount extraction unit Storing means in a randomly accessible storage medium ;
About the plurality of recorded images,
Time + clothing weight setting value consisting of four numerical values corresponding to four combinations of the proximity of whether the shooting time information is near or non-near and the identity of the clothes information being the same or non-identical group,
Time + position weight setting consisting of four numerical values corresponding to four combinations of the proximity of whether the photographing time information is near or non-neighbor and the proximity of whether the photographing position information is near or non-neighbor Value group,
Position + clothing weight setting value consisting of four numerical values corresponding to combinations of four conditions of proximity of whether the photographing position information is near or non-near and identity of the clothing information being the same or non-identical At least one of the group
Is selected according to the relationship between the recorded image and the search input image, and the total similarity obtained by weighting the difference between the recorded image and the face input of the search input image by the selected numerical value is selected. And a general judging means for judging the identity of the recorded person.

Images