JP6762754B2 - Information processing equipment, information processing methods and programs - Google Patents

Description

The present invention relates to an information processing device, an information processing method and a program.

Conventionally, there has been known a technique of detecting a lost child or a passenger who missed a train or the like by searching the face of a person from each frame of a surveillance image. In Patent Document 1, a person's face is detected from each frame of a surveillance image, image features are calculated from the face and stored in a database (DB) in association with the image frame, for example, a lost child's face is displayed. As a query, a technique for searching a video of a lost child from a DB is disclosed. Further, Patent Document 2 discloses a technique of registering a face of a person to be detected in advance and detecting the person in real time from the input video.

Japanese Unexamined Patent Publication No. 2013-153304 Japanese Unexamined Patent Publication No. 2014-215747

However, in the technique of Patent Document 1, although the video stored in the DB at the time when the query is issued is searched, the target person cannot be searched from the video obtained thereafter. For this reason, it is not possible to inform and secure the current whereabouts of lost children and late passengers. On the other hand, in the technique of Patent Document 2, the processing speed becomes slow when the person is searched retroactively to the video before the time when the query is issued. Therefore, the technique of Patent Document 2 is not sufficient for the purpose of securing lost children and late passengers.

The present invention has been made in view of such problems, and an object of the present invention is to continuously search for a person from the past without causing a decrease in processing speed.

Therefore, the present invention is an information processing device, which is based on a reception means for receiving a search instruction for a person and features extracted from the target person related to the search instruction, before the reception time when the search instruction is received. A first search means for performing a first search process for searching for the target person, and the search, using the characteristics of a person input from the imaging means and extracted from the image stored in the storage means as the first search range . Based on the characteristics of the target person extracted from the query image according to the instruction, the characteristics of the person extracted from the image input from the imaging means after the reception time are set as the second search range, and then the target person is selected. It has a second search means for performing a second search process for searching, and the first search means is further prepared for the second search process by the second search means from the time of the reception. It is characterized in that the process of searching for the target person from the features extracted from the video input before the start of the second search process is performed.

According to the present invention, a person can be continuously searched from the past without causing a decrease in processing speed.

It is a figure which shows the monitoring system which concerns on 1st Embodiment. It is a figure which shows the software configuration of an information processing apparatus. It is a flowchart which shows the video recording process. It is explanatory drawing of the 1st search process and the 2nd search process. It is a flowchart which shows the search process. It is a flowchart which shows the search process which concerns on 2nd Embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First Embodiment)
FIG. 1 is a diagram showing a monitoring system according to the first embodiment. The monitoring system has an information processing device 100 and a camera 112 as an imaging unit, and searches for a person designated as a query. Although FIG. 1 illustrates a case where the person detection system has three cameras 112, the number of cameras 112 is not limited to the embodiment. The information processing device 100 and the camera 112 are connected to each other via the network 111.

The CPU 101 is a Central Processing Unit that controls the entire information processing apparatus 100. The ROM 102 is a Read Only Memory that stores programs and parameters that do not require changes. The RAM 103 is a Random Access Memory that temporarily stores programs and data supplied from an external device or the like. The external storage device 104 is a storage device such as a hard disk or a memory card fixedly installed in the information processing device 100. The external storage device 104 may include an optical disk such as a flexible disk (FD) or a compact disk (CD) that can be attached to and detached from the information processing device 100, a magnetic or optical card, an IC card, a memory card, and the like. The functions and processes of the information processing device 100, which will be described later, are realized by the CPU 101 reading a program stored in the ROM 102 or the external storage device 104 and executing this program.

The input I / F 105 is an interface with an input unit 109 such as a pointing device or a keyboard that receives a user's operation and inputs data. The output device I / F 106 is an interface with the monitor 110 for displaying the data held by the information processing device 100 and the supplied data. The communication I / F 107 connects to a network 111 such as the Internet. The camera 112 is an image capturing device such as a surveillance camera, and is connected to the information processing device 100 via a network 111. The system bus 108 is a transmission line for communicably connecting the units 101 to 107.

FIG. 2 is a diagram showing a software configuration of the information processing device 100. The information processing device 100 uses a face image feature obtained from a face as an image feature, and performs an image feature search using the face image feature. The image receiving unit 201 receives an image taken by the camera 112 and input from the camera 112 to the information processing device 100. The video storage unit 202 stores the video received by the video reception unit 201.

The tracking unit 203 tracks a person in the video received by the video reception unit 201. Specifically, the tracking unit 203 detects an object from the motion vector, estimates the search position in the next frame, and performs person tracking by template matching. Regarding the person tracking process, the following documents can be referred to.

JP-A-2002-373332

The tracking unit 203 issues the same tracking track ID to the tracking tracks tracking the same person, and issues different tracking track IDs to the tracking tracks of different persons. As a result, uniqueness is guaranteed, and the same person can be identified from the tracking track ID. The tracking unit 203 issues a different tracking track ID even if the person is the same person, if the tracking is interrupted.

The face detection unit 204 detects the face from a frame image or the like in the video. For example, the face detection unit 204 detects the face from each of the frame images of the person tracked by the tracking unit 203. In addition, face detection is performed from the video input from the video reception unit 201 or the query image described later. The face detection unit 204 detects a one-eye candidate region for the image to be processed, performs pairing from the plurality of one-eye candidate regions, and determines the face region based on the paired eye positions. The following documents can be referred to for a method of detecting a person's face from an image.

JP-A-2010-165156

The representative determination unit 205 selects a representative face image from the frame image group of the tracked person. Hereinafter, a representative face image will be referred to as a representative image. The representative determination unit 205 selects, for example, an image having a large face size from the face images detected by the face detection unit 204. The reason for using the face size is the problem of accuracy of image features. This is because the larger the face image, the more accurate the image features can be obtained. That is, when calculating the image features from the face image, it is necessary to perform the face size normalization process for scaling the size of the face image to a constant size. At that time, if the face image is larger than the above-mentioned constant size, reduction processing is performed and the loss of information is relatively small, but if it is smaller than the above-mentioned constant size, it is necessary to perform pixel complementation such as super-resolution processing. There is severe deterioration of information.

Further, as another example, the representative determination unit 205 may select a plurality of images from the frame image group as representative images. For example, there is a method of selecting images with a plurality of face orientations. This is because even if the image of the same person is different in face orientation, the image features obtained from the image are different.

The representative determination unit 205 extracts a gradient direction histogram (Histogram of Oriented Gradient, HOG) as a feature, and estimates the orientation of the face by SVR. The gradient direction histogram is a feature in which the luminance gradient information of an image is made into a histogram for each local part of the image, and is known as a feature that is robust to local noise and light and darkness of the image. By selecting robust features for fluctuations that are not related to face orientation, such as noise and lighting fluctuations, stable face orientation estimation is achieved even in the actual environment. The following documents can be referred to for a method of detecting the face orientation of a person from an image.

Eric Muphy-Chutorian, "Head pose establishment for driver assistance systems: A robot algorithm and experimental evaluation," in Proc. IEEE Conf. Intelligent Transport Systems, 2007, pp. 709-714.

Further, the representative determination unit 205 may detect the orientation of the human body of the person instead of the orientation of the person's face. For the detection of the orientation of the human body, for example, the method described in the following documents can be used.

Japanese Unexamined Patent Publication No. 2011-186576

Further, as another example, the representative determination unit 205 may further select an image with less blurring as the representative image. Similar to a still image camera, a camera that shoots a moving image may change the shutter speed according to the brightness of the place. Therefore, blurring of the face image may occur depending on a dark place or the movement speed of the subject, which directly causes deterioration of image features and attribute information. Regarding the estimation of blur, the representative determination unit 205 obtains the frequency component of the face image region, obtains the ratio of the low frequency component to the high frequency component, and causes blur when the ratio of the low frequency component exceeds a predetermined value. You can judge that it is. In addition, the representative determination unit 205 may not select a face image such as eye contact or mouth opening as a representative image. This is because the image features of the organ may be altered if there is eye contact or mouth opening.

The feature calculation unit 206 calculates the image features of the face image to be processed. Specifically, the feature calculation unit 206 obtains organ points such as eyes and mouth in the face image of a person, and calculates the SIFT (Scale Invariant Feature Transfer) features of each organ point. It should be noted that this feature is an example, and the type of feature calculated is not limited to the embodiment. The first feature storage unit 207 stores the image features calculated by the feature calculation unit 206 with respect to the video received by the video reception unit 201. The first feature storage unit 207 further stores the image feature metadata in association with a person's ID, a tracking track ID when the person is tracked, a shooting time, and a camera ID of the camera 112 that has taken a picture. The first feature storage unit 207 is, for example, an external storage device 104.

The query reception unit 208 accepts the designation of the query image. Here, the query image is a face image of a person who is a search key. The person who becomes the search key is called the target person. For example, the CPU 101 displays the face image stored in the external storage device 104 on the monitor 110. Then, while looking at the monitor 110, the user selects the face image of the person to be searched as the query image via the input unit 109. Correspondingly, the query reception unit 208 accepts the query image selected by the user. The query image may be one image or two or more images.

The second feature storage unit 209 stores the image features of the query image (face image) received by the query reception unit 208. Here, the image feature of the query image is an example of a reference feature predetermined for the target person. The image feature of the query image is calculated by the feature calculation unit 206. The second feature storage unit 209 is, for example, an external storage device 104. The recording of the image feature of the query image in the second feature storage unit 209 is performed by the CPU 101. The process of recording the image features of the query image in the second feature storage unit 209 is a preparatory process for starting the search process of the input image executed by the second search unit 212 described later. Further, when the CPU 101 receives the instruction of exclusion of the target described later, the CPU 101 deletes the image feature from the second feature storage unit 209.

The first search unit 211 executes the first search process. Here, the first search process is a process of using the image feature calculated from the query image as a query and searching for the target person from the images in the search range set for the first search process. The first search unit 211 according to the present embodiment uses the image features stored in the first feature storage unit 207 as the search range, and the image features whose similarity is higher than a predetermined threshold value in the first feature storage unit 207. To identify. In calculating the similarity, the sum of the distances of the SIFT features of each organ point is obtained, the sum of the distances is normalized, and the similarity is obtained.

The second search unit 212 executes the second search process. Here, in the second search process, the image feature (reference feature) stored in the second feature storage unit 209 is used as a query, and the target person is selected from the video in the search range set for the second search process. This is the process of searching. The second search unit 212 according to the present embodiment sets the image feature calculated from the video received by the video reception unit 201 after the time when the search instruction for which the query image is specified is received as the search range. Here, the search range includes not only the video received by the video receiving unit 201 but also the video stored in the temporary storage unit 210. The second search unit 212 identifies an image feature having a similarity higher than a predetermined threshold value in the search range. The method of calculating the similarity is the same as that described in the first search unit 211.

The display processing unit 213 integrates the search result of the first search unit 211 and the identification result of the second search unit 212, and displays the result on the monitor 110. For example, the display processing unit 213 classifies the search results for each camera 112 and displays them on the monitor 110 in the order of shooting time. As another example, the display processing unit 213 displays the installation position of the camera 112 on a map, and displays the movement history on the monitor 110 based on the shooting time. The displayed content is not limited to the embodiment.

The exclusion reception unit 214 receives information on a person who no longer needs to be searched by securing a lost child or a late passenger, and deletes the image feature of the person from the second feature storage unit 209. It is possible to specify on the user interface of the display processing unit 213 that a lost child or a late passenger has been secured. Further, when searching for a passenger who missed the aircraft, it is possible to detect that the passenger has passed the boarding gate of the aircraft and use this as a trigger to delete the image feature of the person from the second feature storage unit 209.

FIG. 3 is a flowchart showing the video recording process by the information processing device 100. The video recording process is a process of accumulating the video received by the video receiving unit 201 so that it can be searched. In S301, the video reception unit 201 receives the video. Next, in S302, the CPU 101 records the video received in S301 in the video storage unit 202. At this time, the CPU 101 records the shooting time and the camera ID of the camera 112 as the video metadata in association with each other. This process is an example of a recording process for recording the image input from the camera 112 in the storage unit.

Next, in S303, the tracking unit 203 detects a person from each frame image and performs tracking. Here, a different person ID is assigned to the detected person for each frame image, and is temporarily stored together with the coordinates of the person in the frame image. Further, the same tracking track ID is assigned to the person performing the tracking, and is temporarily stored together with the ID of the frame image being tracked. Next, in S304, the tracking unit 203 advances the process to S305 when there is a person whose tracking is interrupted (YES in S304). If there is no person whose tracking is interrupted (NO in S304), the tracking unit 203 advances the process to S301.

In S305, the face detection unit 204 detects the face from each of the frame images of the person tracked by the tracking unit 203. Next, in S306, the representative determination unit 205 selects one or a plurality of representative face images from the frame image group of the tracked person. Next, in S307, the feature calculation unit 206 calculates facial image features from one or a plurality of facial images selected in S306.

Next, in S308, the CPU 101 records the image feature calculated in S307 in the first feature storage unit 207. The CPU 101 further records a person ID, a tracking track ID when the person is tracked, a shooting time, and a shooting camera as metadata of image features. Next, in S309, the CPU 101 confirms whether or not the instruction to end the storage process has been received. When the CPU 101 receives the end instruction (YES in S309), the CPU 101 ends the process. If the CPU 101 does not accept the end instruction (NO in S309), the CPU 101 proceeds to S301. By the above processing, the image features of the face image of the person appearing in the image input from the camera 112 are accumulated in the first feature storage unit 207, and are in a searchable state.

Next, the person search process by the information processing device 100 will be described. Prior to the description of the person search process, the first search process by the first search unit 211 and the second search process by the second search unit 212 will be described with reference to FIG. FIG. 4A is an explanatory diagram of the first search process and the second search process. In the first search process, collation is performed using the extracted image features as a query. Therefore, it is possible to search by using an arbitrary person as a query. On the other hand, in the second search process, it is necessary to pre-register the image features of the query person.

Regarding accuracy, the second search process has an advantage in terms of accuracy due to machine learning or the like. Regarding the speed, the first search process is fast for the search of the registered video, but it is slightly inferior in terms of throughput (registration and search) because it takes time to reflect it in the first feature storage unit 207. On the other hand, in the second search process, if the number of persons to be pre-registered is suppressed to a certain number, authentication can be performed in real time, so that the throughput is considered to be high.

Therefore, as a preferred application, the first search process is a search from a video accumulating an arbitrary person from a large number of past images, and the second search process identifies a limited number of specific people in real time. It is to be.

Therefore, the information processing device 100 according to the present embodiment receives the instruction for designating the query person, performs the first search process for the video before (past) the time when the person search is started, and after that time. The second search process is performed on the (future) video. As a result, for example, a lost child search can be appropriately performed. The reason why the past video is also searched is that the information on the past location is important because it is possible that the camera is in a location that is not reflected by all the cameras 112, and it is also useful for estimating the destination.

However, simply combining the first search process and the second search process causes a problem. FIG. 4B is a diagram showing a search range (time range) searched in the first search process and the second search process. In the first search process, the image recorded in the first feature storage unit 207 is set as the search range. Therefore, a slight preparation period (delay) 401 occurs before the video at the time of acceptance of the query designation can be searched. Here, the preparation period is a period from the time when the video reception unit 201 receives the search instruction to the time when the preparation for the search process by the second search unit 212, which will be described later, is completed.

Also in the second search process, it takes time to perform initialization processes such as setting and learning of image features to be searched from the images of each camera 112, and a preparation period (delay) 402 until the detection process becomes possible occurs. Therefore, a searchable time zone 403, which is a time zone obtained by adding the preparation period 401 and the preparation period 402, occurs. In security applications such as surveillance systems, it is desirable to reduce the non-searchable time zone 403.

FIG. 5 is a flowchart showing a search process by the information processing device 100. In S501, the query reception unit 208 receives a search instruction specifying a query image of a person to be a search query. After the process of S501, the CPU 101 executes the process in three threads. The first thread is a thread that performs the first search process, and includes the processes of S502 to S505. The second thread is a thread that performs the second search process, and includes the processes of S511 to S513. The third thread is a process of integrating and displaying the search results, and includes the processes of S521 and S522.

In S502 of the first thread, the first search unit 211 searches for the target person by performing the first search process. The search range in this case is input to the information processing device 100 before the image stored in the first feature storage unit 207, that is, the reception time when the search instruction is received, and is stored in the image storage unit 202. This is an image feature of the image. Further, the search query is an image feature of the face image detected from the query image related to the search instruction. The first search unit 211 sends information such as the image obtained by the search, the shooting location of the image, and the time to the display processing unit 213.

Next, in S503, the first search unit 211 confirms whether or not the second search process by the second search unit 212 has started. When the second search process is started (Yes in S503), the first search unit 211 advances the process to S505. If the second search process has not started (No in S503), the first search unit 211 advances the process to S504.

In S504, the first search unit 211 again performs the first search process. The search range of the first search process in S504 is an image feature that is not set in the search range of the first search process that has already been executed among the image features stored in the first feature storage unit 207. Is. The first search unit 211 then proceeds to S503 for processing. In this way, the first search unit 211 repeatedly executes the first search process with the newly input video feature as the search range until the second search process starts. The first search unit 211 further sends information such as the image obtained by the search, the shooting location of the image, and the time to the display processing unit 213. Then, in S505, the first search unit 211 stops the first search process. This completes the processing of the first thread. In this way, the video input during the period until the second search process is started is set in the search range of the first search process, and the first search process is performed. Therefore, the non-searchable time zone 403 can be shortened.

In S511 of the second thread, the second search unit 212 performs a preparatory process for starting the second search process. The preparatory process is a process of registering (recording) the image feature of the query image of the target person in the second feature storage unit 209 as a reference feature. In S511, the display processing unit 213 may display the query image on the monitor 110. Next, in S512, the second search unit 212 starts a second search process for searching the target person from the video input to the information processing device 100 after the reception time based on the reference feature. Then, the second search unit 212 sends information such as the image obtained by the search, the shooting location of the image, and the time to the display processing unit 213. When the second search process is started in S512, the first search unit 211 determines that the second search process has started in the above-mentioned S503, and the first search process is stopped.

Next, in S513, the second search unit 212 confirms whether or not the CPU 101 stop instruction has been accepted. If the CPU 101 does not accept the stop instruction (No in S513), the CPU 101 continues the second search process. When the CPU 101 receives the stop instruction (Yes in S513), the CPU 101 ends the process. For example, when it is no longer necessary to search for the person specified as the query by securing a lost child or a late passenger, the user inputs a stop instruction.

In S521 of the third thread, the display processing unit 213 integrates the detection results obtained in S502, S504, and S512, and displays the integrated detection result on the monitor 110. Next, in S522, the CPU 101 confirms whether or not the stop instruction has been accepted. If the CPU 101 does not accept the stop instruction (No in S522), the CPU 101 advances the process to S521 and continues the display process. When the CPU 101 receives the stop instruction (Yes in S522), the CPU 101 ends the process.

As described above, the information processing apparatus 100 searches for the target person by the first search process for the search impossible time zone 403, and then stops the first search process. Then, the information processing device 100 receives the stop instruction. The target person is searched by the second search process. In addition, the search impossible time zone corresponding to the time zone in which the feature amount is not indexed by the face image feature search when the query person is specified and the search is executed and the time zone until the face image feature identification start is combined. Can be reduced by performing the first search process recursively. As described above, the information processing apparatus 100 of the present embodiment can continuously search for a person from the past without causing a decrease in processing speed.

(Second Embodiment)
Next, the monitoring system according to the second embodiment will be described. The information processing device 100 of the monitoring system according to the second embodiment performs the first search process before the second search process, and the image detected in the first search process is used in the second search process. Use. Thereby, the search accuracy can be improved.

Hereinafter, the difference between the monitoring system according to the second embodiment and the monitoring system according to the first embodiment will be described. FIG. 6 is a flowchart showing a search process by the information processing apparatus 100 according to the second embodiment. In S601, the query reception unit 208 receives a search instruction specifying a query image of a person to be a search query. The CPU 101 executes the processing in two threads after the processing of 601. The first thread is a thread that performs a search, and includes the processes of S602 to S617. The second thread is a process of integrating and displaying the search results, and includes the processes of S621 and S622.

In S602 of the first thread, the first search unit 211 searches for the target person by performing the first search process. The search range in this case is input to the information processing device 100 before the image stored in the first feature storage unit 207, that is, the reception time when the search instruction is received, and is stored in the image storage unit 202. This is an image feature of the image. Further, the search query is an image feature of the face image detected from the query image related to the search instruction. The first search unit 211 sends information such as the image obtained by the search, the shooting location of the image, and the time to the display processing unit 213.

The CPU 101 further executes the processing in two threads after the processing of S602. The eleventh thread is a thread that continues the first search process, and includes the processes S603 to S605. The twelfth thread is a thread that performs the second search process, and includes the processes of S611 to S617. The processing of S603 to S605 of the eleventh thread is the same as the processing of S503 to S505 of the search processing of the first embodiment described with reference to FIG.

In S611 of the twelfth thread, the second search unit 212 confirms whether or not the target person is detected in the first search process in S602. When the second search unit 212 detects the target person (Yes in S611), the second search unit 212 advances the process to S612. If the second search unit 212 does not detect the target person (No in S611), the second search unit 212 proceeds to S615. In S612, the second search unit 212 prepares to perform the second search process by using not only the query image but also the detected image in which the target person is detected in the first search process in S602. Do.

Next, in S613, the second search unit 212 starts the second search process using the query image and the detected image. Next, in S614, the second search unit 212 confirms whether or not the stop instruction has been accepted. When the second search unit 212 receives the stop instruction (Yes in S614), the second search unit 212 ends the 21st thread processing. If the second search unit 212 does not accept the stop instruction (No in S614), the second search unit 212 continues the second search process. In S615, the second search unit 212 performs the preparatory process to perform the second search process using only the query image. The processing of S615 to S617 is the same as the processing of S511 to 513 of the search processing of the first embodiment described with reference to FIG.

In S621 of the second thread, the display processing unit 213 integrates the detection results obtained in S602, S604, S613, and S616, and displays the integrated training results on the monitor 110. Next, in S622, the CPU 101 confirms whether or not the stop instruction has been accepted. If the CPU 101 does not accept the stop instruction (No in S622), the CPU 101 advances the process to S621 and continues the display process. When the CPU 101 receives the stop instruction (Yes in S622), the CPU 101 ends the process. The other configurations and processes of the monitoring system according to the second embodiment are the same as the configurations and processes of the monitoring system according to the first embodiment.

As described above, the information processing device 100 of the second embodiment has the characteristics of the video input to the information processing device 100 and stored in the video storage unit 202 as the search range before the time when the search instruction is received. The first search process is executed before the second search process. Then, after the first search process, the information processing apparatus 100 performs a second search process using the detected image of the target person obtained in the first search process. Thereby, the search accuracy can be improved. Further, the information processing apparatus 100 continues the first search process in a different search range until the second search process is started, as in the first embodiment. Therefore, the information processing apparatus 100 can continuously search for a person from the past without causing a decrease in processing speed.

Hereinafter, examples of modification of the first embodiment and the second embodiment will be described. As a first modification, the image features are not limited to the image features of the face image. As another example, the image feature may be a human body image feature obtained from the entire human body. Examples of human body image features include numerical values such as the ratio of height to body width and the ratio of foot length to head and body and height. As another example, the information processing apparatus 100 may use the attribute information of a person obtained from the entire human body including the face, instead of the image feature. Examples of attribute information include race, gender, age, presence / absence of glasses, beard, color of clothes, and the like.

As another example, the information processing apparatus 100 may use both the image feature and the attribute information as the image feature. For example, the information processing apparatus 100 individually calculates the scores of the first search process and the second search process for image features and the scores of the first search process and the second search process for attribute information. To do. Then, the information processing apparatus 100 may calculate the weighted sum of both scores as the total score.

As a second modification example, the information processing apparatus 100 may display the query person and the image as the search result in a distinguishable manner. This allows the user to confirm whether or not the search is erroneous.

As a third modification example, when at least one of the first search unit 211 and the second search unit 212 detects a plurality of images of the target person, the images having different face orientations are preferentially detected. You may do it.

As a fourth modification example, at least one of the first search unit 211 and the second search unit 212 may preferentially detect an image taken at a time point closer to the processing time point. It is useful when the purpose is to secure a person such as a lost child.

As a fifth modification example, if the first search unit 211 of the information processing device 100 performs the second search process on the video input to the information processing device 100 by the time the second search process starts. Often, the specific processing for that purpose is not limited to the embodiment. As another example, when the second search process starts, the first search unit 211 is input to the information processing device 100 and stored in the video storage unit 202 by the time the second search process starts. The first search process may be performed once with the feature of the moving image as the search range.

As described above, according to each of the above-described embodiments, it is possible to continuously search for a person from the past without causing a decrease in processing speed.

Although the present invention has been described in detail based on the preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various embodiments within the scope of the gist of the present invention are also included in the present invention. included. Some of the above-described embodiments may be combined as appropriate.

In the above description, in the first search process, the detected image of the target person can be acquired by specifying the image feature whose similarity is higher than the predetermined threshold value. However, for example, when a user (observer) who uses the information processing device 100 identifies a target person and the information is input by the input means of the information processing device 100, the detected image of the target person can be acquired. You may do so.

(Other Examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Information processing device 201 Video reception unit 202 Video storage unit 211 First search unit 212 Second search unit

Claims (11)

A reception means that accepts search instructions for people,
Based on the characteristics extracted from the target person related to the search instruction, the characteristics of the person input from the imaging means before the reception time when the search instruction is received and extracted from the image stored in the storage means are first. As the search range of , the first search means for performing the first search process for searching the target person, and
Based on the characteristics of the target person extracted from the query image related to the search instruction, the characteristics of the person extracted from the image input from the imaging means after the reception time are set as the second search range, and then the target. It has a second search means for performing a second search process for searching for a person.
The first search means is further described from the video input from the time of the reception until the preparation for the second search process by the second search means is completed and the second search process is started. An information processing device characterized by performing a process of searching for the target person from the extracted features . An integrated means that integrates the search results of the first search means and the search results of the second search means, and
The information processing apparatus according to claim 1, further comprising a display means for displaying the search result after integration by the integration means. The first search means is input from the imaging means before the reception time, and after the first search process in which the characteristics of the person extracted from the image stored in the storage means is the search range is completed. 1. When the second search process has not started, the target person is searched using the feature extracted from the video newly input by the imaging means as the search range. 2. The information processing apparatus according to 2. When the second search process is started, the first search means completes the preparation for the second search process by the second search means from the reception time, and performs the second search process. The information processing apparatus according to any one of claims 1 to 3, wherein the process of searching for the target person from the features extracted from the video input before the start is completed. When the target person is detected by the first search means, the second search means searches for the target person based on the image in which the target person is detected and the query image. The information processing apparatus according to any one of claims 1 to 4, characterized in that. In the second search means, the first search means performs the first search process targeting the features of an image input from the imaging means before the reception time and stored in the storage means. The information processing apparatus according to claim 5, wherein the second search process is started after the completion. The information processing device according to any one of claims 1 to 6, wherein the second search means ends the second search process when a stop instruction is received. The information processing device according to claim 2, wherein the display means displays the query image and the search result in a distinguishable manner. A claim, wherein at least one of the first search means and the second search means preferentially detects an image in which the orientation of the face of the target person or the orientation of the human body of the target person is different. The information processing apparatus according to any one of 1 to 8. It is an information processing method executed by an information processing device.
A reception step that accepts search instructions for people,
Based on the characteristics extracted from the target person related to the search instruction, the characteristics of the person input from the imaging means before the reception time when the search instruction is received and extracted from the image stored in the storage means are first. As the search range of , the first search step of performing the first search process for searching the target person, and
Based on the characteristics of the target person extracted from the query image related to the search instruction, the characteristics of the person extracted from the image input from the imaging means after the reception time are set as the second search range, and then the target. Includes a second search step that performs a second search process to search for a person.
From the in the first search step, further characterized by the step of providing from the reception time of the second search processing in the second search step is completed, it is input prior to initiating the second search processing video An information processing method characterized by performing a process of searching for the target person from the extracted features . Computer,
A reception means that accepts search instructions for people,
Based on the characteristics extracted from the target person related to the search instruction, the characteristics of the person input from the imaging means before the reception time when the search instruction is received and extracted from the image stored in the storage means are first. As the search range of , the first search means for performing the first search process for searching the target person, and
Based on the characteristics of the target person extracted from the query image related to the search instruction, the characteristics of the person extracted from the image input from the imaging means after the reception time are set as the second search range, and then the target. To function as a second search means that performs a second search process to search for a person,
The first search means is further extracted from the video input between the time of the reception and the time when the preparation for the second search process by the second search means is completed and the second search process is started. A program that performs a process of searching for the target person from the features .

Images