JP2021086271A - Information processing apparatus, information processing system, information processing method, and program - Google Patents

Abstract

To make it easier to detect a subject again, which has been detected once, in the system that detects the subject to be detected.SOLUTION: An information processing apparatus comprises an image acquisition unit for acquiring an image, a detection unit for detecting a subject to be detected from the image, a confirmation result acquisition unit for acquiring a result of visual confirmation by a user for the detection result by the detection unit, and a detection condition change unit for changing a detection condition when the detection unit detects the subject to be detected, and changes the detection conditions for detecting the subject to be detected based on the result of visual confirmation acquired by the confirmation result acquisition unit.SELECTED DRAWING: Figure 3

Description

The present invention relates to an information processing device, an information processing system, an information processing method, and a program for detecting a specific subject from an image.

A person authentication technique for determining a person in an image is known. Applications of person authentication technology include systems for detecting people requiring attention, detecting important customers, and detecting lost children using surveillance cameras. In these systems, since the cameras are installed in various places, the way in which a person is photographed varies depending on the lighting conditions, face orientation, etc., and the difficulty of person authentication is high. In addition, since the authentication result may be incorrect, the authentication result by the device may be visually confirmed in actual operation. For example, in the case of detecting a person requiring attention, when a person requiring attention registered in advance is detected by the device, the user visually confirms whether the detection result is correct, and if it is correct, calls out to that person. Further, Patent Document 1 proposes a technique of detecting a customer by an apparatus and correcting the detection result based on the result of visual confirmation by a person.

Japanese Unexamined Patent Publication No. 2012-141708

However, the technique described in Patent Document 1 is premised on being able to grasp the position of the detected person, and cannot deal with the case where the person is lost. Even if you try to contact the detected person, you may not be able to contact it immediately due to congestion, etc., and if you lose sight of it, the detected result may be wasted. If the cameras are installed in multiple locations, it would be good if they could be detected again by another camera, but unlike cameras that have detected shooting conditions such as lighting conditions and face orientation, there is no guarantee that this will be possible. The present invention has been made in view of such circumstances, and an object of the present invention is to make it easier to detect a subject once detected again.

The information processing apparatus according to the present invention has an image acquisition means for acquiring an image, a detection means for detecting a subject to be detected from the image, and a confirmation result for acquiring the result of visual confirmation by the user of the detection result by the detection means. Having the acquisition means and the detection condition changing means for changing the detection condition when the detection means detects the subject to be detected based on the result of the visual confirmation acquired by the confirmation result acquisition means. It is a feature.

According to the present invention, the possibility that a once detected subject can be detected again increases, and the convenience of the system is improved.

It is a figure which shows the structural example of the information processing system in 1st Embodiment. It is a figure which shows the hardware configuration example of the information processing apparatus in 1st Embodiment. It is a figure which shows the functional structure example of the information processing apparatus in 1st Embodiment. It is a figure which shows the functional structure example of the detection part in 1st Embodiment. It is a flowchart which shows the example of the detection process in 1st Embodiment. It is a flowchart which shows the example of the object detection processing in 1st Embodiment. It is a flowchart which shows the example of the condition change process in 1st Embodiment. It is a figure explaining the display example of the detection result in 1st Embodiment. It is a flowchart which shows the example of the dictionary registration process in 1st Embodiment. It is a figure explaining the display example of the detection result in 2nd Embodiment. It is a figure explaining the transition of the degree of similarity with the photographed face image.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
The first embodiment of the present invention will be described. In the following, in a detection system including a plurality of cameras, a user such as a security guard visually confirms a person to be detected such as a person requiring attention detected by the device, and the detection condition is changed so that the person can be easily detected again. An example will be described. The user visually confirms the person detected by the device from the image taken by a certain camera, and if it is confirmed that the person is the person to be detected, it is easier to detect again from the image taken by that camera and other cameras. In addition, the detection conditions related to the detection of the person are changed. It should be noted that the user has a mobile terminal, and the detection result by the device is acquired via the mobile terminal and visually confirmed.

FIG. 1 is a diagram showing a configuration example of a detection system as an information processing system according to the first embodiment. The detection system shown in FIG. 1 includes an information processing device 101, a network 102, cameras 103 and 104, and a mobile terminal 105. The configuration shown in FIG. 1 is an example, and the present embodiment is not limited to this. For example, the number of cameras and mobile terminals connected via the network 102 is arbitrary. Further, apart from the mobile terminal 105, a display device such as a monitor for displaying the detection result by the device, an input device for the user to input the result of visual confirmation, and the like may be provided.

The information processing device 101 communicates with each device such as the cameras 103, 104 and the mobile terminal 105 via the network 102, performs a predetermined process based on the input from each device, and sends the processing result to each device or the like. Output. The information processing device 101 detects, for example, a person to be detected from the acquired image, and outputs the detection result to each device.

The network 102 is, for example, a local area network, and is used for communication between each device. The network 102 is not limited to the local area network, and other communication networks may be used. The network 102 may be a wirelessly connected communication network, a wiredly connected communication network, or a communication network capable of both wirelessly and wiredly connected.

The cameras 103 and 104 include a camera unit composed of a lens and an image sensor such as a CCD and a CMOS sensor, and a communication device for connecting to the network 102. The cameras 103 and 104 output the image captured by the camera unit to the information processing device 101 via the network 102. As the camera, another camera having a communication function may be used. Further, the camera unit is not limited to visible light, and may be an infrared camera unit or the like. Cameras 103 and 104 are, for example, surveillance cameras set at a location.

The mobile terminal 105 has a touch panel input / output device, a display device, and a communication device. The mobile terminal 105 displays the detection result of the information processing device 101, and outputs the result of visual confirmation by the user to the information processing device 101 via the network 102. In the present embodiment, the communication device of the mobile terminal 105 is a wireless communication unit.

The configuration and number of devices in the detection system shown in FIG. 1 are not limited to this. Further, in the above description, each device is configured as a separate device, but it may be configured by one device having a plurality of functions. For example, the information processing device 101 may be built in the camera 103.

(Hardware configuration of information processing device)
FIG. 2 is a block diagram showing a hardware configuration example of the information processing apparatus 101 according to the first embodiment. As shown in FIG. 2, the information processing device 101 includes a CPU 201, a ROM 202, a RAM 203, a secondary storage device 204, a communication device 205, and an input / output device 206. The CPU 201, ROM 202, RAM 203, secondary storage device 204, communication device 205, and input / output device 206 are connected to each other via a connection bus 207 so that data can be communicated with each other.

The CPU (Central Processing Unit) 201 controls the present device and the detection system in the present embodiment by executing a control program stored in the ROM 202 or the RAM 203. The ROM (Read Only Memory) 202 is a non-volatile memory, and stores a control program, various parameter data, and the like. The control program is executed by the CPU 201, and each process described later is realized. The RAM (Random Access Memory) 203 is a volatile memory, and temporarily stores an image, a control program, an execution result thereof, and the like.

The secondary storage device 204 is a rewritable storage device such as a hard disk or a flash memory, and stores images and the like from the cameras 103 and 104 received via the communication device 205. It also stores control programs, various setting contents, processing results, and the like. This information is output from the secondary storage device 204 to the RAM 203 and used when the CPU 11 executes the process according to the control program. The communication device 205 communicates with various devices via the network 102. The communication device 205 may be a wired communication unit or a wireless communication unit. The input / output device 206 is a device that serves as an interface with the outside. The input / output device 206 includes a display that displays an image, an execution result of a control program, and the like, and a mouse, keyboard, and the like that acquire input from the user.

In the present embodiment, each process described later is realized by software using the CPU 201, but a part or all of the processes may be realized by hardware. As hardware, a dedicated circuit (ASIC), a processor (reconfigurable processor, DSP), or the like can be used. Further, software describing the processing described later may be acquired via a network or various storage media and executed by a processing device (CPU, processor) such as a personal computer.

(Functional configuration of information processing device)
FIG. 3 is a block diagram showing a functional configuration example of the information processing apparatus 101 according to the first embodiment. As shown in FIG. 3, the information processing device 101 includes an image acquisition unit 301, a detection unit 302, a notification unit 303, a detection condition change unit 304, and a confirmation result acquisition unit 305.

The image acquisition unit 301 acquires an image captured by the camera 103 or the camera 104 and outputs the image to the detection unit 302. The detection unit 302 detects a person to be detected from the image acquired by the image acquisition unit 301, and outputs the detection result to the notification unit 303. The detection conditions for detecting a person to be detected from an image are predetermined, and the detection unit 302 acquires the changed detection conditions from the detection condition changing unit 304 in response to a request.

The notification unit 303 notifies the mobile terminal 105 of the detection result output from the detection unit 102. By displaying the detection result of the notified detection unit 302 on the mobile terminal 105, the user of the mobile terminal 105 confirms the detection result of the person to be detected by the detection unit 302 via the mobile terminal 105. Can be done. The detection condition changing unit 304 changes the detection condition of the person to be detected based on the result of the visual confirmation by the user acquired by the confirmation result acquisition unit 305, and outputs the detection condition to the detection unit 302. The confirmation result acquisition unit 305 acquires the result of visual confirmation by the user from the mobile terminal 105 and outputs it to the detection condition changing unit 304.

In the present embodiment, the information processing device 101 notifies the mobile terminal 105 of the detection result and obtains the result of visual confirmation from the mobile terminal 105, but other devices different from the mobile terminal 105. May be used. For example, the detection result may be notified to the user by using the input / output device 206 of the information processing device 101, or another device may be used.

FIG. 4 is a block diagram showing a functional configuration example of the detection unit 302 shown in FIG. The detection unit 302 includes a detection unit 401, a feature extraction unit 402, a dictionary registration unit 403, a similarity calculation unit 404, a detection result determination unit 405, and a detection condition acquisition unit 406.

The detection unit 401 detects a predetermined object from the image acquired by the image acquisition unit 301, and outputs the detection result to the feature extraction unit 402. In the present embodiment, the detection unit 401 detects the face of a person as an object, and outputs the rectangular coordinates of the area including the face of the detected person as the detection result. The detection result is not limited to this, and may be the center coordinates of the detected face. The feature extraction unit 402 extracts the feature amount of the face image of the person detected from the image based on the detection result output from the detection unit 401, and the extracted feature amount is the dictionary registration unit 403 or the similarity calculation unit 404. Output to.

The dictionary registration unit 404 stores the feature amount extracted by the feature extraction unit 402 as a registration dictionary in the secondary storage device 204, and outputs the feature amount to the similarity calculation unit 404. The dictionary registration unit 404 may store the extracted feature amount in the RAM 203 as a registration dictionary. The registered dictionary is data in which a face image of a person to be detected, a feature amount thereof, and a corresponding person ID (identifier such as a number or a symbol assigned to identify an individual) are associated with each other. The person ID is acquired from, for example, the input / output device 206. The registered dictionary may include a plurality of features of a person, or may include a plurality of features for one person.

The similarity calculation unit 404 calculates the similarity between the feature amount extracted by the feature extraction unit 402 and the feature amount registered in the registered dictionary acquired from the dictionary registration unit 403, and detects the calculated similarity. Output to the result determination unit 405. The detection condition acquisition unit 406 acquires the detection condition of the person to be detected from the detection condition change unit 304 and outputs it to the detection result determination unit 405.

The detection result determination unit 405 is a person to be detected from the detection result, that is, the image acquired by the image acquisition unit 301, based on the similarity output from the similarity calculation unit 404 and the detection condition acquired by the detection condition acquisition unit 406. Determine if it could be detected. In the present embodiment, the detection result determination unit 405 determines whether or not the face of the person detected from the image by the detection unit 401 is the target person registered in advance in the dictionary registration unit 403. The detection result determination unit 405 associates this detection result with the face image included in the registration dictionary and outputs it to the notification unit 303.

The process related to the detection of the person to be detected by the detection system in the first embodiment relates to the detection process for detecting the person to be detected, the condition change process for changing the detection condition, and the registration of the person to be detected. It includes three processes of dictionary registration process. Each process will be described below.

(Detection processing)
First, in the detection system of the present embodiment, a process (detection process) in which the information processing device 101 detects a person to be detected from the captured image while capturing an image with the cameras 103 and 104 will be described. FIG. 5 is a flowchart showing an example of the detection process according to the first embodiment.

In the detection process, first, in step S501, the image acquisition unit 301 determines whether or not there is an instruction to end the detection process. The end instruction of this detection process is acquired from the input / output device 206. When the image acquisition unit 301 determines that there is an instruction to end the detection process (Yes), the detection process shown in FIG. 5 is terminated. On the other hand, when the image acquisition unit 301 determines that there is no end instruction of the detection process (No), the image acquisition unit 301 acquires the image captured by the cameras 103 and 104 in step S502.

Subsequently, in step S503, the detection unit 302 performs an object detection process for detecting a person to be detected from the image using the image acquired in step S502. The object detection process in step S503 will be described later.

Next, in step S504, the detection unit 302 determines whether or not the person to be detected is detected from the image in the object detection process in step S503. When the detection unit 302 determines that any of the people to be detected is detected from the image (Yes), the detection unit 302 outputs the detection result to the notification unit 303 in step S505. On the other hand, when the detection unit 302 determines that no person to be detected has been detected (No), the process returns to step S501.

In step S505, the notification unit 303 notifies the mobile terminal 105 of the detection result output from the detection unit 302. The detection result is notified by associating the detected person ID with the face image registered in the registration dictionary and the face image at the time of detection from the image. As described above, the method of notifying the detection result is transmitted to the mobile terminal 105 using wireless communication. By displaying this information on the mobile terminal 105, the user of the mobile terminal 105 can confirm the detection result by the detection unit 302 via the mobile terminal 105. The content to be notified will be described in detail in the description of the condition change process. When the notification unit 303 notifies the mobile terminal 105 of the detection result in step S505, the process returns to step S501.

The above-mentioned processing is repeated until the image acquisition unit 301 determines that there is an instruction to end the detection processing in step S501. The above is the detection process.

FIG. 6 is a flowchart showing an example of an object detection process executed by the detection unit 302 in step S503 of FIG.
When the object detection process is started, in step S601, the detection unit 401 performs detection processing on the image acquired in step S502 of FIG. 5 to detect a person's face from the image. As a method for detecting a person's face from an image, a known technique may be used. For example, the technique described in Document A below can be used.

Reference A: P.I. Viola and M. Jones, "Robust real-time face detection", pp. 747, Eighth International Conference on Computer Vision (ICCV'01), Volume 2, 2001.

A face image (face image) is cut out from the entire image based on the coordinates on the face image detected by such a face detection technique. At this time, image normalization is performed in order to make the in-plane rotation of the face with respect to the image surface constant. For example, the image may be rotated so that the straight line connecting both eyes of the face is horizontal to the image. In the present embodiment, the entire range in the acquired image is targeted for processing, but the range in the acquired image is specified in advance, and only the faces in the range are targeted for processing. May be good.

Next, in step S602, the feature extraction unit 402 performs a feature extraction process on the face image detected in step S601 to extract the feature amount of the face image. As a method for extracting the feature amount of the image, a known technique may be used, and for example, an LBP (Local Binary Pattern) feature amount can be used. Further, a HOG (Histogram of Oriented Gradients) feature quantity or a SIFT (Scale-Invariant Feature Transition) feature quantity may be used. Moreover, you may use the feature quantity which mixed these. The feature amount extracted by CNN (Convolutional Neural Network) learned in advance may be used. Further, the extracted feature amount may be dimensionally compressed using PCA (Principal Component Analysis) or the like. In the present embodiment, the feature amount is extracted from the entire face image, but the feature amount may be extracted from a partial region of the face image. For example, the position of an organ such as the eyes or nose of the face may be detected, a partial region may be set based on the detected organ position, and a feature amount may be extracted from each partial region.

Next, in step S603, the similarity calculation unit 404 acquires a registration dictionary in which information relating to the person to be detected is registered. As described above, the registration dictionary is one in which the feature amounts of the person to be detected (the person to be authenticated) are registered in advance, and includes the feature amounts of a plurality of persons. Here, it is assumed that the registration of the information related to the person to be detected has been completed in advance. The process of registering the information related to the person to be detected in the registration dictionary (dictionary registration process) will be described later.

Next, in step S604, the similarity calculation unit 404 determines the similarity for all combinations of the feature amount extracted by the feature extraction process in step S602 and the feature amount registered in the registration dictionary acquired in step S603. Is calculated. As a method for calculating the similarity, a known technique may be used. For example, the cosine similarity shown in the following (Equation 1) can be used. In (Equation 1), x and y are feature vectors, S is the similarity between feature vectors x and y, and · means an inner product.

When a plurality of feature quantities are registered in the registration dictionary for one person, the similarity calculation unit 404 calculates the similarity degree with the feature quantity extracted by the feature extraction process for each of them, and obtains the maximum similarity degree. Adopt as a similarity for that person. As a result, the similarity calculation unit 404 calculates the same number of similarity as the number of persons registered in the registration dictionary. Other methods may be used for the processing when a plurality of feature quantities are registered in the registration dictionary for one person. For example, a plurality of registered feature quantities may be integrated into one feature quantity and then the similarity may be calculated, or when determining the detection result after calculating the plurality of similarities, a plurality of features may be calculated. The detection result may be determined based on the similarity of.

Next, in step S605, the detection result determination unit 405 determines the detection result based on the similarity calculated in step S604 and the detection condition acquired by the detection condition acquisition unit 406. In the present embodiment, the detection condition is a threshold value, and the detection result determination unit 405 determines that the person is detected when the similarity degree of each person registered in the registration dictionary exceeds the threshold value. If it is less than or equal to the threshold value, it is determined that the person is not the person. That is, the detection result determination unit 405 determines a person ID satisfying S> θ as a detection result with respect to the similarity S for each person as a threshold value θ. When the similarity exceeds the threshold value for a plurality of people, the detection result is narrowed down to one person by selecting the person ID having the larger threshold value. In addition, the method of narrowing down when the similarity exceeds the threshold value for a plurality of persons may be another method. The above is the description of the object detection process.

(Condition change processing)
Next, the process related to the detection condition (condition change process) after the detection result is notified to the user in the above-mentioned detection process will be described. This condition change process is executed in parallel with the detection process described above, and the detection condition is changed according to the result of the user visually confirming the person detected by the detection process. FIG. 7 is a flowchart showing an example of the condition change process in the first embodiment.

In step S701, the confirmation result acquisition unit 305 determines whether or not the detection result has been notified to the mobile terminal 105 by the detection process described above. When the confirmation result acquisition unit 305 determines that the detection result has been notified to the mobile terminal 105 (Yes), the process proceeds to the next process. On the other hand, when the confirmation result acquisition unit 305 determines that the detection result has not been notified to the mobile terminal 105 (No), it waits until the detection result is notified.

Subsequently, in step S702, the mobile terminal 105 that has received the detection result displays the detection result and prompts the user of the mobile terminal 105 to visually confirm the detection result. Specifically, as shown in FIG. 8, the detection result is displayed on the mobile terminal 105, and whether or not the detection result displayed on the mobile terminal 105 is correct, that is, the detected person is the person to be detected. Prompt the user to visually confirm whether or not. The mobile terminal 105 accepts the input of the result of the visual confirmation by the user and notifies the information processing apparatus 101.

FIG. 8 is a schematic diagram illustrating a display example of the detection result displayed on the mobile terminal 105. As shown in FIG. 8, for example, the detection result 801 and the detected registration dictionary 802, the face frame 803 of the detected person, the buttons 804 and 805 used to input the result of visually confirming the detection result, and the registration dictionary are registered. Face images 806, 807, etc. are displayed. The visual confirmation result input button includes a <correct> button 804 used when it is determined that the detection result is correct, and an <error> button 805 used when it is determined that the detection result is incorrect. When a plurality of face images are registered in the registration dictionary, the plurality of face images 806 and 807 are displayed on the screen as shown in FIG.

The detection result 801 is an image taken by the camera 103 or the camera 104. Since there is a possibility that a plurality of people are captured in the images taken by the cameras 103 and 104, the face frame 803 is displayed at the position of the face of the detected person. In this example, it is shown that the person registered as the person ID "0001" is detected, and the registered face images 806 and 807 are displayed side by side in the registration dictionary 802. By displaying in this way, the user of the mobile terminal 105 can easily determine whether or not the detection result is correct. The first face image 806 and the second face image 807 are different in the time and place of shooting, and the appearance is different because the shooting conditions are different. In this way, displaying facial images in various ways helps visual confirmation. The number of face images displayed on the mobile terminal 105 is not limited to this example. The user compares the detection result 801 displayed on the mobile terminal 105 with the registration dictionary 802, selects the <correct> button 804 if the detection result is determined to be correct, and presses the <error> button 805 if the detection result is determined to be incorrect. Select and enter the result of visual confirmation. In addition, when it is necessary to apply to the detection of a person requiring attention, the detection of a lost child, etc., and the user such as a security guard needs to go to the person who was finally visually confirmed, a map showing the location of the target person. Etc. may be displayed simultaneously on the mobile terminal 105.

Returning to FIG. 7, in step S703, the confirmation result acquisition unit 305 acquires the result of visual confirmation by the user of the mobile terminal 105 about the detection result from the mobile terminal 105.

Next, in step S704, the detection condition changing unit 304 indicates whether the result of the visual confirmation acquired in step S703 is <correct> indicating that it is determined to be correct, or <indicating that it is determined to be incorrect. Determine if error>. When the detection condition changing unit 304 determines that the result of the visual confirmation is <correct> (Yes), the process proceeds to step S705. On the other hand, when the detection condition changing unit 304 determines that the result of the visual confirmation is <error> (No), the condition changing process for this detection result is completed, and a new detection result is notified. ..

In step S705, the detection condition changing unit 304 confirms whether or not there is an instruction to change the setting of the detection condition. The instruction for changing the setting of the detection condition may be acquired via the input / output device 206 of the information processing device 101, or may be acquired from the mobile terminal 105. When operating such a system, it may not be necessary to change the detection conditions depending on the situation. Therefore, when the detection result is determined to be correct by visual confirmation, the detection conditions are not changed uniformly, but are detected. Convenience is enhanced by confirming whether to change the conditions. If the detection condition changing unit 304 determines that there is an instruction to change the setting of the detection condition (Yes), the process proceeds to step S706. If the detection condition changing unit 304 determines that there is no instruction to change the setting of the detection condition (No), the condition change processing for the detection result is completed, and a new detection result is notified.

In step S706, the detection condition changing unit 304 changes the detection condition related to the detection of a person. In the present embodiment, the detection condition changing unit 304 changes the threshold value, which is the detection condition used by the detection result determination unit 405, based on the following (Equation 2).
θ'= θ + r ... (Equation 2)
In (Equation 2), θ'is the new threshold value after the change, θ is the threshold value before the change, and r is the change amount.

The change amount r is calculated in advance. In this embodiment, the change amount r is a negative value so as to lower the threshold value so that the person confirmed to be the person to be detected by the user's visual confirmation can be easily detected again. If the threshold value, which is the detection condition, is lowered, it becomes easier to detect the person again, but there is a possibility that false positives will increase. It is advisable to decide how much the quantity r should be. The method of changing the detection condition is not limited to this. For example, the detection condition may be changed by multiplying the threshold value before the change by a predetermined coefficient or calculating based on the number of registered persons.

Here, the changed detection condition is not applied to the detection of all the persons registered in the registered dictionary, but is applied only to the detection of the person confirmed to be the person to be detected by visual confirmation. By doing so, not only is it easier to detect the person again, but also the detection condition (threshold value) is relaxed only for that person, so that false detection occurs as compared with the case where the detection condition is uniformly relaxed for all people. Can be reduced.

Since the detection conditions have been changed as described above, the subsequent detection processing is performed using the new detection conditions. That is, the detection system can more easily detect a person who is confirmed to be a person to be detected by visual confirmation by the user. This is because the threshold value of the similarity used for the determination in S605 of FIG. 6 is lowered for the person confirmed to be the person to be detected. In the present embodiment, since the images of all the cameras of the detection system are processed by the information processing device 101, only the detection conditions of the information processing device 101 need to be changed, but when the detection process is performed inside each camera. The changed detection conditions may be transferred to each camera. The above is the description of the condition change process.

In the detection system of the present embodiment, the person detected by the detection process is visually confirmed from the images taken by the cameras 103 and 104, and the user can immediately go to the detected person. However, even if you lose sight of the person because you cannot go immediately due to congestion, etc., the detection conditions have been changed so that the person can be easily detected, so there is a possibility that it can be detected again from another captured image. Increase. For example, even if a person who is detected and visually confirmed from the image taken by the camera 103 is lost, the possibility that the person can be detected again from the image taken by the camera 104 is increased. Further, even when the camera 103 deviates from the shooting range and then returns to the shooting range, the detection condition is changed in the same manner, so that the possibility of detection again increases.

(Dictionary registration process)
Next, a process related to registration of a specific person to be detected (dictionary registration process) will be described. In the example described below, a case where information on a person to be detected is registered based on an image taken by a camera different from the above-described configuration will be described. The camera for registering an image in the registration dictionary is not limited to this. This dictionary registration process operates in parallel with the above-described detection process in response to a request. FIG. 9 is a flowchart showing an example of the dictionary registration process according to the first embodiment.

In the dictionary registration process, first, in step S901, the image acquisition unit 301 determines whether or not there is an instruction to end the dictionary registration process. The end instruction of this dictionary registration process is acquired from the input / output device 206. When the image acquisition unit 301 determines that there is an instruction to end the dictionary registration process (Yes), the dictionary registration process shown in FIG. 9 is terminated. On the other hand, when the image acquisition unit 301 determines that there is no end instruction of the dictionary registration process (No), the image acquisition unit 301 acquires the image to be registered in step S902. As mentioned above, this image is an image pre-taken by another camera.

Subsequently, in step S903, the detection unit 401 of the detection unit 302 performs detection processing on the image acquired in step S902, and detects a person's face from the image. Since the process of detecting a person's face from an image has already been described, the description thereof will be omitted here. Next, in step S904, the feature extraction unit 402 of the detection unit 302 performs a feature extraction process on the face image detected in step S903, and extracts the feature amount from the face image. Since the process of extracting the feature amount from the face image has already been described, the description thereof is omitted here.

Next, in step S905, the dictionary registration unit 403 of the detection unit 302 acquires the person ID corresponding to the feature amount extracted in step S904 via the input / output device 206. Next, in step S906, the dictionary registration unit 403 associates the extracted feature amount with the corresponding face image and person ID and stores them in the registration dictionary. After registering the information of the person to be detected in step S906 in the registration dictionary, the process returns to step S901, and the subsequent processing is continued until there is an instruction to end the dictionary registration process. The above is the explanation of the dictionary registration process. The person registered in this dictionary registration process is detected by the detection process described above.

According to the first embodiment, by changing the detection condition using the result of the user visually confirming the detection result by the detection process, the judgment result by the human eye can be reflected in the system, which is convenient. improves. In addition, since the detection condition is changed so that the visually confirmed person can be easily detected again, the possibility that the person can be detected again by the camera at another place increases. As a result, the possibility of being able to detect the lost person to be detected again is increased, and the convenience is improved.

Further, in the present embodiment, the visual confirmation of the detection result is performed using only the mobile terminal 105, but another means may be used. For example, the mobile terminal may display only the face image of the registered dictionary, and the user of the mobile terminal may go to the detected person and directly visually confirm and input the result. However, in a crowded situation, it is difficult to visually check directly. In such a case, it is desirable to take the method as described above.

Further, in the present embodiment, the changed detection conditions are used to detect the images acquired from all the cameras, but the detection conditions may be changed only for the images acquired from a specific camera. Good. For example, the changed detection condition is applied only to the image acquired from the camera which is close to the detected place. By doing this, it is possible to change the setting of the detection condition only for the camera in the place where the detected person is likely to appear next, and it is possible to prevent the false detection by the camera that is unlikely to appear from increasing. it can.

Further, in the present embodiment, as shown in FIG. 8, the detection result is displayed together with the image of the person to be detected, but the display method is not limited to this. In particular, when the detection conditions are changed, the number of false positives is expected to increase, so the display method may be switched accordingly. For example, in FIG. 8, the detection results are displayed one by one, but when a plurality of candidates for the person to be detected are detected, for example, the images of the candidates are displayed in tiles. This makes it easier for the user to visually confirm which is the true person to be searched.

<Second embodiment>
Next, a second embodiment of the present invention will be described. In the first embodiment described above, an example in which the user visually confirms whether the detection result in the detection process by the information processing device is correct or incorrect has been described. However, in reality, it may not be possible to clearly determine whether the detection result is correct or incorrect in the visual confirmation by the user, so it is convenient to be able to indicate whether the detection result is correct or incorrect.

In the second embodiment, the result of visual confirmation of the detection result of the detection process is expressed by the degree of correctness, that is, a plurality of values (continuous value or multi-value) instead of binary, and according to the result of visual confirmation. An example of changing the detection condition will be described. Since the second embodiment is the same as the first embodiment described above except for the condition change process, the description other than the condition change process will be omitted, and only the condition change process will be described below. Since the basic flow of the condition change process in the second embodiment is the same as the condition change process in the first embodiment shown in FIG. 7, the specific process will be described with reference to FIG. 7 as appropriate. ..

In the condition change process of the second embodiment, in step S701, the confirmation result acquisition unit 305 determines whether or not the detection result of the detection process has been notified to the mobile terminal 105. When the confirmation result acquisition unit 305 determines that the detection result has not been notified to the mobile terminal 105 (No), it waits until the detection result is notified. On the other hand, when the confirmation result acquisition unit 305 determines that the detection result has been notified to the mobile terminal 105 (Yes), in step S702, the mobile terminal 105 that has received the detection result displays the detection result and the user of the mobile terminal 105. Is prompted to visually confirm the detection result. In the second embodiment, the user is urged to visually confirm the degree of correctness of the detection result displayed on the mobile terminal 105, that is, the certainty that the detected person is the person to be detected. The mobile terminal 105 accepts the input of the result of the visual confirmation by the user and notifies the information processing apparatus 101.

FIG. 10 is a schematic diagram illustrating a display example of a detection result displayed on the mobile terminal 105 in step S702 in the condition change process in the second embodiment. As shown in FIG. 10, for example, the detection result 1001, the registration dictionary 1002, the face frame 1003 of the detected person, the slide bar 1005, the face images 1006, 1007 registered in the registration dictionary, the visual confirmation result transmission button 1008, and the like. Is displayed. The slide bar 1005 is used to input the degree of correctness of the detection result as a result of visual confirmation.

In the first embodiment, the binary value of <correct> or <wrong> was selected as the result of visual confirmation, but in the second embodiment, the degree of correctness of the detection result can be expressed by the slide bar 1005. ing. When the user of the mobile terminal 105 selects the degree with the slide bar 1005 and selects the visual confirmation result transmission button 1006, the mobile terminal 105 transmits the visual confirmation result to the information processing device 101.

As a result, in step S703, the confirmation result acquisition unit 305 acquires the result of visual confirmation of the detection result from the mobile terminal 105. In the present embodiment, the degree w indicating the correctness of the detection result takes a real value, for example, the left end (wrong) of the slide bar 1005 has a value of 0.0, and the right end (positive) has a value of 1.0. To do so. By doing so, even if the user who performs the visual confirmation cannot make a judgment with confidence due to the influence of the appearance of the detection result or the like, an ambiguous judgment can be expressed by the visual confirmation.

In the next step S704, in the condition change process in the second embodiment, the detection condition change unit 304 determines whether or not the degree of correctness acquired as a result of visual confirmation is larger than a predetermined threshold value. When the detection condition changing unit 304 determines that the acquired degree of correctness is larger than a predetermined threshold value (Yes), the detection condition changing unit 304 determines that the result of visual confirmation is correct and proceeds to step S705. On the other hand, when the detection condition changing unit 304 determines that the acquired correctness is equal to or less than a predetermined threshold value (No), the result of the visual confirmation is regarded as an error, and the condition changing process for the detection result is terminated. Wait for the notification of the new detection result.

In step S705, the detection condition changing unit 304 confirms whether or not there is an instruction to change the setting of the detection condition, as in the first embodiment. If the detection condition changing unit 304 determines that there is an instruction to change the setting of the detection condition (Yes), the process proceeds to step S706. If the detection condition changing unit 304 determines that there is no instruction to change the setting of the detection condition (No), the condition change processing for the detection result is completed, and a new detection result is notified.

In step S706, the detection condition changing unit 304 changes the detection condition related to the detection of a person. In the second embodiment, the detection condition changing unit 304 changes the threshold value, which is the detection condition used by the detection result determination unit 405, based on the following (Equation 3).
θ'= θ + wr ... (Equation 3)
In (Equation 3), θ'is the new threshold value after the change, θ is the threshold value before the change, w is the degree of correctness, and r is the amount of change. In the present embodiment, as described above, the degree of correctness w is indicated by a value of 0.0 to 1.0, and the value obtained by multiplying the predetermined change amount r is used as the actual threshold change amount. By doing so, when it is confidently judged that the detection result is correct, the degree w approaches 1.0 and the threshold change amount becomes large, and conversely, when the judgment is not confident. The degree w approaches 0.0, and the amount of change in the threshold value becomes small. By doing so, the result of unconfident visual confirmation has almost no effect on the accuracy of detection, and it is possible to suppress the change of the detection condition based on the result of erroneous visual confirmation, so that the reliability of the detection system can be suppressed. Will increase.

The method of changing the detection condition is not limited to this. Even if you are not confident in the result of the visual confirmation, there may be a use case where you want to detect a potential person again anyway. In such a case, contrary to the present embodiment, if the result of the visual confirmation is not confident, a large value may be corrected to make it easier to detect again.

According to the second embodiment, a more flexible system can be realized by expressing the result of visual confirmation not by the binary value of correctness or error but by the degree of correctness.

<Other Embodiments>
Various methods other than the above-mentioned methods can be considered for displaying the image used for visual confirmation. The upper part of FIG. 11 shows an image sequence (enlarged only of the face image) taken by the cameras 103 and 104. Frames 1101-1104 are shown in order of time taken. In the above-described embodiment, only one frame is displayed on the mobile terminal 105, but all frames of the detected (tracked) image sequence may be displayed. By displaying a plurality of frames in this way, the user can visually confirm the images having different appearances before and after, which enhances convenience.

Various methods other than the above-mentioned methods can be considered for changing the detection conditions. In the above-described embodiment, the threshold value is changed by a predetermined constant, but it may be changed dynamically. At the bottom of FIG. 11, the transition of the similarity of a person corresponding to the surveillance camera image sequence at the top is shown by the solid line 1105. The similarity exceeds the threshold from the left to the third frame, and the similarity is less than the threshold in the last frame. As a result of visual confirmation, if it is determined that the detection result of this person is correct, the threshold value is changed based on this information. For example, if the lowest similarity (last frame) among them is set as a threshold value, the possibility that this person can be detected again increases. In addition, a method such as multiplying the average, median, or maximum value of the similarity of all frames by a predetermined ratio can be considered. Either method is a trade-off between the merit of increasing the possibility of re-detection and the demerit of increasing false positives, so it is desirable to select it according to the use case.

Further, the registered dictionary may be used as a detection condition and this may be changed. That is, in the first embodiment, the detection condition changing unit 304 and the dictionary registration unit 403 have separate configurations, but these are put together so that the detection condition changing unit has the function of the dictionary registration unit. .. If the image of the camera when the person to be detected is additionally registered in the registration dictionary, the possibility that the person can be detected by another camera increases. For example, when the user visually confirms by looking at the frame 1102 of FIG. 11 and determines that the detection result is correct, the image of the frame 1102 is additionally registered in the registration dictionary of the person. In addition, all the images of frames 1101 to 1104 may be additionally registered, or may be selected and registered based on temporal information, that is, the time when the images were taken. For example, it is conceivable to additionally register only continuous images (frames 1101 and 1102) to the visually confirmed image.

Further, the parameters used for calculating the similarity may be changed. With the cosine similarity shown as an example in this embodiment, if the origin when calculating the angle is calculated based on the image taken by the camera, it can be compared in the feature space of the shooting site, so that the accuracy is higher. It can be detected well. However, if the calculation method of the similarity is different for each camera, the comparison cannot be performed as it is, so it is necessary to calculate the normalization parameters and the like for comparison. Further, in the present embodiment, the person is detected based on the face image, but only the person who can be visually confirmed may be detected based on other information. For example, if it is detected by the attribute information such as the clothes and gender of the person who can be visually confirmed, the false detection increases, but the possibility of detecting it again increases.
In the above description, an example of detecting a person to be detected has been described, but the subject to be detected is not limited to the person. The subject to be detected may be, for example, a car or a specific object.

Of the above-mentioned processing units, artificial intelligence (AI) may be applied to the feature extraction unit 402 and the like. For example, a machine-learned trained model may be used instead of the processing unit. In that case, a plurality of combinations of input data and output data to the processing unit are prepared as learning data, knowledge is acquired from them by machine learning, and output data for the input data is obtained as a result based on the acquired knowledge. Generate a trained model that outputs as. This trained model can be constructed, for example, by a neural network model. Then, the trained model performs the processing of the processing unit by operating in collaboration with the CPU, GPU, or the like as a program for performing the same processing as the processing unit. Further, the trained model can be updated every time a certain amount of data is processed, as needed.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

It should be noted that the above-described embodiments are merely examples of embodiment of the present invention, and the technical scope of the present invention should not be construed in a limited manner by these. That is, the present invention can be implemented in various forms without departing from the technical idea or its main features.

101: Information processing device 102: Network 103, 104: Surveillance camera 105: Mobile terminal 301: Image acquisition unit 302: Detection unit 303: Notification unit 304: Detection condition change unit 305: Confirmation result acquisition unit 401: Detection unit 402: Features Extraction unit 403: Dictionary registration unit 404: Similarity calculation unit 405: Detection result determination unit 406: Detection condition acquisition unit

Claims (18)

Image acquisition means to acquire images and
A detection means for detecting a subject to be detected from the image and
Confirmation result acquisition means for acquiring the result of visual confirmation by the user of the detection result by the detection means, and
Information characterized in that the detection means has a detection condition changing means for changing the detection condition when detecting the subject to be detected based on the result of the visual confirmation acquired by the confirmation result acquisition means. Processing equipment. When the result of the visual confirmation acquired by the confirmation result acquisition means indicates that the detection result by the detection means is correct, the detection condition changing means makes it easier to detect the subject to be detected. The information processing apparatus according to claim 1, wherein the information processing apparatus is changed. The information processing device according to claim 1 or 2, wherein the detection condition changing means changes only the detection condition of the visually confirmed subject among the registered subjects to be detected. The information processing apparatus according to any one of claims 1 to 3, wherein the detection condition changing means changes the detection condition in response to an instruction to change the setting of the detection condition. The detection means
A dictionary registration means for registering the subject to be detected in the registration dictionary, and
A detection means for detecting a predetermined object from the image, and
A similarity calculation means for calculating the similarity between the registered dictionary and the object detected by the detection means, and a similarity calculation means.
It has a detection result determining means for determining a detection result based on the similarity and a threshold value calculated by the similarity calculating means.
The information processing apparatus according to any one of claims 1 to 4, wherein the detection condition changing means changes the threshold value based on the result of the visual confirmation. The detection means has a feature extraction means for extracting a feature amount of the object.
The information processing according to claim 5, wherein the similarity calculation means calculates the similarity between the feature amount registered in the registration dictionary and the feature amount of the object detected by the detection means. apparatus. The information according to any one of claims 1 to 6, wherein the result of the visual confirmation acquired by the confirmation result acquisition means is information indicating whether or not the detection result by the detection means is correct. Processing equipment. The information according to any one of claims 1 to 6, wherein the result of the visual confirmation acquired by the confirmation result acquisition means is information indicating the degree of correctness of the detection result by the detection means. Processing equipment. The information processing device according to any one of claims 1 to 8, further comprising a notification means for notifying the detection result by the detection means. The information processing device according to claim 9, wherein the notification means displays an image in which the detection means detects a subject to be detected together with a detection result by the detection means. The notification means is characterized in that, when the detection condition changing means changes the detection condition, a plurality of images obtained by the detection means detecting the subject to be detected are displayed together with the detection result by the detection means. The information processing apparatus according to claim 9. The information processing device according to any one of claims 1 to 11, further comprising a display means for displaying the detection result by the detection means. The information processing device according to any one of claims 1 to 12, wherein the subject to be detected is a specific person. With one or more cameras
An information processing system including the information processing device according to any one of claims 1 to 13, wherein the image acquisition means acquires the image taken by the camera. The information processing system according to claim 14, further comprising a terminal that receives and displays the detection result by the detection means from the information processing device and transmits the result of visual confirmation by the user to the information processing device. .. When the result of the visual confirmation acquired by the confirmation result acquisition means indicates that the detection result by the detection means is correct, the claim is characterized in that an image of the subject of the detection result is registered in the registration dictionary. The information processing apparatus according to 5. Image acquisition process to acquire images and
A detection process that detects a subject to be detected from the image, and
A confirmation result acquisition process for acquiring the result of visual confirmation by the user of the detection result in the detection process, and
Based on the result of the visual confirmation acquired in the confirmation result acquisition step, the detection step is characterized by having a detection condition changing step of changing the detection condition when detecting the subject to be detected. Information processing method to do. Image acquisition steps to acquire images and
A detection step that detects the subject to be detected from the image, and
A confirmation result acquisition step for acquiring the result of visual confirmation by the user of the detection result in the detection step, and
In order to cause the computer to execute a detection condition change step of changing the detection condition when detecting the subject to be detected in the detection step based on the result of the visual confirmation acquired in the confirmation result acquisition step. Program.

Images