JP2021027399A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To provide an information processing apparatus detecting a person detection and a behavior detection of a person from an image, in which the image to be input is suppressed and a processing load applied to the behavior detection is reduced.SOLUTION: An information processing apparatus 100 including a person detection part 203 performing a person detection from an image and a behavior detection part 204 performing a person behavior detection, includes: a determination part 202 determining each imaging device that imaged the image that the same person is detected by the person detection part 203 as a first imaging device imaging the image for detecting the other person different from the same person by the person detection part 203 or a second imaging device imaging the image input into the behavior detection part 204 that detects a behavior of the same person; and a control part 206 that inputs the image imaged by the first imaging device into the person detection part 203 and inputs the image imaging the same person by the second imaging device into the behavior detection part 204.SELECTED DRAWING: Figure 2

Description

The present invention relates to a technique for detecting a specific person or the behavior of a specific person from an image.

In recent years, a technique of installing a plurality of imaging devices in a store and monitoring the behavior of a visitor based on the captured images has become widespread. In such a technique, a plurality of stages of detection processing may be performed. For example, a person detection is performed in which a person existing in a monitoring area of each imaging device in a store is detected from an image. Furthermore, behavior detection is performed to detect from the video whether the detected person is performing suspicious behavior (look around, looking for something) that is a sign of a crime. In Patent Document 1, a person detected from an image taken by an image pickup device for person detection is tracked by the image pickup device for tracking.

Japanese Unexamined Patent Publication No. 2013-168757

In Patent Document 1, when a specific person can be imaged by a plurality of tracking imaging devices, tracking of the specific person, that is, behavior detection is recognized and processed by images captured by the plurality of imaging devices. In that case, the processing load required for behavior detection may increase. The present invention has been made in view of the above problems, and an image input to behavior detection is performed by selecting an imaging device suitable for behavior detection, instead of all imaging devices capable of detecting behavior of a specific person. The purpose is to reduce the processing load required for behavior detection.

The information processing device according to the present invention that solves the above problems is an information processing device having a person detection unit that detects a person from an image and an action detection unit that detects a person's behavior, and the same person is detected by the person detection unit. The first imaging device that captures an image that detects another person different from the same person by the person detection unit, or an action that detects the behavior of the same person, for each of the imaging devices that have captured the detected image. A determination means for determining as a second imaging device that captures an image input to the detection unit, and an image captured by the first imaging device are input to the person detection unit, and the same person is captured by the second imaging device. It is characterized by having a control means for controlling the captured image to be input to the behavior detection unit.

According to the present invention, by selecting an imaging device suitable for behavior detection instead of all imaging devices capable of detecting the behavior of a specific person, the image input to the behavior detection is suppressed and the processing related to the behavior detection is performed. The load can be reduced.

The figure which shows the hardware configuration example of an information processing apparatus Block diagram showing a functional configuration example of an information processing device Flowchart showing an example of processing executed by an information processing device The figure which shows an example which shows the monitoring target of an image pickup apparatus The figure which shows an example of the table (imaging device information table) which manages the information for each image pickup apparatus. The figure which shows an example which shows the monitoring target of an image pickup apparatus The figure which shows an example of the table (imaging device information table) which manages the information for each image pickup apparatus. The figure which shows an example which shows the monitoring target of an image pickup apparatus The figure which shows an example of the table (imaging device information table) which manages the information for each image pickup apparatus. The figure which shows an example which shows the monitoring target of an image pickup apparatus Block diagram showing a functional configuration example of an information processing device Flowchart showing an example of processing executed by an information processing device The figure which shows an example of the table (imaging device information table) which manages the information for each image pickup apparatus. Block diagram showing a functional configuration example of an information processing device Flowchart showing an example of processing executed by an information processing device The figure which shows an example of the table (imaging device information table) which manages the information for each image pickup apparatus. Flowchart showing an example of processing executed by an information processing device The figure which shows an example of the table (imaging device information table) which manages the information for each image pickup apparatus. Flowchart showing an example of processing executed by an information processing device

<Embodiment 1>
Hereinafter, embodiments will be described with reference to the drawings. The configuration shown in the following embodiments is only an example, and the present invention is not limited to the illustrated configuration.

FIG. 1 is a diagram showing a hardware configuration example of the information processing device 100. The information processing device 100 includes a CPU 10, a ROM 11, a RAM 12, an external memory 13, a network I / F (interface) 14, a display unit 15, an input unit 16, and a system bus 18 as hardware configurations.

The central processing unit (CPU) 10 uses the RAM 12 as a work memory to read and execute the OS and other programs stored in the ROM 11 and the external memory 13, and controls each configuration connected to the system bus 18 to control various configurations. Performs processing operations and logical judgments. The ROM 11 stores data, programs, and the like used by the CPU 10 for processing. The external memory 13 is a hard disk, SSD, or the like, and stores data, programs, or the like that cannot be stored in the ROM 11. These data and programs may be acquired from other devices on the network via the network I / F14. The network I / F14 is an interface for connecting the information processing device 100 to the network. The display unit 15 is a liquid crystal display device or the like, and displays the result of processing by the CPU 10 or the like. The input unit 16 is a mouse, a button, or the like, and inputs a user's operation to the information processing device 100. In addition, each of these hardware may have a configuration of another device connected to the information processing device 100. When the CPU 10 executes processing based on the programs stored in the ROM 11 and the external memory 13, the functional configuration of the information processing apparatus 100 of FIG. 2 described later and the processing of the flowchart of FIG. 3 described later are realized.

FIG. 2 is a diagram showing an example of the functional configuration of the information processing device 100. The information processing device 100 is an information processing device that detects a person from images captured by a plurality of imaging devices, and includes a determination unit 202, a person detection unit 203, an action detection unit 204, a presentation unit 205, a control unit 206, and an evaluation unit. A unit 208, a storage unit 207, and a selection unit 209 are included. The imaging devices 401, 403, 405, and 407 are installed in advance in the store, which is the monitoring target area, so that no blind spots are generated in the store and the monitoring areas of the imaging devices do not overlap as much as possible. Here, an example of monitoring by four imaging devices 401, 403, 405, and 407 will be described.

The determination unit 202 uses each of the image pickup devices that capture an image capable of detecting the same person based on the result of the person detection process for detecting a person having a predetermined feature from the image, and sets another person different from the same person. A first imaging device for capturing an image to be input to the person detection process to be detected is determined. Alternatively, it is determined as a second imaging device that captures an image to be input to the second detection process for detecting the behavior of the same person. That is, the image to be input to the person detection and the image to be input to the behavior detection are classified by using the information on the detection rate for each imaging device prepared in advance. Here, among all the imaging devices, among the imaging devices that have captured the image obtained by detecting a specific person in the first detection process, the imaging device that continues the first detection process is referred to as the first imaging device. Further, among the image pickup devices that have captured the image obtained by detecting a specific person in the first detection process, the image pickup device that satisfies a predetermined condition as the image pickup device suitable for the second detection process is referred to as the second image pickup device. When there are a plurality of imaging devices that image a specific same person, the amount of images for which the behavior is detected can be reduced and the processing load can be reduced by performing the behavior detection with the minimum necessary number. The predetermined conditions are, for example, a condition that the imaging region that can be imaged by the first imaging apparatus is larger than a predetermined size, a condition regarding the correct answer rate that each detection process should satisfy, and a condition that the face of the same person is oriented. In this way, the processing load can be reduced by inputting only the image captured by the second imaging device into the second detection process. Detailed processing will be described later. Here, the first detection process is a process of detecting a person requiring attention (same person) registered in advance. When the person requiring attention is detected in the first detection process, the second detection process monitors the behavior of the person requiring attention detected in the first detection process and detects whether or not it is a criminal action.

The person detection unit 203 performs a person detection process (first detection process) on images captured by a plurality of imaging devices. Here, a person requiring attention having a pre-registered feature is detected. A face recognition process will be described as an example of the first detection process. First, the person detection unit 203 detects the face image included in the image by extracting the image features of the eyes and nose, which are the constituent elements of the face, from the input image. If the image does not include a person, the face image is not detected. When the image contains a plurality of people, the face image of each person is detected. As the face detection method, an existing known method may be used. For example, an image feature showing a shape corresponding to a component of a face image such as a nose, mouth, and eyes is extracted. The size of the face is estimated from the size of the extracted eyes and their distance, and the area surrounded by the area of the extracted size is used as the face image based on the position corresponding to the center of the nose.

Next, from the face image, a feature indicating the positional relationship of the part of the person who can identify the individual is acquired. It is assumed that the feature referred to here is an image feature that can be obtained from an image by edge detection or the like. The acquisition method may be arbitrary, but in the present embodiment, the features of the human face are extracted as follows. First, based on the positions of typical organs such as eyes, nose, and mouth identified from the image, make the image so that the line segment connecting both eyes is horizontal on the image so that the width of both eyes is a predetermined distance. Rotate / enlarge / reduce. Then, a rectangular area for extracting features is set in the face image. The size of the area is arbitrary, but a square with one side about 1.5 times the width of the eyes so that organs such as eyes and mouth that well represent individual characteristics can be inserted, but the background etc. are not included. Should be placed in the center of the face. Next, the pixel values in the rectangular area are taken out in order from the upper left to the lower right, and connected in a row to form a vector. The extraction of the characteristics of the person used in this embodiment is not limited to the method shown above. For example, a deep neural network may be used to extract individual features. Specifically, the image features and the like that can be acquired from the convolutional layer in the CNN (Convolutional Neural Network) are acquired. In this case, by inputting the face image into the neural network (learned model) trained to identify the individual, the features necessary for the identification of the individual are acquired. The trained model is a network structure based on a neural network that outputs a detection result corresponding to the captured image from the captured image and its parameters. The recognition may be, for example, identifying an individual from an image. By using the trained model, individual recognition can be performed more accurately.

The behavior detection unit 204 performs the second detection process on the image captured by the second image pickup device determined by the determination unit 202. The second detection process recognizes the behavior of a person and detects a sign of a pre-registered criminal behavior. As an example of the second detection process, abnormal behavior detection using a neural network will be described. As learning data, an image of a person who commits an abnormal act (for example, theft or violence) to be detected is labeled as an abnormal case. Other normal activities (eg walking, speaking) are labeled as normal cases. A lot of such training data is created and input to the neural network. When the trained neural network inputs an image (a time-series image obtained by capturing a series of actions), the trained neural network outputs whether the person shown in the image is a normal action or an abnormal action. The learning method is not limited to the above.

In behavior recognition, it is necessary to recognize not only the movement of the person's face but also the movement of the hands and feet. That is, in face recognition, an image in which the face is clearly reflected is appropriate, but an image suitable for behavior recognition is not limited to this. Therefore, it is necessary to select an imaging device capable of capturing an image suitable for behavior recognition. Further, if the user wants to monitor the store in real time, it is ideal that all the input images in the plurality of areas are subjected to the first detection process and the second detection process. However, computational resources are limited. For example, abnormal behavior detection has a larger processing load than face recognition because it extracts information about the whole body of a person. Therefore, if all the captured images are input to the behavior detection, the processing load becomes enormous and there is a possibility that the processing cannot be performed in real time. In such a case, a wider area can be monitored with a smaller processing load by appropriately controlling which area (imaging device) image is input to which detection process. The combination of the first detection process and the second detection process is not limited to the above combination. When the processing load of the second detection process is larger than that of the first detection process, the processing load on the information processing apparatus 100 can be further reduced.

The presentation unit 205 presents the detection result of the person detection unit 203 or the action detection unit 204 to the operator of the information processing device 100. The detection result is displayed by, for example, a display device. The detection result may be uttered by the voice generator. Specifically, when a sign of a crime of a person requiring attention is detected by the second detection process, the face and position of the person requiring attention are presented to the operator.

The control unit 206 controls the image pickup device determined by the determination unit 202 so as to image the same person. Specifically, the angle of view of the image pickup device that captures the image input to the second detection process is controlled so as to image the body of the same person detected in the first detection process.

The storage unit 207 stores the image pickup device information table of FIG. 5, which will be described later. The evaluation unit 208 evaluates the detection performance of the first detection process or the second detection process. A specific evaluation method will be described in the second embodiment. When the number of persons requiring attention detected by the first detection process is more than a predetermined number according to the number of imaging devices, the selection unit 209 is based on the priority set for each person requiring attention. Select the same person whose behavior is detected by the detection process. The priority indicates that the action is detected in descending order of priority among a plurality of persons requiring attention. For example, information about the priority for detecting an action is set in advance together with a list of people requiring attention.

FIG. 3 is a flowchart showing an example of processing executed by the information processing apparatus 100. In the following description, the notation of the process (step) is omitted by adding S at the beginning of each process (step). However, the information processing apparatus 100 does not necessarily have to perform all the steps described in this flowchart. The process shown in the flowchart of FIG. 3 is executed by the CPU 10 of FIG. 1, which is a computer, according to a computer program stored in the ROM 11.

In S301, the person detection unit 203 performs the first detection process of detecting the same person having a predetermined feature from the images captured by the plurality of imaging devices. In the first detection process, first, the characteristics of a person are extracted for all the input images. Furthermore, the characteristics of the extracted person are compared with the characteristics of the set person requiring attention. A person who is similar to a predetermined threshold value is detected as a person requiring attention. After that, unless otherwise specified, the image captured by the image pickup apparatus shall be continuously output to the person detection unit 203 or the behavior detection unit 204 for detection processing.

Subsequently, in S302, the person detection unit 203 executes the detection process (first detection) of the person requiring attention on the image captured by each imaging device, and determines whether or not the person requiring attention has been detected. Here, it is assumed that the facial features of the person requiring attention for identifying the person requiring attention are set in advance in the first detection process. The feature of the human part used for person detection may be other than the facial feature as long as it is a feature that can identify an individual. If a person requiring attention is detected as a result of the first detection process, the process proceeds to S303. At this time, information such as the ID of the imaging device that captured the person requiring attention, the ID of the person requiring attention, the position of the face of the person requiring attention on the image, and the size of the face of the person requiring attention is output to the determination unit 202. If it is not detected, the process returns to S301, the next time-series image is acquired, and the face recognition process is continuously performed on the images captured by all the imaging devices.

In S303, when the selection unit 209 detects more than a predetermined number of people requiring attention detected by the first detection process according to the number of imaging devices, the selection unit 209 is based on the priority set for each person requiring attention. The same person whose behavior is detected in the second detection process is selected. As a result, behavior monitoring can be performed stably even when there are a plurality of persons requiring attention. If there is only one person requiring attention, this step may be skipped.

In S304, the determination unit 202 captures an image for detecting an action that is a sign of a crime of a person requiring attention (second detection process) based on the processing result of the person detection unit 203 and the information of the storage unit 207. An image pickup device (hereinafter referred to as a second image pickup device) is determined. In S303, when a priority is set for a plurality of persons requiring attention, an imaging device for capturing an image for performing the second detection process is determined for each person requiring attention. As an example of the determination method, first, as a processing result of the first detection process, the first imaging device that has captured the image of the person requiring attention is specified among the detected persons. Further, among the specified first imaging devices, at least one image used for the second detection process is captured so that the monitoring area by the first image pickup device covered by the image input to the first detection process is maximized. Determine one or more second imaging devices. Here, when the same person requiring attention is detected from the images of a plurality of imaging devices in the first detection process, the processing load becomes large if all the images of the plurality of imaging devices are input to the second detection process. It ends up. As a result, there is a risk that the person requiring attention cannot be detected in real time. In order to prevent the processing load from becoming too large, some of the imaging devices that have captured the image of the person requiring attention in the first detection process should capture the image to be input to the second detection process. To do. As a result, the processing load can be reduced.

Hereinafter, the flow of the switching process will be described with reference to FIGS. 4 and 5. FIG. 4 is an example showing how the image pickup devices 401, 403, 405, and 407 are installed in the store 410. The images taken by the image pickup devices 401, 403, 405, and 407 are subjected to a person requiring attention detection process (first detection process).

Here, in the areas 402, 404, 406, and 408, the first detection process (face recognition process) and the second detection process (behavior detection) of the person whose image pickup devices 401, 403, 405, and 407 are the detection targets are predetermined correct answers, respectively. It shows the monitoring area that can be implemented at a rate (eg 70%).

The monitoring area is calculated from the position / orientation of the image pickup device, the angle of view, and the maximum distance from the recognizable image pickup device to the subject described in the image pickup device information table of FIG. The image pickup device information table is created by the user measuring a monitoring area in which the image pickup device is actually installed, the detection process is executed, and the detection can be performed correctly. The image pickup device information table is held in the storage unit 207, and is updated by the determination unit 202 as necessary.

441-444 are customers in the store and should be monitored for criminal activity. 5A, 5B, and 5C are examples of an image pickup device information table. Here, it is assumed that the position and orientation of the image pickup apparatus are expressed based on the coordinate axes shown in FIG. FIG. 5A is an image pickup device information table in the initial state.

In S304, the determination unit 202 further determines from the person detection unit 203 the ID of the imaging device that captured the person requiring attention, the ID of the person requiring attention, the position of the face of the person requiring attention on the image, and the face of the person requiring attention. Get information about size. The position of the person requiring attention is estimated from this information and the position / orientation, angle of view, and maximum distance of the image pickup device described in the image pickup device information table, and the image pickup device information table is updated.

FIG. 5B is an image pickup device information table when the customer 443 is detected from the images of the image pickup devices 401 and 405 as a person requiring attention. Subsequently, the position of the person requiring attention is obtained from the ID and position of the person requiring attention for each imaging device described in the image pickup device information table. In the present embodiment, the values obtained for each imaging device are averaged as the positions of people requiring attention, but the position is not limited to this and may be obtained by another method.

Subsequently, for each imaging device, whether or not the second detection process of the person requiring attention 443 is possible is determined from the monitoring area of each imaging device and the position of the person requiring attention, and from the logical sum of the monitoring areas of the detectable imaging device, Calculate the area excluding the monitoring area. Subsequently, the image pickup device output that maximizes this region is switched to the second detection process.

FIG. 5C is an image pickup device information table after the image pickup devices capable of the second detection process of the person requiring attention 443 are 401, 403, and 405, and the image pickup device for the second detection process is determined to be 405. .. FIG. 6 is an example showing the imaging device and the monitoring area corresponding to FIG. 5 (c). A person requiring attention detection processing (first detection processing) is performed on the captured images from the image pickup devices 401, 403, and 407. Behavior detection (second detection processing) is performed on the captured image from the image pickup apparatus 405.

In the subsequent S305, the behavior detection unit 204 performs a second detection process for detecting a predetermined behavior for a specific person based on the image captured by the image pickup device determined by the determination unit 202. Here, the predetermined action refers to an action that predicts a criminal action.

In S306, the action detection unit 204 determines whether or not a predetermined action of a specific person has been detected. If abnormal behavior is detected from the image of the image pickup device switched to the second detection process, the process proceeds to S307. In S307, the presentation unit 205 presents that a special person, that is, a person requiring attention has behaved abnormally. The presentation indicating that there was an abnormal behavior may be, for example, a display or a voice.

If no abnormal behavior is detected in S305, it is necessary to redetermine the second imaging device, so the process proceeds to S310. In S310, the person detection unit 203 has a predetermined interval with respect to the image captured by the image pickup device (second image pickup device) determined by the determination unit 202 together with the image captured by the image pickup device other than the second image pickup device. The first detection process is performed with. Detects whether a person requiring attention exists in the monitoring area of the second imaging device. If it does not exist in the monitoring area of the second imaging device, the process proceeds to S311. In S311 the control unit 206 inputs the image captured by the second imaging device to the person detection unit 203 and controls the angle of view to be switched so as to capture the default monitoring area. In S310, when the person requiring attention is in the monitoring area of the second imaging device, the process returns to S302 and the process is continued.

When the image pickup device controlled by the control means cannot image a person detected in the first detection process, the determination unit 202 inputs an image input to the second detection process from an image pickup device different from the second image pickup device. Determine the imaging device to image. Among other imaging devices, an imaging device whose distance from the person requiring attention is within a predetermined range (for example, 1 m to 15 m) may be determined as the second imaging device. If the distance between the person requiring attention and the imaging device is too far, it will be difficult to determine the posture of the body, and if it is too close, the state of the body cannot be captured. The distance between each image pickup device and the person requiring attention is measured from the position estimated from the image and the known absolute position of the image pickup device. In addition, for example, each imaging device may be provided with a distance sensor (not shown) to measure the distance between the person requiring attention and each imaging device. Based on the measurement result, the image pickup device in which the distance between the person requiring attention and the image pickup device is within a predetermined range is determined. As a result, even if the person requiring attention moves, the person requiring attention can be continuously detected by another imaging device.

<Modification example>
In the above specific example, the security camera in the store has been described, but other use cases may be used. For example, the present invention can be applied even when a fall detection of an elderly person is performed in a facility having many elderly people, mainly in a hospital or a long-term care facility. Since it is necessary to process the posture estimation of a person over several frames in the fall detection, if it is applied to all the people in the monitoring area, the processing load becomes large. At this time, in the first detection process, the age of the person detected as a predetermined feature is estimated, and when the age higher than the predetermined age (for example, 60 years or older) is estimated, the person is detected. For estimating the age of a person, a trained model for estimating the age of a person may be used. In the second detection process, the fall state is detected as a predetermined action of the person. For fall detection, first, a trained model that estimates the posture of a person is used, and then a trained model that determines whether or not the posture of the person is in a predetermined state, that is, a fall state is used. By limiting the persons who perform the recognition process in this way, it is possible to efficiently monitor even in a facility such as a hospital where an unspecified number of people enter and exit.

By switching the input to an image suitable for each recognition by the procedure as described above, stable recognition can be performed, that is, an image captured by an imaging device at a position where a predetermined person can be imaged. Is input to the second detection process. The images captured by the remaining image pickup devices other than that are input to the face recognition process. This makes it possible to monitor the behavior of a predetermined person while detecting a plurality of persons requiring attention in the widest possible area. In addition to this, by selecting an imaging device suitable for behavior detection instead of all imaging devices capable of detecting the behavior of a specific person, the image input to the behavior detection is suppressed and the processing load applied to the behavior detection. The effect of reducing is obtained.

<Embodiment 2>
In this embodiment, in addition to controlling the input of the image captured by each imaging device to maximize the area included in the image input to the first detection process, the performance of the second detection process is also increased. The case to consider will be described. The same parts as in the first embodiment will be omitted, and only the differences will be described. Thereby, the accuracy of the second detection process can be improved.

FIG. 7 is an example of the image pickup device information table according to the second embodiment. In addition to the first embodiment, the evaluation unit 208 evaluates the correct answer rate of the abnormal behavior monitoring process (second detection process) measured in advance for each output of the imaging device.

For example, the correct answer rate is measured as follows. First, the imaging device of the present embodiment is installed so that it can actually be operated. Next, a person actually performs a predetermined operation that the second detection process accurately detects as an abnormal behavior within the monitoring area of each imaging device. Meanwhile, another person visually checks the presentation unit 205 to see if it is detected as abnormal behavior. This is repeated a predetermined number of times, and the ratio between the abnormal behavior and the number of times detected is calculated.

In S2000 in FIG. 19, the determination unit 202 determines an imaging device that captures an image to be input to the second detection process based on the correct answer rate of the second detection process. From the person detection unit 203, the determination unit 202 obtains information regarding the ID of the imaging device that captured the person requiring attention, the ID of the person requiring attention, the position of the face of the person requiring attention on the image, and the size of the face of the person requiring attention. get. The position of the person requiring attention is estimated from this information and the position / orientation, angle of view, and maximum distance of the image pickup device described in the image pickup device information table, and the image pickup device information table is updated.

Subsequently, the position of the person requiring attention is obtained from the ID and position of the person requiring attention for each imaging device described in the image pickup device information table. In the present embodiment, the values obtained for each imaging device are averaged as the positions of people requiring attention, but the position is not limited to this and may be obtained by another method.

Subsequently, for each imaging device, whether or not the second detection process of the person requiring attention 443 is possible is determined from the monitoring area of each imaging device and the position of the person requiring attention, and the correct answer rate of the abnormal behavior monitoring process is the largest. The image pickup device output is switched to the second detection process.

In S2001 in FIG. 19, the action detection unit 204 determines whether or not a predetermined action of a specific person has been detected. If an abnormal behavior of a person requiring attention is detected, the process proceeds to S307. If the abnormal behavior of the person requiring attention is not detected, it is necessary to redetermine the second imaging device, so the process proceeds to S2002. In addition, regardless of the judgment of the action detection unit, the process may proceed to S2002 at predetermined intervals.

In S2002 in FIG. 19, the evaluation unit 208 evaluates the detection performance of the first detection process or the second detection process for each of the plurality of imaging devices. In the present embodiment, the detection rate of the person requiring attention in the first detection process is evaluated as the detection performance of the second detection process. Images captured by all the image pickup devices including the second image pickup device are input to the first detection process at a predetermined interval (for example, once every 100 frames). The detection performance of the image pickup device that captures an image of a person requiring attention is highly evaluated, and the image pickup device that captures an image of a person requiring attention is evaluated low. If there is an image pickup device that captures an image in which a person requiring attention is detected more often than the current second image pickup device, that is, an image pickup device having high detection performance, the determination unit 202 returns to S2000 to change the second image pickup device. If there is only an imaging device having a lower detection performance than the current second imaging device, that is, if there is no imaging device having a higher detection performance, the process proceeds to S310.

FIG. 7 is an image pickup device information table after the image pickup devices capable of the second detection process of the person requiring attention 443 are 401, 403, and 405, and the image pickup device for the second detection process is determined to be 401. FIG. 8 is an example showing the imaging device and the monitoring area corresponding to FIG. 7. The images taken from the image pickup devices 403, 405, and 407 are subjected to the attention-requiring person detection process. A second detection process is performed on the captured image from the image pickup apparatus 401.

Since the number of image pickup devices that can be installed in a store is limited, the number of people requiring attention may be limited according to the number of image pickup devices. In that case, in S303, when a plurality of persons requiring attention are detected in the first detection process in excess of a predetermined number, the same person whose behavior is detected in the second detection process is selected based on the priority. The determination unit determines a second imaging device that captures an image whose behavior is detected by the second detection process for each person requiring attention, based on the priority of the person requiring attention. Alternatively, the order in which the images of the corresponding person are input to the second detection process may be determined. As the priority of the person requiring attention, the priority set in advance by the user or a predetermined standard is stored in the storage unit 207. For example, a database in which a user assigns a priority flag (large or small, etc.) to a preset list of people requiring attention is used. In addition, from the records so far, among the detected persons requiring attention, the person who has detected abnormal behavior by the second detection process is given higher priority. As a result, the processing load can be reduced even when a large number of people requiring attention are detected. The accuracy of the second detection process can be improved by the procedure as described above.

<Embodiment 3>
The present embodiment describes a case where the input control of the image captured by each imaging device is set to maximize the area where the first detection process is possible, and the performance of the first detection process is also taken into consideration. To do. The same parts as in the first embodiment will be omitted, and only the differences will be described. Thereby, the accuracy of the first detection process can be improved.

FIG. 9 is an example of the image pickup device information table in the third embodiment. The evaluation unit 208 evaluates the accuracy rate of the face recognition process measured in advance for each imaging device.

The correct answer rate of the face recognition process is evaluated as follows. First, the imaging device of the present embodiment is installed so that it can actually be operated. Next, a person actually enters the monitoring area of each imaging device, turns his face in various directions within the range in which the face recognition process operates accurately, and makes various facial expressions. Another person visually checks the presentation unit 205 to confirm the face recognition result. This is repeated a predetermined number of times and a predetermined number of people, and the ratio with the number of times that the face is correctly authenticated is obtained.

In S304 in FIG. 3, the determination unit 202 determines an imaging device that captures an image to be input to the second detection process based on the correct answer rate of the first detection process. From the person detection unit 203, information on the ID of the imaging device that captured the person requiring attention, the ID of the person requiring attention, the position of the face of the person requiring attention on the image, and the size of the face of the person requiring attention is acquired. The determination unit 202 estimates the position of the person requiring attention from the information and the position / orientation, angle of view, and maximum distance of the image pickup device described in the image pickup device information table, and updates the image pickup device information table.

Subsequently, the position of the person requiring attention is obtained from the ID and position of the person requiring attention for each imaging device described in the image pickup device information table. In the present embodiment, the values obtained for each imaging device are averaged as the positions of people requiring attention, but the position is not limited to this and may be obtained by another method.

Subsequently, for each image pickup device, whether or not the second detection process of the person requiring attention 443 is possible is determined from the monitoring area of each image pickup device and the position of the person requiring attention, and the correct answer rate of the face recognition process is minimized from among them. The image captured by the image pickup device is input to the second detection process.

FIG. 9 is an image pickup device information table after the image pickup devices capable of the second detection process of the person requiring attention 443 are 401, 403, and 405, and the image pickup device for the second image detection is determined to be 403.

FIG. 10 is an example showing the imaging device and the monitoring area corresponding to FIG. 9. The first detection process is performed on the captured images from the image pickup devices 401, 405, and 407. A second detection process is performed on the captured image from the image pickup apparatus 403.

The accuracy of the first detection process can be improved by the procedure as described above.

<Embodiment 4>
This embodiment is an extension of the second embodiment, and describes a case where input control of an image captured by each imaging device is performed based on external information regarding brightness that may affect the detection processing performance. The same parts as those in the first and second embodiments will be omitted, and only the differences will be described. As a result, the accuracy of the detection process is stabilized by inputting an image having a brightness suitable for each detection process.

FIG. 11 is a diagram showing an example of the functional configuration of the information processing device 101. The difference from FIG. 2 is that the acquisition unit 210 is added. The acquisition unit 210 acquires external information regarding the brightness of the imaging range for each imaging device. That is, external information is acquired from the external sensor 17 that measures the ambient light in the imaging range for each imaging device. Specifically, the outside world sensor 17 is an illuminance sensor or the like. Then, external information indicating the brightness of the outside world is output to the control unit 206. In the present embodiment, the brightness indicated by the external information is assumed to be in three stages of bright and usually dark. For example, if the imaging range of a certain imaging device is brighter than a predetermined brightness, the accuracy of face recognition may decrease. On the contrary, when the imaging range of a certain imaging device is darker than a predetermined brightness, the accuracy of face recognition and behavior recognition may decrease. Therefore, when the brightness of a certain area satisfies the range of brightness ideal for the detection process, the image pickup apparatus that images the area can capture an image suitable for the detection process.

FIG. 12 is a flowchart showing an example of processing of the information processing device 101. The difference from FIG. 3 is S1200 and S1201. The process of the same step number as in FIG. 3 is the same process as in the first embodiment. Here, the processing with a difference will be described. In S1200, the acquisition unit 201 acquires external information regarding the brightness of the monitoring area for each imaging device. In S1201, the determination unit 202 determines, based on the external information, an image pickup device whose brightness in the monitoring region satisfies a predetermined range as an image pickup device for capturing an image to be input to the second detection process. First, similarly to S304 of the second embodiment, the position information of the person requiring attention is acquired. Further, for each imaging device, it is determined whether or not the second detection process of the person requiring attention 443 is possible based on the monitoring area of each imaging device and the position information of the person requiring attention. The image of the image pickup apparatus having the maximum accuracy rate of the second detection process is input to the second detection process. At this time, the performance of the second detection process is determined by further referring to the external information of the acquisition unit 210. Since the imaging device basically cannot capture an image suitable for each detection process when the illuminance is too high or too low, a range of illuminance suitable for the first or second detection process is set in advance and within that range. An imaging device capable of imaging with illuminance may be determined.

In FIG. 13, when the image pickup devices capable of the second detection process of the person requiring attention 443 are 401, 403, and 405 and the information of the acquisition unit 210 is “dark”, the image pickup device for the second detection process is set to 405. It is an image pickup device information table after the decision.

The external sensor 17 is not limited to the illuminance sensor shown in the present embodiment, but is a device for acquiring information on opening / closing of a light-shielding curtain that indirectly changes ambient light, weather information at a place with a window glass, time information, and the like. It may be. Furthermore, these devices may be connected via the network I / F14.

In the procedure as described above, the input control of the image captured by each imaging device is performed based on the external information regarding the brightness.

<Embodiment 5>
In the present embodiment, the input control of the image captured by each imaging device is set so that the area included in the image input to the first detection process is maximized, and the attribute information of the person (face orientation). ) Will be described for input control for each detection process. The same parts as in the first embodiment will be omitted, and only the differences will be described. For example, since there is a high possibility that the body of a person also faces the front from an image pickup device that is in a position where the face of a person requiring attention can be imaged from the front, the image captured by the image pickup device can be input to the second detection process. The effect of improving the accuracy of the second detection process can be expected.

FIG. 14 is a diagram showing an example of the functional configuration of the information processing device 102. A face orientation detection unit 211 is added to the first embodiment. In the face orientation detection process, for example, face attribute detection (face orientation determination) is performed. Further, the person detection unit 203 outputs the input image to the face orientation detection unit 211 when the person requiring attention is detected. That is, the images input to the first detection process and the face orientation detection process are common.

The face orientation detection unit 211 outputs information indicating the face orientation of the person to the control unit 206 based on the image or the face feature sent from the person detection unit 203. For face orientation determination, a neural network or the like that has learned the orientation of a human face is used.

The determination unit 202 determines which of the person detection unit 203 and the second detection processing unit 204 should input the images for each of the plurality of imaging devices 401 to 407 based on the processing results of the face orientation detection unit 211.

FIG. 15 is a flowchart showing an example of processing of the information processing apparatus 102. Since the processing of the same reference numerals as in FIG. 3 is the same as that in the first embodiment, the description thereof will be omitted. In S1500, the person detection unit 203 determines whether or not the person requiring attention has been detected by the detection process (face recognition processing) of the person requiring attention for the image output from each imaging device. When a person requiring attention is detected as a result of face recognition processing, the determination unit determines information on the ID of the imaging device, the ID of the person requiring attention, the position of the face of the person requiring attention on the image, and the size of the face of the person requiring attention. Output to 202. In addition, the processed image is transmitted to the face orientation detection unit 211. If the person requiring attention is not detected, face recognition processing is continuously performed for all the imaging devices.

In S1501, the face orientation detection unit 211 outputs the face orientation information of the person requiring attention, which is the detection result, to the control unit 206. In S1502, the determination unit 202 determines as an image pickup device that captures an image to be input to the second detection process by an image pickup device that captures an image in which the face of a person is determined to be front by the face orientation detection unit. Similar to S304 of the first embodiment, from the person detection unit 203, the ID of the imaging device that captured the person requiring attention, the ID of the person requiring attention, the position of the face of the person requiring attention on the image, and the size of the face of the person requiring attention. Get information about the face. The position of the person requiring attention is estimated from this information and the position / orientation, angle of view, and maximum distance of the image pickup device described in the image pickup device information table, and the image pickup device information table is updated.

Subsequently, the position of the person requiring attention is obtained from the ID and position of the person requiring attention for each imaging device described in the image pickup device information table. In the present embodiment, the values obtained for each imaging device are averaged as the positions of people requiring attention, but the position is not limited to this and may be obtained by another method.

Subsequently, for each imaging device, it is determined from the monitoring area of each imaging device and the position of the person requiring attention whether the second detection process of the person requiring attention 443 is possible. In addition, the control unit 206 causes the second detection process to input the image captured by the image pickup device having the maximum accuracy rate of the abnormal behavior monitoring process in consideration of the orientation of the person requiring attention.

FIG. 16 is an example of the image pickup device information table according to the fifth embodiment. In addition to the first embodiment, the correct answer rate of the abnormal behavior monitoring process is input in advance for each output of the imaging device. These correct answer rates are the values measured in advance input from the storage unit 207.

Further, the image pickup devices capable of the second detection process of the person requiring attention 443 are 401, 403, and 405, and the state after the image pickup device for the second detection process is determined to 403 is shown. The orientation of the person requiring attention in the captured image of each imaging device is 401 for the front, 403 for the side, and 405 for the side, and these are taken into consideration. The correct answer rate of the second detection process of the person requiring attention of each image pickup device is 65%, 75%, and 60%, respectively, and the output of the 403 image pickup device having the highest correct answer rate of the second detection process is switched under this situation. ..

In the present embodiment, the output of the imaging device to be switched is determined in consideration of "face orientation" as the attribute information of the person, but other factors such as the presence or absence of wearing glasses, masks, and the degree of congestion (number of people) are different. The detection result may be used.

With the procedure as described above, for example, there is a high possibility that the body of the person also faces the front from the imaging device located at a position where the face of the person requiring attention can be imaged from the front. Therefore, it can be expected that the accuracy of the second detection process is improved by inputting the image captured by the image pickup apparatus into the second detection process.

<Embodiment 6>
In this embodiment, in addition to the first embodiment, a case where the internal parameter (exposure) of the imaging device is changed for each output of the imaging device so that the correct answer rate of each detection process is high will be described. By setting the internal parameters of the imaging device that are optimal for the detection process, it is possible to increase the accuracy rate of each detection process. The same parts as in the first embodiment will be omitted, and only the differences will be described.

The control unit 206 enables the exposure to be set for the image pickup apparatus of the input system 201. FIG. 17 is a flowchart showing an example of processing of the information processing apparatus 103. In S1901, the control unit 206 causes all the image pickup devices to set the optimum exposure for the face recognition process described in the image pickup device information table.

The image pickup device parameters are stored in the storage section 207 in advance in the form of an image pickup device information table as shown in FIG. 20, which will be described later, using the evaluation unit 208.

In S1903, the control unit 206 changes the parameter of the image pickup apparatus switched to the second detection process in S303 to the optimum exposure for the second detection process as in S1901.

FIG. 18 is an example of the image pickup device information table according to the sixth embodiment. In addition to the first embodiment, the optimum exposure of the person requiring attention detection processing and the abnormal behavior monitoring processing is input in advance for each output of the imaging device. The image pickup devices capable of detecting the second image of the person requiring attention 443 are 401, 403, and 405, and the image pickup device information table after the image pickup device for the second detection process is determined to be 405. At this time, the exposure of the imaging devices 401, 403, and 407 is set to +3, -1, and -2, which are optimal for the first detection process, respectively, and the exposure of the imaging device 405 is set to -1, which is optimal for the second detection process. ing.

In this embodiment, the shooting parameter to be set is described as "exposure", but the present invention is not limited to this, and is generally used in an imaging device such as frame rate, bit rate (image quality), and white balance. You may use the internal parameters that are available.

With the procedure as described above, it is possible to increase the accuracy rate of each detection process. In addition, by setting the internal parameters of the imaging device that are optimal for the detection process, it is possible to increase the accuracy rate of each detection process.

<Other Embodiments>
The present invention is also realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiment is supplied to the system or device via a network for data communication or various storage media. Then, the computer (or CPU, MPU, etc.) of the system or device reads and executes the program. Further, the program may be recorded and provided on a computer-readable recording medium.

100 Information processing device 202 Decision unit 203 Person detection unit 204 Behavior detection unit 205 Presentation unit 206 Control unit 208 Evaluation unit 207 Storage unit 210 Selection unit

Claims (17)

An information processing device having a person detection unit that detects a person from an image and a behavior detection unit that detects a person's behavior.
Each of the imaging devices that captures an image in which the same person is detected by the person detection unit is the first imaging device that captures an image that detects another person different from the same person by the person detection unit, or the same. A determination means for determining as a second imaging device that captures an image to be input to a behavior detection unit that detects a person's behavior, and
A control means for inputting an image captured by the first imaging device to the person detection unit and controlling the image of the same person captured by the second imaging device to be input to the behavior detection unit. An information processing device characterized by having. The information processing apparatus according to claim 1, wherein the determination means determines at least one or more imaging devices as the first imaging device. The information processing apparatus according to claim 1 or 2, wherein the determination means determines not to input an image captured by an imaging device other than the second imaging device to the behavior detection unit. .. The person detection unit detects a person requiring attention having a feature registered in advance from the image.
The information processing device according to any one of claims 1 to 3, wherein the behavior detection unit detects a sign of a criminal behavior registered in advance for the person requiring attention. The person detection unit detects a person who is older than a predetermined value from the image, and determines the person.
The information processing device according to any one of claims 1 to 3, wherein the behavior detection unit detects a fall of a person who is older than the predetermined value. When the same person cannot be imaged by the second imaging device, the determination means captures an image input to the behavior detection unit from the first imaging device that has not been determined as the second imaging device. The information processing device according to any one of claims 1 to 5, wherein the information processing device is determined as the second image pickup device. Each of the plurality of imaging devices further includes a measuring means for measuring the distance to the same person.
When the same person cannot be imaged by the second image pickup device, the determination means inputs an image pickup device having a distance from the same person within a predetermined range among the plurality of image pickup devices to the behavior detection unit. The information processing apparatus according to claim 6, wherein the information processing apparatus is determined as an imaging apparatus for capturing an image to be processed. Each of the plurality of imaging devices further has an evaluation means for evaluating the detection performance of the person detection unit.
The control means controls so that the images captured by the second imaging device are input to the person detection unit at predetermined intervals.
The determination means is characterized in that an imaging device having a detection performance higher than that of the second imaging device evaluated by the evaluation means is determined as an imaging device that inputs to the behavior detection unit that detects the behavior of the same person. The information processing apparatus according to any one of claims 1 to 7. When the number of people detected by the person detection unit is more than a predetermined number according to the number of the image pickup devices,
The first acquisition means for acquiring the priority to be input to the action detection unit for each person, and
Further, it has a selection means for selecting a person who performs action detection from the person whose action is detected by the action detection unit based on the priority acquired by the first acquisition means.
The determination means according to any one of claims 1 to 8, wherein the determination means determines a second imaging device for capturing an image to be input to the behavior detection unit for each person selected by the selection means. The information processing device described. Any of claims 1 to 9, wherein the determination means is to determine the second image pickup apparatus that satisfies the predetermined condition regarding the size of the area reflected in the image input to the person detection unit. The information processing device according to item 1. Each of the plurality of imaging devices further has a storage means for storing the correct answer rate of the behavior detection unit.
Any of claims 1 to 10, wherein the determination means determines the second imaging device that satisfies the predetermined condition regarding the correct answer rate of the behavior detection unit based on the correct answer rate of the behavior detection unit. The information processing device according to item 1. The storage means stores the correct answer rate of the person detection unit for each of the plurality of imaging devices.
The information according to claim 11, wherein the determination means determines the second imaging device that satisfies the predetermined condition regarding the correct answer rate of the person detection unit based on the correct answer rate of the person detection unit. Processing equipment. Further, it has a second acquisition means for acquiring external information regarding the brightness of the monitoring area for each imaging device.
Any one of claims 1 to 12, wherein the determination means determines the image pickup device whose brightness in the monitoring region satisfies a predetermined range as the second image pickup device based on the external information. The information processing device described in the section. Further having a face orientation detecting means for performing a face orientation detecting process for recognizing the orientation of the face of the same person reflected in the image.
The determination means is characterized in that the image pickup device that has captured the image detected by the face orientation detection means that the face orientation of the same person is the front is determined as the second image pickup device. The information processing apparatus according to any one of 1 to 13. An information processing device having a person detection unit that detects a person from an image and a behavior detection unit that detects a person's behavior.
When a plurality of imaging devices photograph the same person, each of the plurality of imaging devices is used as a classification means for classifying whether to use the same person for detecting a person different from the same person or for detecting the behavior of the same person. ,
An information processing device having a control means for inputting each of the images captured by the image pickup device to either the person detection unit or the behavior detection unit based on the classification result. A program for causing a computer to function as each means included in the information processing apparatus according to any one of claims 1 to 15. An information processing method having a person detection step for detecting a person from an image and a behavior detection step for detecting the behavior of the person.
Each of the image pickup devices that captured the image in which the same person was detected by the person detection step is the first image pickup device that captures an image that detects another person different from the same person by the person detection step, or the same. A determination step for determining as a second imaging device that captures an image to be input to the behavior detection unit that detects the behavior of a person, and
A control step in which an image captured by the first imaging device is input to the person detection step, and an image captured by the same person by the second imaging device is input to the behavior detection step. An information processing method characterized by having.

Images