JP2021106330A - Information processing apparatus, information processing method, and program - Google Patents

Abstract

To enable information in responding to result of detection of an object to be managed as a list in a preferred manner even when an erroneous detection of the object may occur.SOLUTION: An information processing apparatus 100 manages each object detected from within a region of interest as an object list 104 by associating a feature value corresponding to a detection result of the object. The information processing apparatus 100 compares the number of registered objects in the object list 104 with the estimated number of individuals of the objects existing in the region of interest to perform a process for correcting the object list 104 based on the result of the comparison.SELECTED DRAWING: Figure 1

Description

The present disclosure relates to information processing devices, information processing methods, and programs.

In recent years, with the development of various analysis technologies, by detecting a predetermined object such as a person from information such as an image according to the image capture result by an imaging unit such as a surveillance camera, the detection result is so-called behavior analysis or the like. The technology to be applied to various analyzes such as the above is being studied. Patent Document 1 discloses a technique of detecting the same person from an image corresponding to an image captured by each of a plurality of surveillance cameras and analyzing the behavior of the person based on the detection result.

When the detection result of the object is used for various analyzes, the detection result of the object may be managed as a list. In such a list, even if the objects are detected at different moments, if they are determined to be the same object based on the judgment of similarity, etc., they are managed as information of the common object. May be done. By using such a list, it is possible to acquire the behavior history of the detected person (object), and by using the behavior history of a plurality of people, the correlation of the behavior among the plurality of people is possible. It is also possible to analyze the above. Patent Documents 2 and 3 disclose techniques for generating and managing the above list according to the detection result of the object.

Japanese Unexamined Patent Publication No. 2014-78150 Japanese Patent No. 4778532 Japanese Patent No. 4753193

On the other hand, the accuracy of detecting an object is not always high, and some objects may not be detected. Further, for example, it is possible to assume a situation in which the same object detected by different triggers (for example, the same object detected by different detection units) is recognized as a different object. Further, it is highly possible that the detection unit does not detect the object that exists at a position that becomes a blind spot from the detection unit. Under such circumstances, a list of objects whose information is registered in the list and a list of objects that actually exist in the area to be detected (hereinafter, also referred to as "area of interest"). There may be a discrepancy between and. Such dissociation of information can be a factor that reduces the accuracy of analysis using the above list, for example.

In view of the above problems, it is an object of the present invention to make it possible to manage information according to the detection result of the object as a list in a more preferable manner even in a situation where false detection of the object may occur. ..

The information processing apparatus according to the present invention provides first information according to the detection result of the object by each of one or more detection units whose target is at least a part of a predetermined area of interest. The first acquisition means to be acquired, the management means for managing the first information in association with each object as a list, and the first according to the estimation result of the number of individuals of the object existing in the region of interest. The management means includes a second acquisition means for acquiring the information of 2, and the management means compares the number of registered objects in the list with the number of individuals indicated by the second information, and makes the comparison. Based on the result of, the process related to the correction of the list is executed.

According to the present invention, it is possible to manage information according to the detection result of the object as a list in a more preferable manner even in a situation where false detection of the object may occur.

It is a block diagram which showed an example of the functional structure of an information processing apparatus. It is a figure which showed an example of the hardware configuration of an information processing apparatus. It is a figure which showed the application condition of the system which concerns on Example 1. FIG. It is a flowchart which showed an example of the processing of the information processing apparatus which concerns on Example 1. FIG. It is a flowchart which showed an example of the processing of the information processing apparatus which concerns on Example 1. FIG. It is a flowchart which showed an example of the processing of the information processing apparatus which concerns on Example 1. FIG. It is a flowchart which showed an example of the processing of the information processing apparatus which concerns on Example 1. FIG. It is a flowchart which showed an example of the processing of the information processing apparatus which concerns on Example 2. It is a flowchart which showed an example of the processing of the information processing apparatus which concerns on Example 2. It is a flowchart which showed an example of the processing of the information processing apparatus which concerns on Example 2.

Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In the present specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, so that duplicate description will be omitted.

<Detection of objects and use of detection results>
In order to make the features of the information processing device according to the present embodiment easier to understand, first, an outline of an example of a technique for detecting an object such as a person is explained, and then a technical technique for using the detection result of the object is given. Explain the issues.

As an example of a method of acquiring the behavior history of a person who visits a facility or the like, the same person imaged as a subject by a plurality of imaging units (for example, a digital video camera or the like) is extracted from an image according to the imaging result. A technique for identifying a subject based on the characteristics of the subject is known.
Specifically, when a person is detected from an image captured by the imaging unit using a list of people in the facility, information according to the detection result and information on a series of people registered in the list are used. By collating with and, it is possible to identify the detected person. This makes it possible to identify when and where the person detected from the image captured by the imaging unit was based on the position where the imaging unit is installed and the timing at which the image was captured.
In addition, by identifying a person from the captured image across a plurality of imaging units, the person can be identified when, where, and over a longer period of time in a region wider than the imaging range of one imaging unit. It is possible to acquire an action history indicating whether or not the person was there.
In addition, by analyzing the captured image and detecting what kind of behavior the person captured as the subject is taking based on the result of the analysis, it is possible to acquire a more detailed behavior history. ..
By acquiring the behavior history of various persons as described above, based on the behavior history, for example, analysis of the correlation of the behavior of each of a plurality of persons, detection of a person performing an abnormal behavior, etc. Can be done.

As a specific example, by accumulating and using the behavior history of people in theme parks, we analyze what kind of attractions people currently in the theme park tend to visit, and the results of the analysis. It is possible to take congestion mitigation measures in real time using.
In addition, more effective marketing is achieved by accumulating what kind of products a customer picks up across multiple stores in a commercial facility as an action history and analyzing the pattern of customer interest using the action history. It is also possible to take measures.

Here, an outline of an example of a method for creating a list (hereinafter, also referred to as an "object list") used for acquiring a person's action history based on an image corresponding to an image captured by each of a plurality of imaging units will be described. do. The object list for a person existing in the facility can be obtained by, for example, registering the person detected in the facility in the object list and deleting the person away from the facility from the object list. It becomes possible to create and manage.

As a specific example, an object list is created for a person existing in the facility by using an imaging unit that detects a person entering from the entrance / exit of the facility and an imaging unit that detects a person leaving the entrance / exit of the facility. And methods of management have been proposed.
Specifically, the imaging unit that detects a person entering from the entrance / exit adjusts the installation position and the angle of view so that the person entering the facility can be imaged from the front. In addition, the imaging unit that detects a person leaving the entrance / exit adjusts the installation position and angle of view so that the person leaving the facility can be imaged from the front. Based on such a configuration, when a person entering is detected, the person is registered in the object list, and when a person leaving is detected, the person is deleted from the object list. This makes it possible to create and manage an object list of persons existing in the facility.
In the following description, the method of creating and managing the list of objects described above will also be referred to as "Comparative Example 1" for convenience.

Further, as another example, when the detection result of a person from the image captured by the imaging unit is collated with a series of persons registered in the object list and it is determined that the person is different from any of the persons, the person is detected. A method has been proposed in which a new character is registered in the object list as a new character. Further, in this method, when the person registered in the object list is not detected for a certain period of time, it is determined that the person has already left the facility, and the person is deleted from the object list. This makes it possible to create and manage an object list of persons existing in the facility.
In the following description, the method of creating and managing the list of objects described above will also be referred to as "Comparative Example 2" for convenience.

On the other hand, the accuracy of detecting a predetermined object, such as when detecting a person from an image, depends on the ability of the device used for the detection, the environment in which the detection is performed, and the like, and is not necessarily high. It is not always possible to achieve detection accuracy, and some objects may not be detected. Therefore, for example, in the method according to Comparative Example 1, there is a possibility that the detection of some of the persons entering and exiting from the doorway may fail.
Further, in a facility where a relatively large area is secured as an entrance / exit, the entrance / exit is within the angle of view of the imaging unit used for detecting or detecting a person (hereinafter, these may be collectively referred to simply as "detection"). It can be assumed that it is difficult to store. Under such circumstances, for example, a person who has passed through an area that does not fit within the angle of view of the imaging unit is not detected and is excluded from the target of registration in the object list or deletion from the object list. In some cases.
On the other hand, a method of securing a wider angle of view by using a plurality of imaging units may be applied. In this case, for example, the same person may be individually imaged by each of the two or more imaging units, and the person detected in the area where the imaging ranges of the two or more imaging units overlap is the same. It may be determined whether or not the person is a person.
However, in this case, whether or not the object list reflects the actual situation (state in the facility) more accurately depends on the same person detected based on the images corresponding to the imaging results of the different imaging units. It depends on the accuracy of determining whether or not the person is a person. That is, a situation may be assumed in which the same person is erroneously determined as a different person depending on the accuracy of the determination. Under such circumstances, the same person is registered as a different person, the newly appearing person is not registered, the already registered person is registered as a different person, and the person has already left. In some cases, the person may not be deleted.

Also, regarding the method according to Comparative Example 2, whether or not the object list reflects the actual situation (state in the facility) more accurately depends on the detected person and the series of persons registered in the object list. It will depend on the accuracy of the collation between. That is, if an error occurs in the determination in the collation, the person may be erroneously registered in the object list or deleted from the object list in the same manner as in the above-mentioned example for the comparative example 1. There is sex.
Further, in the method according to Comparative Example 2, a person who is located outside the imaging range (blind spot) of the imaging unit or a person who fails to be detected from the image even if the image is captured in the image is listed as an object. It will be excluded from the target of collation with a series of persons registered in. Therefore, even in such a case, there is a possibility that the person is registered in the object list or the person is deleted from the object list by mistake.

Under the above circumstances, a person who exists in the facility but is not registered in the object list over time, a person who does not exist in the facility but is registered in the object list, and an object list. The number of people who have been registered more than once will increase. That is, the divergence between the object list and the actual situation (state in the facility) becomes larger, and the object list is used, for example, acquisition of a person's action history or analysis based on the action history. The accuracy of the analysis may be reduced.
In view of the above circumstances, in the present disclosure, even in a situation where false detection or false detection of an object such as a person may occur, the detection result of the object or the target list according to the detection result is more preferable. We propose an example of a technology that enables management in various aspects. Specifically, in the present disclosure, a technique that makes it possible to further suppress the discrepancy between the list of objects registered in the object list and the list of objects existing in a predetermined area of interest such as a facility. I propose an example.

<Functional configuration>
An example of the functional configuration of the information processing apparatus 100 according to the present embodiment will be described with reference to FIG. The information processing device 100 acquires information according to the detection result of a predetermined object by a predetermined detection unit targeting at least a part of the region of interest with the predetermined region as the region of interest, and obtains the information according to the detection result of the predetermined object. Create and update the list of the above objects (object list) existing in.
In the present disclosure, the object corresponds to an arbitrary object for which various information such as an action history is to be acquired, for example, a person, an animal, a moving body (for example, a vehicle, an airplane, etc.), a robot, or the like. Is included. Further, the area of interest corresponds to an arbitrary area for which various information (for example, behavior history) related to the object is to be acquired, and may correspond to, for example, a commercial facility, a stadium, a residential area, or the like.

The information processing device 100 includes a signal receiving unit 101, a detecting unit 102, an extracting unit 103, an object list 104, a collating unit 105, an aggregation unit 106, a new registration determination unit 107, and a registration deletion determination unit 108. And the population estimation unit 109 and the correction unit 110.

The signal receiving unit 101 receives signals (information) corresponding to the detection results of various states from one or more detection units via a predetermined network. In the present disclosure, the signal received by the signal receiving unit 101 from the detecting unit is not particularly limited as long as it can be used for detecting and identifying an object.

As a specific example, as the signal, an image signal corresponding to an image pickup result by an image pickup unit such as a digital camera may be used. In this case, the imaging unit corresponds to an example of the detection unit. Further, the image signal is not limited to the image signal according to the image pickup result of visible light, and an image signal obtained as an image pickup target of light having a wavelength band different from that of visible light such as infrared light may be used. Further, as another example, an image signal (that is, an image signal including depth information) according to an image pickup result by a so-called three-dimensional camera capable of acquiring depth information (information in the depth direction) may be used.
Further, it is possible to use an acoustic signal according to the sound collection result of acoustics such as voice and environmental sound for detection and identification of an object. As a specific example, it is possible to identify an individual object (for example, an individual in the case of a person) by using a sound collected by a sound collecting device such as a microphone, a walking sound, or the like.
In addition, radio signals transmitted from a communication device carried by each object (for example, a communication terminal held by a person) can be used for detecting and identifying the object.
Further, from the above-mentioned characteristics, the detection unit that is the source of the signal can acquire information according to the surrounding state, such as various sensors and an imaging unit, and the acquired information ( The type of signal) is not particularly limited as long as it can be used for detecting or identifying an object.

When the object list 104 is generated and updated using the signals exemplified above, the signal associated with the object is detected, and which object the signal is associated with is detected. Is collated. Due to these characteristics, the accuracy of the object list 104 generated or updated (for example, the certainty of the association between the signal and the object) depends on the accuracy related to the detection of the object and the result of the detection. It may depend on the accuracy of matching related to the association between the signal and the object. In the technique according to the present embodiment, the accuracy of the object list 104 that is generated or updated is the accuracy related to the detection of the object and the accuracy of the matching related to the association between the signal and the object according to the result of the detection. It is possible to improve the accuracy of the object list 104 in a situation that depends on the above.
In the following description, in order to make the technical features of the information processing apparatus 100 easier to understand, the object is a person and the area of interest is a commercial facility. Further, after the detection unit is an image pickup unit such as a digital camera, the signal reception unit 101 receives an image (in other words, an image signal) according to the image pickup result by the image pickup unit.

The format of the image received by the signal receiving unit 101 may be a moving image or a still image. Further, when the signal receiving unit 101 accepts a moving image, at least a part of the frames of the moving image may be extracted as a still image. In the following description, when the term "image" is used for convenience, both the "moving image" and the "still image" may be included unless otherwise specified.
Further, the image transmitted to the signal receiving unit 101 may be an image directly transmitted from the imaging unit according to the imaging result by the imaging unit, or an image transmitted from a storage device such as a hard disk (stored in the storage device). Image). Further, the image transmitted from the imaging unit and the image transmitted from the storage device may be used in combination.
Further, the number of image pickup units that transmit images to the signal reception unit 101 is not limited to one, and may be a plurality of units. Further, an apparatus in which a plurality of imaging units are integrally configured may be used.
Further, the image transmitted from the imaging unit (for example, a moving image) may be captured indoors or outdoors, and the captured region may change. As a specific example, it may be an image whose imaging area and magnification change, which is captured by a network camera having a pan / tilt / zoom function.
If it is possible to use moving images in which each of a plurality of locations in the region of interest is captured by a plurality of imaging units, the accuracy of the object list 104, which will be described in detail later, will be improved as compared with the case of one imaging unit. It is possible to expect the effect of improving. From such a background, it is more desirable that there are a plurality of imaging units.

Based on the above, in the following description, for convenience, it is assumed that a plurality of imaging units (detection units) perform imaging in the same store, and the inside of the store is set as an area of interest. Further, the signal receiving unit 101 receives a moving image according to the imaging result from each of the plurality of imaging units, and extracts at least a part of the frames of the moving image as a still image. Then, for the still image extracted from the moving image, the signal receiving unit 101 uses information about the imaging unit that is the acquisition source of the moving image and information such as the time corresponding to the still image (frame) as image information. After associating, it shall be output to the detection unit 102.

The detection unit 102 detects a person (object) from a still image (hereinafter, also referred to as “input image”) output from the signal receiving unit 101, and information according to the detection result (hereinafter, also referred to as “detection information”). ) Is associated with the input image and output to the extraction unit 103.
The method is not particularly limited as long as it is possible to detect a person from the image. As a specific example, the person may be detected from the input image by using a recognizer constructed based on machine learning using the image of the person as input data. Further, as another example, the person may be detected from the input image by pattern matching between the visual features such as edges extracted from the input image and the shape of a part such as the head of the person. .. In the following description, by inputting an input image to a recognizer constructed based on machine learning, it is assumed that the recognizer detects a person imaged as a subject in the input image.

The detection information includes at least the above-mentioned image information and information indicating an area in which a person is detected on the input image (hereinafter, also referred to as a "person area"). Further, the detection information may include tracking information.
The tracking information is information used to identify whether or not a person captured as a subject in a continuous still image in chronological order is the same person. In this embodiment, it is assumed that the tracking ID described later is used as the tracking information.
The information indicating the person area may be, for example, coordinates on the image or a mask image indicating the person area, as long as it can be specified at which position on the image the person is imaged. In the following description, it is assumed that the coordinates indicating the points surrounding the person area are used as the information indicating the person area. The coordinates indicating the points surrounding the person area are the coordinates of three or more points forming an area including the person area (hereinafter, also referred to as "cutout area") when a series of points are connected by a line (for example, a straight line). It shall be shown. It is more desirable that the cut-out area is set to include the entire person area, but it does not have to include a part of the parts such as the hands, feet, head, and body.

Further, the detection unit 102 associates the tracking ID with the detection result of the person. When a person is newly detected from the input image, the tracking ID uses a unique value that has not been recorded by the aggregation unit 106, which will be described later. For the input image acquired in chronological order thereafter, the detection unit 102 predicts the position of the person from the past detection result of the person using a Kalman filter or the like, and detects the position in the vicinity of the predicted position. The person may be recognized as the same person as the person to be predicted. In this case, the detection unit 102 may associate the same tracking ID as the person targeted for prediction with the person detected in the vicinity of the predicted position.
If the detection of a person fails, the tracking of the person may be interrupted. In this case, the person newly detected from the input image acquired immediately after may be the same person as the person whose tracking was interrupted immediately before. In such a case, the detection unit 102 matches the image feature of the cutout area and the feature amount described later together with the information of the position where the person is detected, so that the newly detected person is tracked immediately before. It may be specified whether or not the person is the same as the person who is interrupted. In addition, when the newly detected person is the same person as the person whose tracking is interrupted immediately before, the detection unit 102 tracks the newly detected person in the same manner as the person whose tracking is interrupted. The ID may be associated again.

Further, in a situation where the area to be imaged (hereinafter, also referred to as “imaging area”) is congested, tracking associated with each of the plurality of persons is accompanied by the crossing of a plurality of persons in the moving image. The ID may be replaced. Even under such a situation, for example, when the distances between a plurality of people are short, it is possible to prevent the tracking IDs from being replaced based on the matching of the image features and the feature amounts.

In a situation where moving images are input to the signal receiving unit 101 from a plurality of imaging units, it can be assumed that there are overlapping portions between the imaging regions of each of the two or more imaging units. In such a case, the same person is identified between the images corresponding to the two or more imaging units from the position of the person in each of the images corresponding to the imaging results of the two or more imaging units having overlapping imaging regions. It is desirable that the same tracking ID is associated with the person in each image. If a person detected from an image deviates from the range in which the image is captured (hereinafter, also referred to as "imaging range") and re-enters the range (imaging range), it becomes a target. A new tracking ID different from that before the deviation from the imaging range may be associated with the person.

The extraction unit 103 identifies a cutout area in the input image associated with the detection information from the detection information output from the detection unit 102, and information about the object from the cutout area (hereinafter, "object information"). Also called) is extracted. The object information may be the cropped image itself, or may be met by a numerical value or a vector quantity obtained by performing a linear transformation or a non-linear transformation on the cropped image. In the following description, the vector quantity obtained by applying the non-linear transformation to the cut-out image is also referred to as a feature quantity, and the feature quantity is used as the object information. Further, the object information represented by the feature amount corresponds to an example of "first information" according to the detection result of the object (for example, the imaging result of a person).

A recognizer constructed based on machine learning may be used to acquire the above-mentioned vector quantity (feature quantity). In this case, the recognizer has values closer to each other in the vector quantities extracted from each of the plurality of persons having similar appearances, and the vector quantities extracted from each of the plurality of persons having different appearances are more than each other. It is desirable that the learning is done so that it has a distant value.
As a specific example, the above-mentioned recognizer is constructed by constructing a neural network that outputs the above-mentioned vector amount by inputting an image of a person as an input and learning parameters for outputting the above-mentioned vector amount. You may be struck. By using the recognizer constructed in this way, for example, when the persons detected from each of the plurality of images are the same person, it is possible to acquire the feature amounts showing values close to each other. In the case of another person, it is possible to acquire a feature amount showing a value far from each other.

The above is just an example, and the machine learning method used for constructing the recognizer is not particularly limited as long as it is possible to construct the recognizer having the above-mentioned characteristics. Further, the feature amount (vector amount) extracted by the extraction unit 103 may be extracted for each target part (for example, head, torso, arms, legs, etc.).

In addition, the extraction unit 103 may extract or estimate attribute information indicating the characteristics of the detected person. The attribute information includes, for example, the gender, age, height, body shape, body orientation, face orientation, mask presence / absence, glasses presence / absence, sunglasses presence / absence, headgear presence / absence, bag presence / absence, and bag. Includes information that indicates the appearance characteristics of the person, such as type and color of clothing. The method of defining the attribute information is not particularly limited, but for example, the certainty of the estimation result of each attribute information may be indicated by a value in the range of 0 to 1.
As a specific example, for the attribute information indicating the presence or absence of a bag, 0 is set as a value when the target person does not have a bag, and 1 is set as a value when the person has a bag. May be set. In addition, it may be difficult to identify the presence or absence of a bag from the image for which the attribute information is estimated. In such a case, a value between 0 and 1 such as 0.5 may be set depending on the certainty of the presence or absence of the bag.

The method of extracting and estimating the attribute information from the input image is not particularly limited. As a specific example, a recognizer constructed based on machine learning using a neural network, a support vector machine, or the like may be used for extracting and estimating attribute information from an input image. Further, a discriminator related to extraction and estimation of attribute information may be added to a part of the neural network related to the extraction of the vector quantity (feature quantity) described above.
Further, as a method of extracting and estimating the attribute information from the input image, a method other than the method using machine learning may be used. As a specific example, local texture information in the cutout region may be used for extracting or estimating attribute information.

The extraction unit 103 outputs information according to the extraction result of the feature amount to the collation unit 105 in association with the corresponding image information and tracking information.

The collation unit 105 calculates a collation score by collating the feature amount output from the extraction unit 103 with the feature amount of each of a series of persons already registered in the object list 104.
However, if the collation unit 105 can recognize that the feature amount output from the extraction unit 103 matches the feature amount of the person already registered in the object list 104, the feature amount that matches each other It is not necessary to perform the above-mentioned collation because it can be specified that is corresponding to the same person. Further, the collation unit 105 also performs each tracking even when the tracking ID associated with the feature amount output from the extraction unit 103 matches the tracking ID associated with the person already registered in the object list 104. Since the ID corresponds to the same person, the above verification does not have to be performed.
The collation score is a value (for example, a numerical value) for quantifying the degree of similarity between a plurality of feature quantities to be collated. Generally, the score is calculated so that the higher the similarity, the higher the score, but conversely, the higher the similarity, the lower the score. Further, the score may be an integer or a real number. In the following description, the score is calculated so that the higher the similarity, the higher the score, and in order to make the explanation easier to understand, the score is represented by an integer of 0 to 1000. And.

The calculation method of the collation score is not particularly limited as long as it is possible to quantify the similarity between a plurality of features to be collated.
As a specific example, when the feature quantity includes a vector quantity, the similarity of the feature quantity may be calculated based on the Euclidean distance or the Mahalanobis distance between the plurality of vector quantities.
Further, as another example, the distance between the plurality of feature quantities may not be directly used as the collation score, but a value obtained by reversing the positive / negative of the distance or the reciprocal of the distance may be used. This makes it possible to set an index such that the higher the matching score, the higher the similarity.

In the object list 104, a person ID is associated with the registered person, and at least one feature amount is associated with each registered person. As the person ID associated with the person newly registered in the object list 104, the person ID associated with another person already registered in the object list 104 and the aggregation unit 106 described later are aggregated. A unique ID different from the person ID is used.
Further, the above-mentioned tracking ID may be associated with the person registered in the object list 104. As a result, as described above, when the matching unit 105 associates the target person with the same tracking ID as the person registered in the target list 104, the collating unit 105 does not perform collation. It is possible to identify that the person is the same person.
Further, in the object list 104, information on the timing of new registration (for example, time), the number of tracking IDs collated as the same person so far, and the like are associated with each registered person. You may.
In the following description, the person registered in the object list 104 may be simply referred to as a "registered person" for convenience. Further, the process related to the acquisition of the feature amount associated with the person corresponds to an example of the "first acquisition process".

In the object list 104, the person whose registration is determined by the new registration determination unit 107 is newly registered, and the person whose deletion determination is made by the registration deletion determination unit 108 is deleted. By applying such control, ideally, all the persons existing in the area of interest are registered in the object list 104 with a unique person ID associated with them. It will be in the state of being.

The new registration determination unit 107 determines whether or not the person indicated by the feature amount output to the collation unit 105 is a person who newly appears in the region of interest. The method of determining whether or not the person indicated by the feature amount extracted from the input image is a person newly appearing in the region of interest is not particularly limited. Therefore, an example of the determination method will be described below.

First, the first determination method will be described. For example, the new registration determination unit 107 sets a threshold value for determining the collation score (hereinafter, also referred to as “new registration threshold value”), and sets the feature amount output to the collation unit 105 and the feature amount corresponding to the registered person. , Compare the matching score with the threshold. When the collation score for each of the series of registered persons becomes less than the new registration threshold value, the new registration determination unit 107 may register the person indicated by the feature amount output to the collation unit 105 in the object list 104. ..

In this case, if the person A is the same person as the registered person among the two persons A and B detected at a certain time based on the image according to the image pickup result by a certain imaging unit, the person B is , It cannot be a person whose matching matches with person A. Therefore, in this case, as a result of collating the person B with other registered persons other than the registered person whose matching matches with the person A in the series of registered persons, all the matching scores are newly registered. When it becomes less than the threshold value, it may be registered in the object list 104.
Further, based on the same idea as described above, it is possible to determine that a person detected in an area where the imaging areas of the imaging units do not overlap by each of a plurality of imaging units different from each other at a certain time is a different person. .. Even in this case, if it is determined that one person is the same person as the registered person, the other person will be between the other registered person except the registered person whose matching matches with that one person. It suffices if the collation is performed with. Then, when the collation score is less than the new registration threshold value, the other person may be registered in the object list 104.

By applying the above control, for example, even when a plurality of people having similar characteristics and different from each other are detected at the same time, whether or not to register each person as a different person in the object list 104 is determined. It becomes possible to judge. Therefore, it is possible to expect the effect of improving the accuracy of the object list 104.

Next, the second determination method will be described. For example, when the place where the new registration determination unit 107 can enter and exit the area of interest is limited, the new registration determination unit 107 captures the situation of each of the series of entrances and exits with the imaging unit, and based on the imaging result, enters the area of interest. It is possible to register a series of existing persons in the object list 104. Specifically, the new registration determination unit 107 indicates that a person newly detected in any one of a series of entrances and exits (for example, a person whose tracking ID is not registered) is a person who newly appears in the area of interest. It is possible to determine that. Therefore, the new registration determination unit 107 may register the person in the object list 104.

The registration deletion determination unit 108 identifies a registered person who has left the area of interest from a series of registered persons, and deletes the registered person from the object list 104. The method of determining whether or not the registered person has left the area of interest is not particularly limited. Therefore, an example of the determination method will be described below.

First, the first determination method will be described. The registration deletion determination unit 108 records the time when the last detection was performed for each of the series of registered persons, and even if the period between the current time and the time when the last detection was performed is set as the undetected period. good. Then, the registration deletion determination unit 108 determines that the registered person whose undetected period has passed a predetermined threshold value (hereinafter, also referred to as “registration deletion period threshold value”) or more is a person who has left the area of interest, and determines that the object is an object. It may be removed from Listing 104. The time when the registered person was last detected is associated with the feature amount when, for example, the person indicated by the feature amount output to the collation unit 105 is determined to be the same person as the registered person. It is possible to obtain from the image information. That is, the registration deletion determination unit 108 sets the time when the registered person determined to be the same person as the person indicated by the feature amount output to the collation unit 105 is the same as the last detected time from the image information associated with the feature amount. It may be updated at the acquired time.

For example, a collation score can be used to determine whether or not the person indicated by the feature amount output to the collation unit 105 and the registered person are the same person. In this case, for example, the registration deletion determination unit 108 sets a threshold value for determining the collation score (hereinafter, also referred to as “same person determination threshold value”), and the feature amount output to the collation unit 105 and the registered person. The matching score of the feature amount corresponding to is compared with the threshold value. In the registration deletion determination unit 108, when the collation score for some registered persons is equal to or higher than the same person determination threshold value, the person indicated by the feature amount output to the collation unit 105 is the same person as the part of the registered persons. May be determined.
However, if the registration deletion determination unit 108 has a registered person associated with the same tracking ID as the tracking ID associated with the feature amount output to the matching unit 105, those persons regardless of the matching score. May be determined to be the same person. Further, in the registration deletion determination unit 108, when there are a plurality of registered persons whose matching scores are equal to or higher than the same person determination threshold value, the registered person having a higher matching score is the person indicated by the feature amount output to the matching unit 105. It may be determined that the person is the same person as.

Next, the second determination method will be described. For example, when the place where access to the area of interest is possible is limited, the registration deletion determination unit 108 images the situation of each of the series of entrances and exits with the imaging unit, and based on the imaging result, from within the area of interest. It is also possible to identify the person who left. Specifically, the registration deletion determination unit 108 is interested in a registered person having a higher collation score between the feature amount of a person whose exit from a part of the entrance / exit is detected and the feature amount of each of a series of registered persons. The registered person may be recognized as a person who has left the area and the registered person may be deleted from the object list 104.

Although at least one feature amount is associated with each registered person, a plurality of feature amounts may be associated with each other. As a specific example, among the detected persons, the feature amount of the person associated with the same tracking ID as the registered person may be associated with the registered person.
When the feature amount of the detected person is associated with the registered person, the feature amount of the detected person may be associated with the registered person separately from the feature amount previously associated with the registered person.
Further, as another example, the feature amount previously associated with the registered person may be updated by the feature amount of the detected person. As a specific example, when the person is facing the front of the image pickup unit, the feature amount associated with the registered person is given higher priority so that the feature amount of the person extracted from the captured input image is given higher priority. May be updated. In addition, the feature amount of the person extracted from the input image captured in the situation where a part of the person is shielded is given higher priority and is subject to update. May be controlled to be lower.
Further, when the orientation of the body is determined, the feature amount may be retained for each orientation. In this case, the feature amount output to the collating unit 105 and the feature amount having the same body orientation of the target person may be the target of collation by the collating unit 105.
When the feature quantity is represented by a vector quantity, the feature quantity based on the average value or the median value of the vector quantity for each dimension may be the target of collation by the collating unit 105 as a representative feature.
By applying the above control, it is possible to further improve the probability that a person is correctly recognized or identified, and further reduce the probability that a person is mistakenly recognized.

The aggregation unit 106 records the action history for each person identified by the person ID. Specifically, the aggregation unit 106 obtains information related to the person detected from the input image (the person indicated by the feature amount extracted from the input image), which is determined to be the same as the registered person from the collation result. , Record in association with a person ID similar to the registered person.
The information recorded by the aggregation unit 106 is not particularly limited as long as it is information on the behavior of the detected person. As a specific example, the aggregation unit 106 includes information on the time and place where the target person is detected, the imaging unit from which the information is acquired, the behavior estimated from the cut-out image, and the like. Behaviors estimated from the cut-out image include, for example, "looking around", "putting a hand in a bag", "picking up a product", "running", "reaching for another person", and "screaming". And so on. In addition, the tabulation unit 106 may record information such as the line of sight, facial expression, number of people accompanied, and what they have as information related to the above behavior. In addition, the aggregation unit 106 may record the attribute information described above. In addition, the aggregation unit 106 may record the image information and the tracking information in order to associate the detected information with the moving image later.

It should be noted that, among the persons existing in the area of interest, a specific person such as a store staff may be set as not subject to the recording of the action history, and the person may be registered in the object list 104 or information of the information. It may be excluded from the target of aggregation. In this case, an image of a person (for example, a whole body image) that is not subject to the recording of the action history is registered in advance, and a person showing a feature amount having a higher matching score with the feature amount extracted from the image. May be excluded from the targets of registration in the object list 104 and aggregation of information.

It is possible to acquire various action histories of each person according to the information aggregated by the aggregation unit 106. As a specific example, it is possible to recognize the staying time of a person by calculating the difference between the time when the person is newly registered and the time when the person was last detected. In particular, a person who stays for an extremely long time compared to another person can be determined to be a person who has visited for a different purpose than the other person. Therefore, for example, such a person It is also possible to use the characteristics to identify suspicious persons. Further, in this case, a plurality of conditions may be used in combination, such as a combination of a detection result of a prowling behavior imaged by a specific imaging unit a plurality of times and a detection result of a movement of looking around. Therefore, it is possible to further improve the accuracy of identifying a suspicious person.
In addition, as another example, the visit history that can be identified according to which imaging unit detected the person, the interest of the person, such as whether the person stayed in front of which product for a long time, etc. Information about may be aggregated. By using such information, it is possible to perform so-called market analysis such as prediction of the movement route of a newly visited person and prediction of products that the person is likely to purchase. Become.
As described above, the technology that makes it possible to acquire the behavior history of an unspecified number of people can be expected to be used in a wide range of fields such as crime prevention, investigation, marketing, congestion prediction, and city planning. ..

On the other hand, if there is an error in the action history acquired by the aggregation unit 106, it may be difficult for the process using the action history to achieve the original purpose. One of the factors that lower the accuracy of the acquired action history (in other words, the certainty of the action history) is that the accuracy of the object list 104 is lowered.

As a specific example, it is assumed that the person A in the area of interest wears clothes that look different depending on the orientation. In this case, the feature amount extracted from the image captured by the person A in a direction different from the direction registered in the object list 104 is the feature amount of the person A registered in the object list 104. Even if it is collated with, there is a possibility that the collation score will be low and it will be judged as another person. That is, in this case, although the person A is already registered in the object list 104, the detected person A may be newly registered in the object list 104 as another person. .. Under such a situation, the information about the person A detected thereafter (for example, the feature amount) is associated with any one of the plurality of registered persons A, so that the staying time and the number of detections are actually the same. May be recorded in different states. As a specific example, the staying time may be recorded shorter than the actual time, or the number of detections may be recorded less than the actual time. As a result, it may be difficult to obtain accurate information as the action history of the person A.

Such a problem is not limited to the case where the appearance of the person changes depending on the orientation, and may become apparent, for example, when a part of the target person is shielded at the time of detection. Further, as another example, when a person A and a person B having similar clothes are present in the area of interest, if the person A is first registered in the object list 104, the person B is detected thereafter. In that case, there is a high possibility that the collation score between the person B and the person A will be equal to or higher than the new registration threshold. Under such a situation, the behavior history acquired as a result is such that information on the behavior of each of the person A and the person B is mixed, and the staying time and the number of detections are recorded in a state different from the actual one. It may be different. As a specific example, the staying time may be recorded longer than the actual time, or the number of detections may be recorded more than the actual time. The occurrence of such a state can contribute to a decrease in the accuracy of the action history.

In view of the above problems, the information processing apparatus 100 according to the present embodiment estimates the number of individuals (number of persons) in the region of interest, and the number of registered persons in the object list 104 (number of registered persons). And, based on the comparison result, the object list 104 is corrected. By providing such a mechanism, the information processing apparatus 100 according to the present embodiment further improves the accuracy of the object list 104. Hereinafter, the components of the information processing apparatus 100 for realizing the above mechanism will be described in more detail.

The population estimation unit 109 is a component that estimates the number of individuals in the above-mentioned region of interest. The method is not particularly limited as long as it is possible to estimate the number of individuals in the region of interest.

For example, the population estimation unit 109 may estimate the number of people (individuals) in the region of interest based on the number of people captured as subjects in the image in the region of interest. In addition, the population estimation unit 109 may use a recognizer constructed based on machine learning using an image captured by a person as input data for estimating the number of individuals using the image. As described above, when an image is used for population estimation, it is possible to improve the accuracy of population estimation according to the imaging conditions.
If it is difficult to fit the entire region of interest within the imaging range of one imaging unit, a plurality of imaging units may be used.
As described above, with one or more imaging units arranged so that each portion of the region of interest falls within the imaging range of at least a part of the imaging unit, based on each image according to the imaging result by each imaging unit. By estimating the number of people, it is possible to estimate the number of people in the area of interest.
In this case, as the imaging unit used for image acquisition, an imaging unit that transmits a moving image to the signal receiving unit 101 may be used, or an imaging unit different from the imaging unit may be used. It may be used.

In addition, as another method for estimating the number of people (individual number), for example, information such as the number of people passing through a predetermined area such as an entrance / exit gate, the number of tickets sold, and past statistics on the number of people in the area of interest can be obtained. There is a method to use.
In addition, the population estimation unit 109 may estimate the number of people (number of individuals) in a part of the region of interest, and estimate the number of people in the entire region of interest based on the estimation result.

The persons registered in the object list 104 do not necessarily have to be all persons existing in the area of interest. For example, even when some of the persons existing in the area of interest are registered in the object list 104, if it is possible to correctly acquire the action history of the part of the persons, the said one. It can be useful information for analyzing the behavioral tendency of a person in a department. Even in such a case, the actual situation is further improved by correcting the object list 104 based on the comparison result between the estimation result of the number of people in the area of interest and the number of registered persons in the object list 104. It can be expected to have the effect of making it possible to acquire an action history that matches the above.
In the following description, for convenience, the estimation result of the number of people in the area of interest may be simply referred to as the "estimated number of people", and the number of registered persons in the object list 104 may be simply referred to as the "registered number". be.
For example, in a situation where there are a plurality of people wearing the same uniform in the area of interest, a plurality of different people wearing the same uniform are recognized as the same person and are registered in the object list 104. The number of people who are actually registered may be less than the number of people who should actually be registered. Even under such a situation, the number of registrations can be made closer to the number of persons to be actually registered by correcting the object list 104 by using the estimated number of people. In the above example, the number of persons to be actually registered in the object list 104 may be obtained from, for example, statistics of the estimated number of people in the past and the number of registrations.
The information indicating the estimated number of people corresponds to an example of "second information". The type of the second information is not particularly limited as long as it can directly or indirectly indicate the estimated number of people.

For the correction of the object list 104, for example, these differences may be used as an example of the comparison result between the estimated number of people and the registered number. In this case, for example, when the number of registrations is larger than the estimated number of people, the object list 104 may be corrected so that the number of registrations is smaller. When the number of registrations is smaller than the estimated number of people, the object list 104 may be corrected so that the number of registrations is larger. The details of the correction method of the object list 104 will be described later.

The above comparison is based on the premise that the accuracy of estimating the number of people in the region of interest is relatively high. On the other hand, if the accuracy related to the estimation of the number of people in the area of interest is not necessarily high, the error related to the estimation may become apparent on the object list 104 side, and the accuracy related to the acquisition of the action history may decrease. There is. In such a case, if the difference between the number of registrations and the estimated number of people exceeds a predetermined margin, the object list 104 may be corrected.
As a specific example, in a situation where an error of about 10% occurs in the estimation of the number of people in the region of interest, approximately 10% of the estimated number of people may be the above-mentioned predetermined margin.
By applying the above control, it is possible to expect the effect of stabilizing the accuracy related to the acquisition of the action history even in a situation where an error occurs in the estimation of the number of people in the area of interest. ..

Further, the area for acquiring the input image used for registering the person in the object list 104 (the area imaged by the imaging unit) and the area for estimating the number of people are only a part of the area of interest. A situation that covers the above can be assumed. In such a case, for example, after multiplying at least one of the registered number and the estimated number of people by a predetermined ratio (coefficient), the comparison between the registered number and the estimated number of people (for example, acquisition of the difference) ) May be performed. The predetermined ratio may be set based on, for example, the ratio of the region to which the input image is to be acquired and the region to which the number of people is estimated to be the region of interest.

Further, in this case, since the number of registrations varies, the margin for determining that the number of registrations is abnormal is larger than the accuracy of estimating the number of people in the area of interest. It is desirable to secure a large amount.
As a specific example, a method of estimating the number of persons to be registered in the object list 104 from the estimated number of people acquired in the past and the number of registrations may be applied. In this case, for example, it is preferable to sequentially record the estimated number of people and the number of registrations, and perform regression analysis using the estimated number of people as an explanatory variable and the number of registrations as an objective variable. Then, the predicted value and standard deviation of the number of registrations are calculated for an arbitrary estimated number of people, and when the deviation from the predicted value is a predetermined multiple (for example, 3 times) or more of the standard deviation, it is treated as an abnormal value. May be good.
The above is an example of a method for comparing the number of people, and the method for comparing the number of people is not necessarily limited to the above as long as the number of people to be registered in the object list 104 is determined based on the estimated number of people.

In addition, the timing at which the number of registrations and the estimated number of people are compared is not particularly limited.
For example, the number of registrations and the estimated number of people may be compared periodically at predetermined periods. Further, as another example, the number of registrations and the estimated number of people may be compared at a timing based on a predetermined process. As a specific example, the number of registered persons may be compared with the estimated number of people immediately before the person is registered in the object list 104 or the person is deleted from the object list 104. In this case, for example, when the comparison between the registered number and the estimated number of people is correct, the person may be registered in the object list 104 or the person may be deleted from the object list 104.

The correction unit 110 acquires the comparison result between the registered number and the estimated number of people, and executes the process related to the correction of the object list 104 according to the comparison result. As a specific example, when the number of registrations is larger than the estimated number of people and the difference is equal to or greater than the threshold value, the correction unit 110 executes a process related to correction of the object list 104 so that the number of registrations is smaller. You may. Further, when the number of registrations is smaller than the estimated number of people and the difference is equal to or greater than the threshold value, the correction unit 110 may execute the process related to the correction of the object list 104 so that the number of registrations is larger.
The content of the processing related to the correction of the object list 104 is not particularly limited as long as it is the processing related to the correction that brings the number of registrations closer to the estimated number of people. As a specific example, an imaging unit connected to the signal receiving unit 101 may be made to search for a person, and the result of the search may be used for correction of the object list 104. Further, as another example, the object list 104 may be indirectly corrected by controlling the conditions related to the detection of the person (object) in the detection unit 102. Further, as another example, the object list 104 may be indirectly corrected by controlling the determination conditions related to the registration of the person in the object list 104 by the new registration determination unit 107. Similarly, the object list 104 may be indirectly corrected by controlling the determination conditions related to the deletion of the person from the object list 104 by the registration deletion determination unit 108. The details of the processing related to the correction of the object list 104 described above will be described below.
Of the processes of the correction unit 110, the process related to the acquisition of information on the estimated number of people (in other words, the second information) corresponds to an example of the "second acquisition process". Further, the process related to the correction of the object list 104 corresponds to an example of the "management process" related to the management of the object list 104.

If the number of registrations is larger than the estimated number of people, the number of registrations can be determined by the estimated number of people by deleting the person registered in the object list 104 or suppressing the new registration of the person in the object list 104. It is possible to correct the object list 104 so as to bring it closer. At this time, an active means and a passive means are used for the method of deleting the person registered in the object list 104 and the method of suppressing the new registration of the person in the object list 104, respectively. It is possible to selectively apply various means.

As a person who is desirable to be deleted from the object list 104, for example, a person who does not exist in the area of interest or a person who is originally the same person but is registered as another person in the object list 104. Can be the person who is.

As a passive means for deleting a person from the object list 104, the determination condition relating to the deletion of the person from the object list 104 by the registration deletion determination unit 108 is further relaxed (that is, the deletion is more promoted). ) Means can be mentioned.
As a specific example, the registration deletion determination unit 108 corresponds to a shorter period of the registration deletion period threshold value used for comparison with the undetected period in order to determine the registered person as the deletion target from the object list 104. It may be a threshold value. This makes it possible to control the person who has already left the area of interest to be deleted from the object list 104 more quickly.

In addition, as an active means of deleting a person from the object list 104, it is said that the user has already exited by setting the registration deletion period threshold value to a threshold value corresponding to a shorter period and applying the threshold value retroactively. Examples include a means of searching for and deleting a determinable registered person. At this time, the period for retroactively applying the updated threshold value may be set to be after the timing when the number of registrations and the estimated number of people are compared immediately before. This is because it is known that the number of registrations is correct by comparing with the estimated number of people at the timing when the number of people was compared immediately before, and by applying new conditions after this timing, the number of registrations will be the area of interest. It becomes possible to get closer to the number of existing people. Further, since the range in which the new condition (threshold value after update) is applied retroactively is limited, it is possible to minimize the processing cost related to the deletion of the person from the object list 104.

As another example of an active means of removing a person from the object list 104, when it is determined that the number of registrations is larger than the estimated number of people, the person in the area of interest is scanned and duplicated in the object list 104. There is a means to identify and delete a person who has already been registered or a person who has already left. In this case, for example, the PTZ (Pan Til Zoom) function of the imaging unit is used to scan the area of interest and collate the person existing in the area of interest with the registered person to match the object. The person to be deleted from the list 104 may be specified.

Suppression of new registration of a person in the object list 104 is effective, for example, when the object list 104 is corrected so that the number of registrations is closer to the estimated number of people in a situation where the area of interest is suddenly congested. be. Specifically, when the area of interest is congested, the person captured as the subject in the input image is more likely to be partially shielded by another person. Under such circumstances, even if the person detected from the input image is the same person as the registered person, the appearance may be different from that of the registered person, which lowers the matching score, resulting in registration. There is a high possibility that the person will be registered as a different person from the person. In this way, in order to suppress the occurrence of a situation in which the person is originally the same person as the registered person but is registered in the object list 104 as another person, the person is newly registered in the object list 104. Such conditions should be restricted (stricter).
The condition relating to the new registration of a person in the object list 104 corresponds to an example of the "first condition".

As a passive means of suppressing new registration of a person in the object list 104, the determination condition relating to the registration of the person in the object list 104 by the new registration determination unit 107 is further restricted (that is, it is more difficult to be registered). ) Means can be mentioned.
Examples of the condition for newly registering a person in the object list 104 include a condition for detecting a person by the detection unit 102, a determination condition for registering a person in the object list 104 by the new registration determination unit 107, and the like.

As a method of limiting (stricting) the conditions related to the detection of a person by the detection unit 102, for example, when the detection unit 102 detects a person, the image of the person is determined by another person or object. A method of utilizing the ratio of the shielded area can be mentioned. Specifically, the judgment conditions related to the registration of a person in the object list 104 are further restricted by excluding the person whose ratio of the shielded area is equal to or more than the threshold value from the target of collation with the registered person. It becomes possible to do.

As one method of limiting (stricting) the judgment conditions related to registration in the object list 104 by the new registration judgment unit 107, a collation score is used to determine the detected person as a registration target in the object list 104. There is a method of controlling the new registration threshold used for comparison with. Specifically, by setting the new registration threshold value to be lower, it is possible to control that only a person having a lower collation score is registered in the object list 104.

In addition, as another method, there is a method of using a combination of face recognition technology and whole body verification technology.
In general, face recognition technology tends to have a higher ability to identify a person than whole-body verification technology. On the other hand, the face recognition technology preferably uses a face image in which the person faces the front as input, and the number of detection results that can be used for matching tends to be more limited than that of the whole body matching technique. .. On the other hand, the whole body verification technique tends to have a lower ability to identify a person than the face recognition technique. On the other hand, the whole-body matching technology can also use an image in which a person faces sideways or backwards as an input, so the number of detection results that can be used for matching tends to be larger than that of the face recognition technology. be.
In view of the above characteristics, when a determination regarding the registration of a person in the object list 104 is usually performed by using the whole body collation technique and the new registration of the person in the object list 104 is suppressed, the case is performed. Controls may be applied such that the determination of registration is performed using face recognition technology. As a result, by switching the determination related to the registration of the person in the object list 104 to the determination of registration using the face recognition technology, the object is compared with the case where the determination of registration using the whole body verification technology is performed. It is possible to tighten the conditions for newly registering a person in the list 104.

As an active means for suppressing new registration of a person in the object list 104, a method of retroactively applying the changed registration conditions and the like can be mentioned as in the above-mentioned means for deleting a person from the object list 104. ..

Further, when the number of registrations is smaller than the estimated number of people, the number of registrations is estimated by newly registering a person in the object list 104 or suppressing the deletion of the person registered in the object list 104. It is possible to correct the object list 104 so as to get closer to the number of people. At this time, active means and passive means for each of the method of newly registering a person in the object list 104 and the method of suppressing the deletion of the person registered in the object list 104. Means and can be selectively applied.

As a person who is desirable to be newly registered in the object list 104, a person who is not registered in the object list 104 but exists in the area of interest may be applicable. Such a person is included in a person who is not detected by the detection unit 102, a person whose matching score with the registered person is not less than the registration determination threshold value (for example, a person whose clothes are similar to the registered person), and an area of interest. It is roughly classified into persons who exist but are deleted from the object list 104.

As a passive means for newly registering a person in the object list 104, there is a means for relaxing the conditions for registering the person in the object list 104 (that is, making the registration more promoted). .. As described above, the determination conditions for registering a person in the object list 104 include, for example, the condition for detecting a person by the detection unit 102 and the registration of the person in the object list 104 by the new registration determination unit 107. Judgment conditions related to

Examples of the means for relaxing the condition for detecting a person by the detection unit 102 include a means for relaxing the threshold value determination for detecting a person and a means for using another detection means in combination. As a means for using another detection means in combination, for example, when a person is detected, a means for detecting both feet instead of detecting the head shape from the image can be mentioned.
The means for detecting both feet tends to have lower accuracy in detecting a person than the means for detecting the head, but the heads of many people are shielded by an umbrella or the like as in an outdoor image in rainy weather. Under such circumstances, the accuracy of detecting a person may be higher than that of the means for detecting the head. Therefore, when the area of interest is outdoors, if the number of registrations decreases significantly even though the estimated number of people has not changed significantly, it is assumed that it is raining and the head of the person is shielded by an umbrella or the like. , A person may be detected by using both foot detection and head shape detection in combination.
As a means for relaxing the determination condition relating to the registration of the person in the object list 104 by the new registration determination unit 107, a means for raising the new registration threshold can be mentioned.

As an active means for newly registering a person in the object list 104, after relaxing the conditions for registering the person in the object list 104, new conditions retroactively (conditions after relaxation). There are means to apply. Further, in order to find a person who is not registered in the object list 104, a means for controlling the imaging range of the imaging unit may be applied.

As a passive means for suppressing the deletion of a person from the object list 104, there is a means for limiting (stricting) the conditions for deleting the person from the object list 104. As a specific example, by controlling the registration deletion period threshold to be longer, it is possible to suppress the occurrence of a situation in which a person who exists in the region of interest but stays in the blind spot of the imaging unit for a long time is deleted. It becomes possible to do.
Further, as an active means for suppressing the deletion of the person from the object list 104, for example, a means for retroactively applying the restricted (stricted) registration deletion period threshold value can be mentioned. Further, in order to find a person who has been deleted from the object list 104 but exists in the region of interest, means for controlling the imaging range of the imaging unit may be applied.

By applying the above means, even if a person is deleted from the object list 104, if it is confirmed that the person exists in the area of interest, the person is registered in the object list 104 again. It is possible to make it in the state of being done. note that. When a person deleted from the object list 104 is to be registered again, the person ID associated with the person before the deletion is used to further improve the accuracy related to the acquisition of the action history. Is possible.

Further, any of a person who is not detected by the detection unit 102, a person whose matching score with the registered person does not fall below the registration determination threshold value, and a person who exists in the region of interest but is deleted from the object list 104. It is more desirable to be able to identify whether it corresponds to. By being able to identify which of these persons corresponds to, it is more likely that the object list 104 can be maintained in a state closer to the actual state.
For example, when the area of interest is outdoors and the number of people detected decreases due to an increase in the number of people holding an umbrella due to rain, it is desirable to relax the conditions for detecting people. Further, when the number of persons with similar clothes such as a group wearing uniforms increases, it is desirable to relax the determination conditions for registering the persons in the object list 104.

As a method of estimating the cause of the discrepancy between the number of registered persons and the actual number of persons, for example, the number of detected persons per hour, the number of registered persons, and the number of deleted registered persons are used. And the method of using the estimated number of people together. Specifically, after accumulating data on the number of detected persons, the number of registered persons, and the number of deleted registered persons per hour, the estimated number of persons is used as an explanatory variable, and the detected number, the registered number, and the deletion are performed. Regression analysis that predicts numbers as objective variables may be performed. Then, when there is a discrepancy between the number of registered persons and the estimated number of people, the above objective variable is predicted from the estimated number of people, and the item closer to the abnormal value is given a larger weight to the person detection condition, the person registration condition, and the person registration condition. The condition for deleting a person may be corrected.

Items that are close to outliers with respect to past statistics are likely to be the cause of the divergence in the number of people, so by applying the above-mentioned controls, it is appropriate to properly change how various conditions should be changed. It is possible to determine and improve the accuracy of the object list 104.

<Hardware configuration>
An example of the hardware configuration of the information processing apparatus 100 according to the present embodiment will be described with reference to FIG. As shown in FIG. 2, the information processing apparatus 100 according to the present embodiment includes a CPU (Central Processing Unit) 211, a ROM (Read Only Memory) 212, and a RAM (Random Access Memory) 213. Further, the information processing device 100 includes an auxiliary storage device 214, an output device 215, an input device 216, and a communication I / F 217. The CPU 211, ROM 212, RAM 213, auxiliary storage device 214, output device 215, input device 216, and communication I / F 217 are connected to each other via a bus 218.

The CPU 211 is a central processing unit that controls various operations of the information processing device 100. For example, the CPU 211 may control the operation of the entire information processing device 100. The ROM 212 stores a control program, a boot program, and the like that can be executed by the CPU 211. The RAM 213 is the main storage memory of the CPU 211, and is used as a work area or a temporary storage area for developing various programs.

The auxiliary storage device 214 stores various data and various programs. The auxiliary storage device 214 is realized by a storage device that can temporarily or continuously store various data such as an HDD (Hard Disk Drive) and a non-volatile memory represented by an SSD (Solid State Drive). ..

The output device 215 is a device that outputs various information, and is used for presenting various information to the user (for example, notification of notification information). In the present embodiment, the output device 215 is realized by a display device such as a display. The output device 215 presents the information to the user by displaying various display information. However, as another example, the output device 215 may be realized by an acoustic output device that outputs sounds such as voice and electronic sounds. In this case, the output device 215 presents information to the user by outputting sound such as voice or electronic sound. Further, the device applied as the output device 215 may be appropriately changed depending on the medium used for presenting information to the user. The output device 215 corresponds to an example of the “output unit”.

The input device 216 is used for receiving various instructions from the user. In the present embodiment, the input device 216 includes an input device such as a mouse, a keyboard, and a touch panel. However, as another example, the input device 216 may include a sound collecting device such as a microphone and collect sound spoken by the user. In this case, the collected voice is subjected to various analysis processes such as acoustic analysis and natural language processing, so that the content indicated by the voice is recognized as an instruction from the user. Further, the device applied as the input device 216 may be appropriately changed depending on the method of recognizing the instruction from the user. Further, a plurality of types of devices may be applied as the input device 216.

The communication I / F 217 is used for communication with an external device via a network. The device applied as the communication I / F 217 may be appropriately changed according to the type of the communication path and the applicable communication method.

The CPU 211 expands the program stored in the ROM 212 or the auxiliary storage device 214 into the RAM 213, and by executing this program, the functional configuration of the information processing device 100 shown in FIG. 1 and the functional configurations shown in FIGS. 3 to 10 are shown. The processing shown in the flowchart is realized.

<Example>
Hereinafter, specific examples for actually realizing the above-described embodiment will be described.

(Example 1)
In the first embodiment, an example of a mechanism for detecting a suspicious person in a store will be described based on the technique according to the present embodiment. Unless otherwise specified, the configuration of the system or device for realizing the first embodiment shall follow the above-described configuration as an embodiment.

FIG. 3 is a diagram showing application conditions of the system according to the first embodiment, and shows a schematic bird's-eye view of the inside of the store to be detected by a suspicious person. In the example shown in FIG. 3, the store has one entrance / exit 311 and a product shelf 313 and a cash register counter 312 are placed on the store. There is one staff member 314 in the store, and the staff member 314 shall perform accounting and patrol in the store.
The signal receiving unit 101 of the information processing device 100 according to the present embodiment acquires moving images according to the imaging results from each of the plurality of imaging units 301a to 301d (for example, a network camera) arranged in the store. Each of the imaging areas 301a to 301d mainly covers an area that becomes a blind spot when the staff 314 is behind the cashier counter 312. Further, although the imaging regions of some of the imaging units 301a to 301d overlap, it is a base which region of the image corresponding to the imaging result of each imaging unit overlaps with the imaging region of which imaging unit. And. Therefore, for example, even if one person is detected by a plurality of imaging units at the same time, it is determined that the person is the same person from the detected position, and the same tracking ID is assigned.

The totaling unit 106 records the number of times each person in the store appears in the imaging area of the imaging unit and the staying time, and the person who stays longer than usual (for example, a general customer) or the person who is wandering around. Is identified as a disbelief and the staff 314 is notified. Further, the imaging unit 315 (for example, a network camera) is arranged so that the entire floor in the store fits within the imaging area. A moving image according to the image pickup result by the image pickup unit 315 is used for estimating the number of people in the store.

In this embodiment, each function of the information processing apparatus 100 is realized by, for example, software running on a server in a store. In other words, the information processing device 100 according to the present embodiment is realized by the server. The imaging units 301a to 301d and 315 are connected to the server via, for example, a network cable or a network hub. The server acquires moving images according to the imaging results from the imaging units 301a to 301d and 315, respectively, and executes each function of the information processing apparatus 100 according to the present embodiment. Further, when the server (information processing apparatus 100) notifies the staff 314 of information, the server (information processing device 100) causes the communication terminal carried by the staff 314 to display the notification screen.

The detection unit 102 detects a person existing in the store (in the area of interest). Specifically, the detection unit 102 creates a still image (personal image) in which a person area is cut out from a moving image corresponding to each of the imaging results of the imaging units 301a to 301d, and image information and tracking of the person image. After associating the information, the person image is output to the extraction unit 103. In this embodiment, the image information includes identification information (camera ID) for identifying which image pickup unit of the image pickup units 301a to 301d has captured the image, and information indicating the time when the image pickup was performed. ,including. The tracking information includes identification information (detection ID) for identifying each detection result and tracking ID.

The extraction unit 103 extracts the feature amount from the person image output from the detection unit 102. In this embodiment, the extracted features are assumed to be 1000-dimensional normalized vectors. The feature amount extraction is performed based on machine learning, for example, so that the L2 distance of the feature amount becomes shorter for people who are similar to each other, and the L2 distance of the feature amount becomes longer for people who are not similar to each other. An extractor shall be used. The extraction unit 103 outputs the extracted feature amount and the image information and tracking information associated with the person image that is the extraction source of the feature amount to the collation unit 105.

The collation unit 105 collates between the feature amount output from the extraction unit 103 and the feature amount of each of the series of persons registered in the object list 104. As a specific example, the collation unit 105 takes the reciprocal of the L2 distance of each feature amount to be collated and multiplies it by 1000 to convert it into an integer.

(New person registration process)
Here, with reference to FIG. 4, as an example of the processing of the information processing apparatus 100 according to the first embodiment, in particular, an example of the processing related to the new registration of a person for the object list 104 will be described.
In the processing of S101, the feature amount output from the extraction unit 103 (feature amount extracted from the person image) by the collation unit 105 described above and the feature of each of the series of persons registered in the object list 104. Corresponds to the process related to collation between the quantity and.

In S102, the collation unit 105 determines whether or not the collation score of the feature amount extracted from the person image and the feature amount of each of the series of persons registered in the object list 104 is less than the new registration threshold value. judge.
When the collation unit 105 determines in S102 that the collation score of the feature amount extracted from the person image and the feature amount of each of the series of persons registered in the object list 104 is less than the new registration threshold value. Is determined that the target person is unregistered. In this case, the collating unit 105 advances the process to S103. In S103, the new registration determination unit 107 newly registers the target person (that is, the person indicated by the feature amount extracted from the person image) in the object list 104. At this time, the new registration determination unit 107 associates the registration time (for example, the detection time) and the feature amount of the target person with the newly issued person ID, and then the new registration determination unit 107 of the target person with respect to the object list 104. Register.
In S104, the new registration determination unit 107 associates the corresponding image information, the tracking information, and the feature amount with the person ID of the person newly registered in the object list 104, and then aggregates this series of information. Output to 106. The aggregation unit 106 records information about the person indicated by the person ID, including information about the detection time and the imaging unit of the detection source for each person ID. Then, the totaling unit 106 ends a series of processes shown in FIG.

On the other hand, in S102, the collation unit 105 sets the collation score of the feature amount extracted from the person image and the feature amount of each of the series of persons registered in the object list 104 to be equal to or higher than the new registration threshold value. If it is determined, the series of processes shown in FIG. 4 is terminated.

(Processing when a registered person is detected)
Next, with reference to FIG. 5, as another example of the processing of the information processing apparatus 100 according to the first embodiment, in particular, an example of processing when the same person as the person registered in the object list 104 is detected. Will be described.
In the processing of S201, the feature amount output from the extraction unit 103 (feature amount extracted from the person image) by the collation unit 105 described above and the feature of each of the series of persons registered in the object list 104. Corresponds to the process related to collation between the quantity and.

In S202, the collation unit 105 determines whether or not a person whose collation score with the feature amount extracted from the person image is equal to or higher than the same person determination threshold value is registered in the object list 104.
When the collation unit 105 determines in S202 that a person whose collation score with the feature amount extracted from the person image is equal to or higher than the same person determination threshold value is registered in the object list 104, the collation score is higher. It is determined that the highly registered person is the same person as the target person. In this case, the collating unit 105 advances the process to S203. In S203, the aggregation unit 106 updates the detection time of the target person (that is, the person indicated by the feature amount extracted from the person image) with a new detection time.
In S204, the aggregation unit 106 records (for example, additional writing) information about the updated detection time and the detection source imaging unit with respect to the information about the person indicated by the person ID corresponding to the target person. )do.

On the other hand, when the collation unit 105 determines in S202 that a person whose collation score with the feature amount extracted from the person image is equal to or higher than the same person determination threshold value is not registered in the object list 104, FIG. The series of processes shown in 5 is completed.

(Delete registered person)
Next, with reference to FIG. 6, as another example of the processing of the information processing apparatus 100 according to the first embodiment, in particular, an example of the processing related to the deletion of the registered person from the object list 104 will be described.

In S301, the registration deletion determination unit 108 calculates the undetected period for each of the series of persons registered in the object list 104 based on the elapsed period from the time when the last detection was performed.
In S302, the registration deletion determination unit 108 determines whether or not there is a registered person whose undetected period is equal to or greater than the registration deletion period threshold value.
When the registration deletion determination unit 108 determines in S302 that there is a registered person whose undetected period is equal to or greater than the registration deletion period threshold value, the process proceeds to S303. In S303, the registration deletion determination unit 108 deletes the target registered person from the object list 104, and then proceeds to the process in S304.

On the other hand, when the registration deletion determination unit 108 determines in S302 that there is no registered person whose undetected period is equal to or greater than the registration deletion period threshold value, the process proceeds to S304. In this case, the process of S303 is not executed.

In S304, the registration deletion determination unit 108 determines whether or not to end a series of processes.
When the registration deletion determination unit 108 determines in S304 that the series of processes is not completed, the process proceeds to S305. In S305, the registration deletion determination unit 108 waits for a predetermined period of time, and then re-executes the processes of S301 to S304. The above-mentioned predetermined period may be arbitrary, but in this embodiment, it is assumed to be "10 seconds" for convenience.
In this way, the object list 104 is periodically reviewed, and the person who has left the store is deleted from the object list 104.

Then, when the registration deletion determination unit 108 determines in S304 that the series of processes is to be completed, the registration deletion determination unit 108 ends the series of processes shown in FIG.

(Correction processing of object list)
Next, with reference to FIG. 7, as another example of the processing of the information processing apparatus 100 according to the first embodiment, in particular, an example of the processing related to the correction of the object list 104 will be described.

In S401, the population estimation unit 109 estimates the number of people in the store (in other words, the number of individuals in the region of interest) based on the moving image according to the imaging result of the imaging unit 315. Then, the correction unit 110 compares the estimation result (estimated number of people) of the number of people in the store with the number of people (registered number) registered in the object list 104.
In S402, the correction unit 110 determines whether or not the number of registrations is more than a predetermined ratio than the estimated number of people. The predetermined ratio may be set according to the accuracy of estimating the number of people in the store. In this embodiment, it is set to "10%" for convenience.
When the correction unit 110 determines in S402 that the number of registrations is more than a predetermined ratio than the estimated number of people, the correction unit 110 proceeds to S403. In S403, the correction unit 110 executes at least one of restriction (stricter) of the condition for deleting the registered person and relaxation of the condition for registering the person in the object list 104. As a specific example, the correction unit 110 sets the registration deletion period threshold value to a threshold value corresponding to a shorter period, and lowers the new registration threshold value to a lower value. As a result, the registered person who has left the store is deleted, and the situation where the same person is registered in duplicate is prevented, so that the number of registrations can be made closer to the estimated number of people.
Further, in S406, the correction unit 110 may retroactively apply the conditions changed in the processing of S403. In this case, the period for retroactively applying the updated threshold value may be set to be after the timing when the number of registrations and the estimated number of people are compared immediately before. As a result, it is possible to delete a person who has already been registered in duplicate or a registered person who has already left the store from the object list 104.
The condition relating to the deletion of the registered person from the object list 104 corresponds to an example of the "second condition".

On the other hand, if the correction unit 110 determines in S402 that the number of registrations is not more than a predetermined ratio than the estimated number of people, the process proceeds to S404. In S404, the correction unit 110 determines whether or not the number of registrations is smaller than the estimated number of people by a predetermined ratio or more.
When the correction unit 110 determines in S404 that the number of registrations is smaller than the estimated number of people by a predetermined ratio or more, the correction unit 110 proceeds to S405. In S405, the correction unit 110 executes at least one of relaxation of the condition relating to the deletion of the registered person and restriction (stricting) of the condition relating to the registration of the person in the object list 104. As a specific example, the correction unit 110 sets the registration deletion period threshold value to a threshold value corresponding to a longer period, and raises the new registration threshold value to a higher value. This prevents registered persons who exist in the store but have a relatively long undetected period from being deleted from the object list 104, and persons with similar clothes are also registered as different persons. Therefore, it is possible to bring the number of registrations closer to the estimated number of people.
Further, in S406, the correction unit 110 may retroactively apply the conditions changed in the processing of S405. In this case, the period for retroactively applying the updated threshold value may be set to be after the timing when the number of registrations and the estimated number of people are compared immediately before. As a result, when a registered person who is deleted from the object list 104 even though it exists in the store is registered again, or when registration of different persons with similar clothes has failed in the past, the object list 104 Can also be modified.

On the other hand, when the correction unit 110 determines in S404 that the number of registrations is not less than the estimated number of people by a predetermined ratio or more, the correction unit 110 proceeds to the process in S407.

In S407, the correction unit 110 determines whether or not to end a series of processes.
When the correction unit 110 determines in S407 that the series of processes is not completed, the correction unit 110 proceeds to the process in S408. In S408, the correction unit 110 waits for a predetermined period of time, and then re-executes the processes of S401 to S406. The above-mentioned predetermined period may be arbitrary, but in this embodiment, it is assumed to be "10 seconds" for convenience.

Then, when the correction unit 110 determines in S407 that the series of processes is to be completed, the correction unit 110 ends the series of processes shown in FIG. 7.

(Specific operation example)
Subsequently, an example of a specific operation of the information processing apparatus 100 according to the present embodiment will be described as an operation example 1 and an operation example 2.

-Example 1: Operation example 1
It is assumed that there are three people in the store and each of them is registered in the object list 104. Of the three people, person A has a bag with a different color depending on the orientation, and when the person A is detected at a certain time, the person A whose bag orientation is different from that when the person A is newly registered is detected from the image. It shall be assumed that it was done. Further, in this case, the collation score according to the collation result between the feature amount of the person A according to the new detection result and the feature amount of the person A already registered in the object list 104 is less than the new registration threshold value. It is assumed that Therefore, based on the detection result of the new person A, it is assumed that duplicate registration is performed as a person A'different from the person A already registered in the object list 104.
Under such circumstances, the number of registered persons is 4, while the estimated number of persons is 3, and a difference of 10% or more occurs between the number of registered persons and the estimated number of persons. Therefore, the correction unit 110 lowers the new registration threshold value, and then retroactively applies the updated new registration threshold value after the timing when the number of registrations and the estimated number of people are compared immediately before. ..
In this case, it corresponds to the collation result between the feature amount corresponding to the detection result of the person A and the feature amount of the person A already registered in the object list 104 in the registration opportunity of the person A'previously. The collation score becomes equal to or higher than the updated new registration threshold value, and the registration of person A'is deleted. As a result, the number of registrations becomes equal to the number of persons (3 persons) existing in the store, and the object list 104 is maintained in a state reflecting the actual situation.

-Example 1: Operation example 2
It is assumed that there are three people in the store and each of them is registered in the object list 104. It is assumed that a person B newly enters the store at a certain time, but the person B has similar clothes to the person C already registered in the object list 104. The person B is detected by the imaging unit 301a, and as a result of collating the feature amount according to the detection result with the feature amount of the person already registered in the object list 104, the collation score with the person C becomes high. It is assumed that the matching score does not fall below the new registration threshold. Therefore, it is assumed that the person B is not registered in the object list 104.
Under such circumstances, the number of registered persons is 3, while the estimated number of persons is 4, and a difference of 10% or more occurs between the number of registered persons and the estimated number of persons. Therefore, the correction unit 110 raises the new registration threshold value, and then retroactively applies the updated new registration threshold value after the timing when the number of registrations and the estimated number of people are compared immediately before. ..
In this case, the collation according to the collation result between the feature amount according to the detection result of the person B and the feature amount of the person C already registered in the object list 104 in the previous detection trigger of the person B. The score becomes less than the new registration threshold after the update, and the person B is registered. As a result, the number of registrations becomes equal to the number of persons (4 persons) existing in the store, and the object list 104 is maintained in a state reflecting the actual situation.

As described above, according to the information processing apparatus 100 according to the first embodiment, the algorithm related to the management of the object list 104 is used as the estimation result of the number of people in the store (in other words, the number of individuals in the area of interest). By making corrections based on the above, it is possible to improve the accuracy of the object list 104.

(Example 2)
Subsequently, as Example 2, a modified example of Example 1 will be described. Unless otherwise specified, the configuration of the system or device for realizing the second embodiment shall follow the above-described configuration as the first embodiment.
In the second embodiment, in addition to the method described in the first embodiment, a method of using the monitoring result of the entrance / exit for registering a person in the object list 104 is applied to the management of the object list 104. As a bird's-eye view of the store, the example shown in FIG. 3 shall be applied, and an image according to the image capture result by the image pickup unit 301d shall be used for monitoring the entrance / exit 311. Specifically, the feature amount is extracted from the moving image according to the image pickup result by the imaging unit 301d, and the person is registered in the object list 104 or from the object list 104 based on the feature amount. The registered person is deleted. In this case, the region in which the entrance / exit 311 is captured is specified in advance in the moving image according to the imaging result by the imaging unit 301d. Then, the newly detected person in the area is registered in the object list 104, and the registered person having a higher matching score with the person who disappeared in the area (that is, the person who exited from the entrance / exit 311) is deleted from the object list 104. It should be done.
Further, in the present embodiment, the method described in the first embodiment may be used as an auxiliary. Specifically, the method according to the present embodiment is mainly used for the management of the object list 104, and the person who is detected by the method according to the present embodiment is based on the method according to the first embodiment. Registration or deletion should be done. That is, in this case, the method according to the first embodiment plays a role of modifying the object list.

For example, FIG. 8 shows an example of the process related to the modification of the object list 104 when a new person is newly registered in the object list 104. The example shown in FIG. 8 is different from the process shown in FIG. 3 in that the process of S503 is added.
Specifically, in the example shown in FIG. 8, in S503, the collation unit 105 determines whether or not the number of registrations is less than a predetermined ratio with respect to the estimated number of people. Then, when the collation unit 105 determines in S503 that the number of registrations is less than a predetermined ratio with respect to the estimated number of people, the process proceeds to S504. That is, in this case, the detected person is registered in the object list 104 in S504.
The other processing is substantially the same as the example shown in FIG.

Further, FIG. 9 shows an example of the process related to the modification of the object list 104 when the same person as the person registered in the object list 104 is detected. The example shown in FIG. 9 is substantially the same as the example shown in FIG.

Further, FIG. 10 shows an example of the process relating to the deletion of the registered person from the object list 104 among the processes relating to the modification of the object list 104. The example shown in FIG. 10 is different from the process shown in FIG. 5 in that the process of S703 is added.
Specifically, in the example shown in FIG. 10, in S703, the collation unit 105 determines whether or not the number of registrations is more than a predetermined ratio with respect to the estimated number of people. Then, when the collation unit 105 determines in S703 that the number of registrations is more than a predetermined ratio with respect to the estimated number of people, the processing proceeds to S704. That is, in this case, the registered person whose undetected period is equal to or greater than the registration deletion period threshold value is deleted from the object list 104 in S704.
The other processing is substantially the same as the example shown in FIG.

Further, in the second embodiment, the condition correction based on the periodic comparison between the estimated number of people and the registered number may not be performed. In this case, the conditions for registering the person in the object list 104 and the conditions for deleting the registered person from the object list 104 are set in advance so as to be relaxed as compared with the first embodiment. It is good. If the conditions for registering a person in the object list 104 and the conditions for deleting a registered person from the object list 104 are relaxed, the registration of the person in the object list 104 may be omitted or the object list 104 may be deleted. People tend to be less likely to be deleted. On the other hand, in this case, the erroneous registration of the person in the object list 104 and the erroneous deletion of the registered person from the object list 104 tend to occur more easily.
On the other hand, in this embodiment, the above-mentioned erroneous registration or erroneous registration is performed by controlling so that the object list 104 is corrected when there is a deviation of a predetermined value or more between the estimated number of people and the registered number. It is possible to expect the effect of suppressing deletion. Therefore, it is possible to suppress the demerits related to the relaxation of the conditions related to the registration of the person in the object list 104 and the conditions related to the deletion of the registered person from the object list 104, and to enjoy the merits.

As described above, according to the information processing apparatus 100 according to the second embodiment, the object list 104 is modified when a deviation of a predetermined value or more occurs between the estimated number of people and the registered number. Therefore, the accuracy of the object list 104 can be further improved.

(Example 3)
Subsequently, as Example 3, a modified example of Example 1 will be described. Unless otherwise specified, the configuration of the system or device for realizing the third embodiment shall follow the above-described configuration as the first embodiment.
In the third embodiment, an example will be described in which a person who performs distrust behavior is detected in a situation where the area of interest is relatively wide, such as in a large store.
In this embodiment, an imaging unit (for example, a network camera) that includes the passage within the imaging range is provided for each of the series of passages in the store, and an image corresponding to the imaging result by each of these imaging units is produced. Based on this, the number of people in the store shall be estimated.
Specifically, the number of people is estimated based on the image according to the image pickup result by each imaging unit, and the estimation result of the number of people for each of the series of imaging units is added. However, for a plurality of imaging units in which the imaging regions overlap each other, the region where the imaging regions overlap (hereinafter, also referred to as “overlapping region”) and the other regions (hereinafter, also referred to as “non-overlapping region”) The number of people in is calculated individually. Then, the estimation result of the number of people in the non-overlapping area is added, and the estimation result of the number of people in each overlapping area is the estimation of the number of people in any of the plurality of imaging units corresponding to the overlapping area. The number of people in the store is estimated by being added as a result. This makes it possible to estimate the number of people in the store even when a plurality of imaging units are arranged so that overlapping regions exist.
Further, in order to register a person in the object list 104, delete a registered person from the object list 104, and acquire an action history, an imaging unit used for estimating the number of persons described above or the like is arranged. Only one or more imaging units may be used.

In the third embodiment, when the object list 104 is corrected based on the result of comparison between the registered number and the estimated number of people, the cause of the discrepancy between the registered number and the estimated number of people is estimated. Then, according to the estimation result of the cause, the condition for registering the person in the object list 104 and the condition for deleting the registered person from the object list 104 are corrected, so that the object list 104 The accuracy of is improved.
Specifically, the estimated number of people, the number of detected persons for each unit time and the imaging unit, the number of registered persons in the object list 104, and the number of deleted persons from the object list 104 are accumulated. Will be done. Then, from these data, two types of regression analysis are performed for each imaging unit. Specifically, as the first regression analysis, a regression analysis is performed using the estimated number of people in the store as an explanatory variable and the number of people detected per unit time as the objective variable. Further, as the second regression analysis, the number of detected persons per unit time is used as an explanatory variable, and the number of registered persons in the object list 104 and the number of deleted persons from the object list 104 are used as objective variables. Regression analysis is performed.
During the operation of the object list 104, the number of detected persons, the number of registered persons, and the number of deletions of the persons per unit time are estimated from the regression analysis for each of the series of imaging units in the store. The degree of deviation is calculated based on the standard deviation. Then, the factors having a larger degree of divergence and the imaging unit are weighted so that the amount of change in the threshold value becomes larger, and then the conditions for registering a person in the object list 104 and the object list 104. The conditions for deleting the registered person will be changed.

(Specific operation example)
Hereinafter, an example of a specific operation of the information processing apparatus 100 according to the present embodiment will be described as an operation example 1 and an operation example 2.

-Example 3: Operation example 1
The image quality of the moving image acquired from some imaging units deteriorates due to the lighting in the store being cut off and the inside of the store becoming dark, and the number of people detected based on the moving image is actually Suppose it is less than. As a result, the number of registrations becomes smaller than the estimated number of people, and the conditions for registering a person in the object list 104 and the conditions for deleting a registered person from the object list 104 are corrected. Specifically, for an imaging unit in which the number of detected persons is abnormally smaller than the number of detected persons predicted by regression analysis, the conditions for detecting the person are relaxed based on a larger weight. As a result, even for the imaging unit where the number of people detected due to the influence of lighting has decreased, it is possible to detect the person in a more realistic manner, and the difference between the number of registrations and the number of people actually existing in the store. Can be kept smaller.

-Example 3: Operation example 2
It is assumed that a group of multiple people, each wearing a suit, visits the store. Since the people wearing suits are similar in appearance, the matching score is high, and at least some people may not be registered in the object list 104. Therefore, under such a situation, the number of registrations is smaller than the estimated number of people, and the conditions for registering a person in the object list 104 and the conditions for deleting a registered person from the object list 104 are corrected. It will be done. Specifically, since the number of registered persons in the object list 104 per unit time is smaller than the number of registered persons in the object list 104 predicted by regression analysis for each imaging unit, the number of registered persons in the object list 104 is smaller. The conditions for registration will be relaxed. As a result, each of the plurality of persons having similar clothes is registered in the object list 104 as different persons, and the difference between the number of registered persons and the number of persons actually present in the store can be suppressed to a smaller size. ..

(Example 4)
Subsequently, as Example 4, a modified example of Example 1 will be described. Unless otherwise specified, the configuration of the system or device for realizing the fourth embodiment shall follow the above-described configuration as the first embodiment.
In the first embodiment, the conditions for registering a person in the object list 104 and the conditions for deleting a registered person from the object list 104 are corrected based on the result of comparison between the number of registered persons and the estimated number of people. .. Further, at this time, it is possible to retroactively apply the corrected condition after the timing when the number of registrations and the estimated number of people are compared immediately before.
On the other hand, in this embodiment, the corrected condition is not applied retroactively, and instead, the PTZ function of the imaging unit is used to make a difference between the number of registrations and the number of people actually present in the store. Is identified.

Specifically, when the number of registrations deviates from the estimated number of people by a predetermined ratio (for example, 10%) or more, the conditions and targets related to the registration of the person in the object list 104 are the same as in the first embodiment. The condition relating to the deletion of the registered person from the object list 104 is corrected. After that, by using the PTZ function of each imaging unit, a series of areas in the store are scanned, and each person detected based on the result of the scanning and a series of persons registered in the object list 104. Each is collated.
When the number of registrations is smaller than the estimated number of people, not only the person registered in the object list 104 but also the person deleted from the object list 104 based on the comparison result between the previously registered number and the estimated number of people. Is the target of collation including. As a result, the registered person who has already been deleted from the object list 104 according to the result of the collation will be registered in the object list 104 again if it exists in the store.
Further, since the person detected at this time is determined to be registered in the object list 104 based on the relaxed condition for the collation score, each of a plurality of persons having similar clothes to each other is regarded as an object. It becomes possible to register in the list 104.
When the number of registered persons is larger than the estimated number of people, a collation is performed between the detected person and a series of persons registered in the object list 104, and all of the detected persons are the same. Registered persons who are not determined to be persons are determined not to exist and are deleted.

As described above, according to the information processing apparatus 100 according to the fourth embodiment, when there is a discrepancy between the estimated number of people and the registered number, a person existing in the blind spot of the imaging unit is searched for. Then, based on the result of the search, the conditions for registering the person in the object list 104 and the conditions for deleting the registered person from the object list 104 are corrected. This makes it possible to further improve the accuracy of the object list 104.

<Other Embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or recording medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Further, the functional configuration of the information processing apparatus 100 described with reference to FIG. 1 is merely an example, and the configuration for that purpose is not particularly limited as long as each function of the information processing apparatus 100 can be realized.
For example, each function of the information processing device 100 may be realized by the cooperation of a plurality of devices. In this case, some of the functions of the information processing apparatus 100 and other functions may be realized by different devices. Further, the processing load related to at least a part of the functions of the information processing device 100 may be distributed to a plurality of devices.
Further, as long as the plurality of devices can transmit and receive information and data to each other, the type of network connecting the plurality of devices to each other is not particularly limited, and the position where each device is installed is also particularly limited. Not done.

100 Information processing device 109 Population estimation unit 110 Correction unit 301a to 301d Imaging unit

Claims (16)

A first acquisition means for acquiring first information according to a detection result of the object by each of one or more detection units that target at least a part of a predetermined area of interest for detecting the object.
A management means that associates the first information with each object and manages it as a list.
A second acquisition means for acquiring the second information according to the estimation result of the number of individuals of the object existing in the region of interest, and
With
The management means
The number of registered objects in the list is compared with the number of individuals indicated by the second information.
Based on the result of the comparison, the process related to the correction of the list is executed.
Information processing device. The information processing apparatus according to claim 1, wherein the management means executes a process for reducing the difference between the registered number and the individual number as a process related to the correction of the list. The management means, as a process related to the correction of the list,
The first condition regarding the registration of the object in the list and
The second condition regarding the deletion of the registered object from the list, and
Control at least one of the conditions,
The information processing device according to claim 2. When the number of registrations is larger than the number of individuals, the management means
The process of limiting the first condition so that the registration of the object in the list is further restricted.
A process of relaxing the second condition so as to further promote the deletion of the registered object from the list.
Do at least one of the,
The information processing device according to claim 3. When the number of registrations is less than the number of individuals, the management means
A process of relaxing the first condition so that the registration of the object in the list is further promoted.
A process of limiting the second condition so that the deletion of the registered object from the list is further restricted.
Do at least one of the,
The information processing device according to claim 3 or 4. The first condition includes a condition regarding a degree of divergence according to the result of comparison of the first information among a plurality of the objects.
The management means
The acquired first information is compared with each of the other first information associated with one or more of the objects registered in the list.
When the degree of deviation according to the result of the comparison is equal to or greater than the threshold value, the first information acquired is associated with another object different from each of the one or more objects registered in the list. And register the other object in the list,
The information processing device according to any one of claims 3 to 5. The process of limiting the first condition is a process of controlling the threshold value so that the degree of deviation required for registering the object with respect to the list becomes larger.
The process of relaxing the first condition is a process of controlling the threshold value so that the degree of deviation required for registering the object with respect to the list becomes smaller.
The information processing device according to claim 6. The list includes information on the timing at which the object is detected for each registered object.
The second condition includes a condition relating to an elapsed period from the timing at which the object is detected.
The management means deletes from the list the objects registered in the list whose elapsed period from the timing corresponding to the associated first information is equal to or greater than the threshold value.
The information processing device according to any one of claims 3 to 7. The process of limiting the second condition is a process of controlling the threshold value so that the elapsed period required for deleting the object registered from the list is longer.
The process of relaxing the second condition is a process of controlling the threshold value so that the elapsed period required for deleting the object registered from the list is shorter.
The information processing device according to claim 8. The second information is
An estimation method according to the number of the objects included as a subject in an image according to the image pickup result by one or more imaging units, and an estimation method.
An estimation method according to the detection result of the passage of the object in a predetermined area, and
An estimation method according to the past estimation result of the number of individuals of the object existing in the area of interest, and
Estimated based on at least one of
The information processing device according to any one of claims 1 to 9. The detection unit is an imaging unit that images at least a part of the region of interest.
The first information is
The feature amount according to the extraction result of the feature of the object from the image corresponding to the imaging result of at least a part of the region, and
Attribute information of the object extracted from the image and
Including at least one of
The information processing device according to any one of claims 1 to 9. The feature amount is
When extracted from the same object, the degree of similarity according to the result of mutual collation becomes higher.
When extracted from different objects, the similarity is lower according to the results of mutual matching.
Information,
The information processing device according to claim 11. When the object is a person, the attribute information includes the gender, age, height, body shape, body orientation, face orientation, beard presence / absence, mask presence / absence, and glasses of the person estimated from the image. The information processing apparatus according to claim 11 or 12, which includes at least one of the presence / absence of sunglasses, the presence / absence of sunglasses, the presence / absence of a beard, the type of a beard, and the color of clothing. The information processing device according to any one of claims 1 to 13, wherein the object is at least one of a human, a mobile body, an animal, and a robot. It is an information processing method executed by an information processing device.
The first acquisition step of acquiring the first information according to the detection result of the object by each of the one or more detection units that target at least a part of the predetermined area of interest.
A management step in which the first information is associated with each object and managed as a list, and
A second acquisition step of acquiring second information according to the estimation result of the number of individuals of the object existing in the region of interest, and
Including
In the management step
The number of registered objects in the list and the number of individuals indicated by the second information are compared.
Based on the result of the comparison, the processing related to the correction of the list is executed.
Information processing method. A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 14.

Images