JP6909575B2 - Monitoring information collection system - Google Patents

Description

The present invention relates to a monitoring information collecting system that identifies a person photographed by a camera and obtains a movement trajectory of the identified person.

A person who recognizes a person's face taken by a surveillance camera, for example, a person whose face is recognized from a database registered by associating the feature points of the person's face with attribute information such as the person's name, address, and occupation. A monitoring system has been developed that extracts the attribute information of. In this case, the person registered in the database may be, for example, a wanted criminal.
If you register the facial features of the wanted criminal in the database, you can use it to find the hidden wanted criminal.

In addition, for example, if the faces of related persons such as employees are registered in the database, when a person is photographed in an area that is off limits to persons other than the related persons by a surveillance camera, the related persons registered in the database by face recognition. It has been proposed to reduce false alarms by determining whether or not the information is present and not issuing an alarm in the case of a person concerned (see, for example, Patent Document 1).

In addition, a laser radar is used to obtain the movement locus of a moving body based on the distance to a moving body such as a person or a vehicle, and to predict the subsequent movement locus. For example, two moving bodies are at the same position at the same time. It has been proposed to determine that a crime, accident, or the like may occur if it is predicted to move to (see, for example, Patent Document 2).

Japanese Unexamined Patent Publication No. 2013-134675 Japanese Unexamined Patent Publication No. 2014-194686

By the way, even if the face of a person photographed by a surveillance camera is recognized, for example, if the face does not match the face registered in the database, the data of the person whose face is recognized cannot be used. In particular, it is not a facility that is used to move criminals like airports, but the frequency of finding wanted criminals by face recognition with surveillance cameras at convenience stores and other stores is not so high, so surveillance cameras The role of is to record crimes in the event of an accident or incident, and to deter crimes by showing that they are monitoring, and it is not practically used effectively.

In addition, when predicting an accident or the like by monitoring and predicting a movement trajectory using a laser radar, the laser radar cannot recognize the face of a moving person, so the movement speed, movement trajectory, and size of the moving body can be used. Although it is possible to determine a car, a bicycle, a pedestrian, an animal, etc., it is not possible to identify a person or the car name of the car, and the use is limited.

In addition, the images from the surveillance cameras are used to clarify the truth after an accident or incident occurs, but if a long period of time has passed since the accident or accident occurred, the captured image data will be erased. It may not be available for truth elucidation. In particular, when a plurality of surveillance cameras shoot and store (record) a moving image having a high number of pixels and a relatively high frame rate, the image data is stored and stored for a long period of time due to the problem of the storage capacity of the storage device. It's difficult.

The present invention has been made in view of the above circumstances, and by obtaining and recording the movement locus of a person from an image taken by a surveillance camera and performing face recognition on a person whose movement locus is required. An object of the present invention is to provide a surveillance information collection system capable of distinguishing from other persons whose faces are recognized.

In order to solve the above problems, the monitoring information collecting system of the present invention includes an imaging means for imaging a monitoring area including an exclusion zone and an entrance / exit where people are prohibited from entering.
A person detecting means for detecting a person from an image captured by the imaging means, and a human detecting means.
A locus analysis means for analyzing a person detected by the person detection means on the image to obtain a movement locus,
A face recognition means that recognizes a face by obtaining face data that can be distinguished from another face from a face on the image of a person whose movement locus is obtained by the locus analysis means while being detected by the person detection means.
A recording means for recording the face data and the movement locus corresponding to the face data in association with an identification code that makes the face data of different people distinguishable from each other.
The face data obtained by the face recognition means is compared with the face data recorded in the recording means, and it is determined whether or not the condition that the face data is registered in the recording means is satisfied. Face judgment means and
When the above conditions are satisfied, the movement locus corresponding to the face data obtained by the face recognition means is combined with the identification code associated with the face data recorded in the recording means that matches the face data. In association with the recording means, record
When the above conditions are not satisfied, the face data obtained by the face recognition means and the movement locus corresponding to the face data are associated with the identification code which is not yet associated with the face data to the recording means. Means for collecting face data to be recorded and
With
The imaging means is a first imaging means that includes a face recognition area and a person detection area wider than the face recognition area within the shooting range, and the entrance / exit to the monitoring area is included in the face recognition area. And a second imaging means that includes the off-limits area within the person detection area.

According to such a configuration, it is possible to identify a person who is within the photographing range of the imaging means by each face data and record the movement locus of each person. That is, the face data that makes it possible to identify a person and the movement locus of the person corresponding to the face data can be recorded in association with the identification code. Face data and movement loci can be used for verification when an accident or incident occurs. This face data and movement locus have a smaller storage capacity than video data, and can be stored for a long period of time even with the limited storage capacity of the storage device, which is effective for re-verification of past accidents and incidents. It can be utilized.

Further, the movement loci of a plurality of people that can be identified by the face data can be displayed on one screen at the same time, and more information can be obtained in a shorter time than when the moving image data is viewed. In addition, by face recognition, it is possible to determine whether or not the same person has passed even if the date is different, and it is possible to monitor the behavior of the same person over a long period of time. Further, for example, the movement trajectory of a person in the store can be referred to when considering the display of products. In addition, the movement locus can be used to grasp the flow of people on roads such as shopping streets, to control signals, to determine traffic regulations, and the like.

Further, the identification of a person by the face recognition in the present invention distinguishes the face photographed by the imaging means from another face, in other words, the face of the same person is determined to be the same face, and the faces of different people are identified. Since it is only necessary to make it possible to determine different faces, it is not always necessary to link personal information such as name and address to the face data. It suffices if it can be distinguished from the face of the existing customer. As a result, it becomes possible to manage customers in stores regardless of personal information, and it is possible to provide services and the like according to the tastes of customers. In addition, it becomes easy to manage not only patients but also visitors in nursing care facilities and hospitals.

In the configuration of the present invention, the plurality of the imaging means are provided, and the trajectory analysis means is within a range in which the imaging ranges of the plurality of imaging means are combined based on the images captured by the plurality of imaging means. It is preferable to obtain the movement locus with.

According to such a configuration, the movement locus can be obtained even when the same person moves over the shooting range of the plurality of imaging means. In addition, even if a person temporarily deviates from the shooting range and cannot track the movement destination, the movement locus can be obtained again when the face can be recognized again, and the continuous movement locus of the same person even if a part is interrupted. Can be estimated. Therefore, in a store or facility having a complicated structure, it is possible to obtain a movement trajectory without losing sight of the movement destination of a person.

In the configuration of the present invention, there is provided an attribute analysis means for obtaining attribute information which is an attribute of the face data from at least the image, the movement locus, or external information input from the outside.
It is preferable that the recording means records the attribute information obtained by the attribute analysis means in association with the identification code.

According to such a configuration, it is possible to add attribute information to the face data corresponding to each movement locus. For example, the height and body shape of a person recognized by the image taken by the imaging means can be analyzed. It may be used as attribute information, or by adding attribute information as to whether or not the person is a related person such as a staff member to the face data, the related person can be excluded from the monitoring target. For example, even if the movement locus of a person concerned overlaps with a place that is off limits to people other than the person concerned, it can be ignored without issuing an alarm. Further, as the information analyzed from the movement locus, for example, a specific human behavior pattern, for example, an behavior pattern such as passing through a predetermined position at a predetermined time on a predetermined day of the week can be recorded as attribute information.

In the configuration of the present invention, the imaging means is arranged in a store where products are displayed.
In the attribute analysis means, arrangement information indicating the arrangement of the product is input as the external information, and based on the arrangement information and the movement locus, the information indicating the product approached by the person recognized by the face is the attribute information. Is preferable.

According to such a configuration, when the information on the arrangement of the product is input and stored as the external information, the attribute analysis means refers to the movement locus and the arrangement of the product and identifies the product approached by the customer as the attribute information. Can be. In addition, it may be analyzed whether or not the customer stopped when approaching. In addition, when it is not possible to identify a product that the customer has approached, the classification of the same type of products arranged together may be specified as a product. For example, cup noodles, stationery, etc. may be specified as a product. You may.
Therefore, it is possible to grasp the sales floor frequently used by each customer identified by the face data and use it for customer management, and it is easy to relate the movement trajectory of the customer and the arrangement of the products when considering how to display the products. Can be grasped. By using the image taken during the attribute analysis, it is possible to recognize the product picked up by the customer or the product in the shopping cart and use it as the attribute information.

In the configuration of the present invention, the imaging means is arranged in a store having a cash register capable of outputting accounting information indicating the accounted product.
The attribute analysis means corresponds to the movement locus in which the accounting information input as the external information is located close to the cash register when the accounting corresponding to the accounting information is performed in the cash register. It is preferable to use the attribute information.

With such a configuration, it is possible to associate the movement locus with the actually purchased product, and it is possible to analyze the relationship between the movement locus and the purchased product. Further, if the arrangement of the above-mentioned products in addition to the movement locus and the purchased products is known, it becomes possible to examine the product display method in more detail.

In the configuration of the present invention, the face data for a specific person is identifiablely recorded in the recording means, and the face recognition means identifies the specific person from the face data. Is preferable.

According to such a configuration, the person who has been theft can be excluded from the monitoring target by using the fact that the person is the person who is the person concerned as the attribute information as described above, or the person who has experienced theft is visited by the person who has experienced theft as a specific person. It is possible to notify the customer of the visit of the customer as a specific person.

According to the present invention, each person can be identified and the movement locus of each person can be recorded and used.

It is a schematic diagram which shows the case where the monitoring information collection system of embodiment of this invention is installed in a store. It is a block diagram showing the monitoring information collection system. It is a block diagram showing the detection / recognition means of the monitoring information collection system. It is a figure explaining the shooting range of the camera of the surveillance information collection system. It is a figure for demonstrating the monitoring information collected by the monitoring information collection system. It is a flowchart for explaining the monitoring information collection method by the monitoring information collection system.

Hereinafter, embodiments of the present invention will be described.
As shown in FIG. 1, the monitoring information collecting system of the present embodiment has a monitoring camera 1 installed in the monitoring area. In the present embodiment, the camera 1 does not simply capture the monitoring area, but detects a person, analyzes the detected movement locus of the person, recognizes the face of the person who is detected and analyzes the movement locus, and the like. It has a function of recording these monitoring information in association with an ID as an identification code. It is not necessary to provide all of these functions in the camera 1, and functions other than shooting and image output may be provided on the external device side such as a server into which the image output from the camera 1 is input.

In the present embodiment, the monitoring area where the camera 1 is installed is, for example, a store 50 such as a convenience store. The store 50 has a customer entrance / exit 56, and a customer 40 who is a person enters the store from the customer entrance / exit 56. In the store 50, a cashier counter 52 having a register 51 near one wall is arranged parallel to the wall surface at a distance from the wall, and a specific space between the cashier counter 52 and the wall is specified in the store 50. It is an off-limits area 53 where the entry of customers 40 (people) other than the person concerned 41 such as a clerk who is a person is prohibited.

Further, in the store 50, a plurality of product shelves 54 are arranged vertically and horizontally in a matrix shape at a position away from the wall. Further, on a wall other than the wall on which the above-mentioned cash register counter 52 is located, a product shelf 55 is arranged near the wall. In the store, three cameras 1 of the present embodiment are arranged for monitoring. In this example, one unit is arranged on the wall on the side of the cashier counter 52, and two units are arranged on the wall on the opposite side of the cashier counter 52. The arrangement of the cameras 1 can be arbitrarily set and may be one or a plurality of cameras, but it is preferable that as many parts of the store 50 as possible can be photographed.

Each camera 1 constitutes the monitoring information collecting system of the present embodiment, and as shown in FIG. 2, for example, an image pickup means 2 having a lens, a solid-state image pickup element, or the like, and an image pickup means 2 take an image. The detection / recognition means 3 (detection recognition means 3) for detecting a person from the image, analyzing the movement locus of the detected person, and recognizing the face of the person whose movement locus is analyzed, and the image pickup means 2. A recording means 5 that records captured image data and monitoring information obtained by being detected and recognized by the detection / recognition means 3, and a communication means 6 that communicates with an external device via a network 7 (communication line). And the control means 4 for controlling the image pickup means 2, the detection / recognition means 3, the recording means 5, and the communication means 6.

It should be noted that, instead of having all of the above-mentioned components provided in the camera 1, the imaging means 2 and the communication means 6 are arranged in the camera 1, and other components are arranged in a computer system such as a server outside the camera 1 or other components. It may be provided in the device. Further, when a plurality of cameras 1 are arranged in a shooting area of a store 50 or the like, all of the above-mentioned components are arranged in one camera 1, and the remaining cameras 1 basically include an imaging means 2. Only the communication means 6 may be provided. That is, when there are a plurality of cameras 1, one camera 1 may be used as a general surveillance camera by adding a function as a computer system and outputting an image of the remaining cameras 1. In any case, the surveillance information collection system can be composed of a plurality of cameras 1.

The imaging means 2 has, for example, a configuration similar to that of a general surveillance camera. For example, the imaging means 2 captures a shooting range corresponding to an angle of view set according to the orientation of the camera 1, and captures the captured image as a moving image. It is designed to be sent as. As the imaging means 2, in the present embodiment, a stereo camera capable of calculating the distance / size from the parallax is used. By using this stereo camera, the shape of the subject can be calculated, and when performing various recognition from the feature points on the image, the feature points can be obtained from the three-dimensional shape, so that the recognition becomes more accurate and human detection is easy. It becomes possible to perform face recognition in more detail.

Further, the change in the detected distance to the person can be calculated from the parallax, and the movement locus of the person can be easily analyzed based on the distance from the camera 1 to the detected person. Further, based on the image of the stereo camera, the height, body shape, clothes, etc. are required as the attribute information of the detected person, and based on these, the following estimation can be performed. That is, it may be used as attribute information by estimating whether the person photographed by the stereo camera is an adult or a child, fat or thin (body shape), a man or a woman, and the like. The behavior of the customer 40 in the store 50 may be image-recognized to detect criminal acts such as so-called shoplifting. For example, the product picked up may be recognized and the product may be returned or purchased. It may be determined whether the product can be put in the basket or not, and in other cases, the person concerned 41 may be notified. In addition, it may be configured to judge from the movement of a person on the video whether or not shoplifting has been performed by machine learning.

Further, the camera 1 may be provided with a voice input means for inputting voice in addition to the imaging means 2, and the voice in the monitoring area may be digitally recorded or voice recognized, and the voice is recognized. In some cases, it may be determined whether or not the order in the monitoring area is disturbed.

Further, the imaging means 2 is not limited to a stereo camera, and may be a monocular camera. In this case, since the distance cannot be calculated from the disparity, the height, size, and distance of the store 50, which is the background of the person, such as the positions of the product shelves 54 and 55 and the height of each of these shelves, are compared with those of the person. The position and shape of the constituent elements that can be calculated may be stored, and based on this, the detection of a person, the analysis of the movement trajectory of the person, the height, the body shape, etc. of the person may be obtained. Further, as the image pickup means 2, for example, a distance image sensor may be used. As the distance image sensor, for example, TOF (Time Of Flight) can be used. The TOF measures the distance from the time it takes for the projected laser to make a round trip to the target.

If the distance image sensor is used as the imaging means 2, the distance to the subject can be obtained, and face recognition can be performed by increasing the resolution. The distance image sensor can detect a person, move a trajectory, and recognize a face. Become. When the resolution of the distance image is not sufficient and the accuracy of face recognition is not improved, face recognition may be performed by combining the distance image with an image obtained by a monocular camera.

The detection / recognition means 3 is a computer system including an arithmetic processing unit and a memory, and performs image recognition. That is, image analysis and image recognition such as person detection, person tracking (movement locus), face recognition, etc. are performed by image recognition from the image captured by the imaging means 2. In human detection, it is determined by image recognition whether or not the subject on the image is a person. Here, in the store 50, a person enters the store from the entrance / exit 56, and the person who enters the store is detected. In the person detection, the subject moving from the entrance / exit 56 into the store is a person candidate.

In human detection, it is determined whether or not the person is a person, but in face recognition, facial feature points, feature amounts, etc. are obtained by face image recognition and stored as face data. When comparing a plurality of face data, for example, the face data of the same person has many parts where the feature points and the feature amounts are similar, and the face data of different people are not similar as in the case of the same person. Therefore, it is possible to determine whether the two face data compared by the well-known comparison conditions such as the feature points to be approximated when the face data are compared and the number of feature quantities are those of the same person or those of different people. can. That is, face images can be matched. Actually, for example, the above-mentioned person detection, person tracking, and face recognition can be performed by using OpenCV (Open Computer Vision Library), which is a library of functions related to image recognition, and recognition by deep learning. .. The latest OpenCV library also includes machine learning capabilities, such as a deep learning module.

Here, in the present embodiment, not only the person is tracked on the image, but also the actual position of the person tracked on the image in the store 50 is obtained based on the distance obtained by the parallax of the stereo image. The change in this position with the passage of time is obtained as a locus. It should be noted that the determination of whether or not the person is a person in the person detection and the face recognition are also performed based on the three-dimensional image obtained from the stereo image. Therefore, feature points are also obtained on a three-dimensional stereoscopic image. However, as described above, when the monocular camera is used in the imaging means 2, the positional relationship between the fixed structure as the background and the subject detected and tracked is estimated, and the position in the store 50 of the person is estimated. May be determined.

Further, the position of a person in the store 50 may be obtained based on the distance measured by a distance image sensor such as TOF, and the movement of this position may be recorded as a locus. In this case, it is possible to perform person detection, person tracking, and face recognition based on the distance image output from the distance image sensor, and in this case as well, the three-dimensional image is image-recognized. Further, image recognition may be performed by combining the distance image of the distance image sensor and the image of the monocular camera, and the distance to each pixel by the distance image and the image of the monocular camera having a higher resolution than the distance image are used. Image can be recognized.

From the above, as shown in FIG. 3, the detection / recognition means 3 tracks the person detecting means 31 that detects a person from the image and the person detected from the image on the image, and analyzes the position of the person. The locus analysis means 32 for obtaining the movement locus of the person and the face recognition means 33 for recognizing the detected face of the person are provided. The person detecting means 31, the locus analysis means 32, and the face recognition means 33 are connected to the recording means 5 via the control means 4, and the ID and the locus analysis means 32 of the person detected by the person detection means 31 obtain the data. The movement locus obtained and the face data obtained by the face recognition means 33 are stored.

In extracting feature points such as human eyes, nose, mouth, chin, forehead, and hair, the size of the face in the image becomes a problem, and the size of the person in the image that can be detected as a person is used. The size of the face in the face-recognizable image is likely to be larger. That is, within the shooting range of the camera 1, the distance from the camera 1 capable of face recognition to the person is shorter than the distance from the camera 1 capable of detecting a person to the person. That is, as shown in FIG. 4, the range close to the camera 1 in the shooting range of the camera 1 is the face recognition area 10, and the range wider than the face recognition area 10 is the person detection area 11.

The person detection area 11 includes a face recognition area 10. In this case, for example, in a store 50 such as a convenience store, if the entire range in the store 50 is to be contained in the face recognition area 10 of the camera 1, the number of cameras 1 is increased or the resolution of the camera 1 is increased. When it becomes necessary to increase the cost and the cost increases, as shown in FIG. 1, only a part of the area in the store 50 may be used as the face recognition area 10. For example, as shown in FIG. 1, when three cameras 1 are installed so that the entire range in the store falls within the shooting range of the camera and also falls within the person detection area 11 of the camera 1, face recognition The area 10 may cover only a part of the store 50.

In the present embodiment, in the store 50 shown in FIG. 1, almost the entire area of the store 50 is within the person detection area 11 (shooting range) of the camera 1 at the lower right, but the product shelf 54 is actually used. There is a part that is not photographed even in the person detection area 11 due to the obstruction. Cameras 1 are arranged at the left and right ends of the store 50 so as to cover the portion that cannot be photographed. However, the face recognition area 10 covers only a part of the store 50.

Further, the face recognition area 10 of the camera 1 arranged at the left end of the store 50 includes the entrance / exit 56 of the store 50, and even if only a part of the store 50 is the face recognition area 10, the entrance / exit The faces of all the customers 40 who enter the store from 56 can be recognized, and the customers 40 are face-recognized at the time of entering the store, and for example, an existing customer 40 or a new customer 40 can be determined. There is. Further, the camera 1 on the right side of FIG. 1 can take a picture of the checkout counter 52 on the left side of the figure and the exclusion zone 53 in the checkout counter 52, and a customer 40 who is not a related person 41 can take a picture in the exclusion zone 53. It is possible to monitor the entry and also to monitor unauthorized access to the cash register.

In this embodiment, it is sufficient that each customer can be identified. For example, face recognition is used to determine whether the customer who visited the store yesterday morning and the customer who visited the store this morning are the same person or a different person. There is no need to link face data with personal information such as name and address. However, as will be described later, data of accounting information such as purchased items, purchase amount, purchase date and time, cash payment and prepaid card payment output from the register 51 may be linked. That is, as part of customer management, each customer identified by an ID is identified, and actions including purchase of each customer are recorded.

The control means 4 may control the start and end of photography by the image pickup means 2, and may also control focus adjustment, exposure compensation, and the like. Further, the control means 4 records the data of the image taken by the image pickup means 2 in the recording means 5. Further, the control means 4 includes face data recognized by the detection / recognition means 3, movement locus data for each face data, physical information such as height and body shape recognized from the image, and these physical information. Estimated information such as age, gender, race, etc. estimated from information such as clothes read from the image and makeup recognized at the time of face recognition is attached to face data and movement trajectory data as ancillary information that is a part of monitoring information. It is associated with the ID and recorded in the recording means 5.

The communication means 6 communicates with another camera 1 via the network 7 to supply information and take over the movement trajectory of a person. The movement locus is basically taken over from the movement locus of people whose positions match at the overlapping shooting areas, but if the face is recognized by each camera 1, each camera of the same person Each movement locus analyzed in 1 may be connected based on satisfying a predetermined matching condition of the face data obtained by face recognition.

Further, as the information collecting device in the photographing information collecting system, the above-mentioned imaging means 2, detection / recognition means 3, recording means 5, communication means 6, and control means 4 are provided in each camera 1, but they are collected. A device for viewing information may be provided. For example, a terminal provided with an input device such as a display, a keyboard, and a pointing device, and an arithmetic processing device is connected to a communication means 6 of each camera 1 via a network 7. You may. With this terminal, data such as images, movement loci, and attached information stored in the recording means 5 of the camera 1 can be viewed and managed.

It is not necessary to constantly monitor the image of the camera 1. For example, the camera 1 automatically performs the image recognition based on the image recognition by the function of sounding an alarm when a person other than the person concerned 41 invades the exclusion zone. It may be configured to be processed. Further, it is sufficient that the data of the recording means 5 is output to an external computer system so that the computer system can manage customers, examine the display method of products, and the like, and the store 50 does not have to have a terminal.

Further, the communication means 6 of the camera 1 is connected to the cache register 51 via the network 7. The cash register 51 is, for example, a POS cash register for a POS system (point-of-sale information management system), and includes a product name, price, purchase date and time, payment method (cash, card payment, etc.) of a product purchased by the customer 40 at the time of sale. Outputs the points of the point card. In the present embodiment, the above-mentioned various information is transmitted from the cache register 51 to the camera 1.

When the information corresponding to the purchase of the product is input from the cash register 51, the control means 4 of the camera 1 is the customer 40 closest to the cash register outside the off-limits area 53 of the cash register counter 52 when this information is input. Information about the purchased product or the like transmitted from the cash register 51 is recorded in the recording means 5 in association with the ID of the above. In the recording means 5, the information transmitted from the cache register 51 is associated with the ID and recorded as incidental information.

As a result, the purchased product of the customer 40 can be known, and the purchased product and the movement locus can be known, so that it can be used as a basic material when examining the display method of the product. That is, it is possible to analyze which product has passed by before or after purchasing the product, and to find the relationship between the display of the product and the movement of the customer 40. In addition, analyze information such as what kind of products the customer whose movement trajectory stopped in front of the magazine section purchased after that, and whether the customer who stopped in front of the lunch section purchased a lunch box. As a result, it is possible to obtain a method for displaying products, an optimum replacement time for products having a short display time such as bento boxes, and a suitable display method for beverage-based products and bento-based products.

Next, the data stored in association with the ID (a) in the recording means 5 will be described. The data stored in association with the ID (a) are the face data (b), the profile (c), the purchase history (d), and the movement locus (e). As described above, the face data (b) is feature points, feature amounts, and the like, and is data that enables identification of the same person or another person when there are two face images. The profile (c) is data obtained by analyzing (recognizing) an image taken by the camera 1 and data input from the outside, and is ancillary to face data (b) and movement locus (e). Information.

The profile (c) includes, for example, data based on image recognition such as height, body shape, adult / dwarf, gender, clothes (color, etc.), and is also allowed to enter and exit the related persons (access prohibited area 53) described later. Person) or not, etc., which is input from the outside is included. The purchase history (d) is data input from the cash register 51, and includes a product name (product ID), quantity, purchase amount, type of purchased product, and the like of the purchased product. The movement locus (e) basically has data on the position in the store 50 and the date and time, and connects the positions in chronological order. In addition, information such as a person requiring attention may be registered in the profile (c).

Next, a method of collecting monitoring information by such a monitoring information collecting system and a monitoring method will be described with reference to the flowchart of FIG. The process shown in the flowchart is basically the process in the control means 4.
First, when a customer 40 enters the store from the entrance / exit 56 of the store 50, the customer 40 who entered the store is photographed by the camera 1, and the customer 40 who enters the store from the entrance / exit 56 in the photographed image is a detection / recognition means of the camera 1. 3 is detected as a person (step S1). The detection and recognition means 3 tracks the person detected on the image and analyzes the movement, and detects the movement locus until the detected person is no longer detected, for example, until the person leaves the store 50, in response to the passage of time. (Step S2).

The detection recognition means detects and recognizes the face of a person who is detected and tracked on the image in order to obtain the movement locus, obtains the feature points and feature amounts of the face, and records this as face data in the recording means 5. (Step S3). Note that face recognition is performed, for example, by taking a picture with the camera 1 when the customer 40 enters the store from the entrance / exit 56, but if face recognition fails here, the next step is to obtain the movement trajectory. Face recognition is performed when the face recognition area 10 of any of the cameras 1 is entered and the face is turned toward the camera 1.

When face recognition is performed, the face data that has already been face-recognized and stored in association with the ID as described later is compared with the face data acquired by face recognition this time, and the newly recognized face data is already present. It is determined whether or not the face data is recorded, that is, whether or not the recognized face data is registered face data (step S4). At this time, when the number of matching points of the feature points and the feature amounts of the face data is small and the matching condition is not satisfied, an unused new ID is registered as a new customer 40 (step S5). Information such as face data and accompanying information associated with this new ID is recognized as a profile (step S6).

On the other hand, when the newly recognized face data is compared with the face data already recorded in association with the ID, it is determined that there are many matching points of feature points and feature amounts and the matching conditions are satisfied. That is, when the recognized face data has already been registered, the above-mentioned face data and accompanying information are recognized as a profile of the registered ID (step S7). Then, the profile recognized in association with the registered ID is updated by the recording means 5 as the profile of the registered ID (step S8).

As described above, a new profile is stored together with the new ID or the registered ID, and the movement locus data is recorded in the recording means 5 as the locus profile in association with any of these IDs (step S9). The movement locus is basically detected until the customer 40 goes out of the store, and the locus profile is updated periodically.
Next, it is determined whether or not the customer 40 is still in the store 50 (in the area) (step S10). When the customer 40 is not in the store 50 and is determined to be outside the area, the detection of the movement locus is completed, and the recording means 5 records the information associated with the ID including the information of the movement locus (file, etc.). ) Is closed (step S11).

If there is still a person detected in the store 50 (in the (monitoring) area), it is determined whether or not the current movement locus or the movement locus within the predetermined time overlaps with the exclusion zone 53 (prohibited area). (Step S12). That is, it is determined whether or not the detected customer 40 has invaded the restricted area 53. Next, referring to the profile recorded in the recording means 5 associated with the ID of the invading customer 40, the person concerned with the store 50 such as a clerk is permitted to invade the exclusion zone 53 in the profile of the ID. It is determined whether or not the person has an ID (step S13).

In the profile registered for each ID of the recording means 5, in addition to the items recognized from the image and the items estimated from the image and the recognized items, the attached information input from the outside can be stored. Here, as ancillary information (profile), it is recorded in the profile of the store-related person 41 that, for example, the entry / exit permission to the restricted area 53 is given.

Therefore, when the face data of the person concerned 41 is recognized and the ID is read out, the entry / exit permission to the restricted area is given as the attached information associated with the ID. Therefore, the movement locus of the person concerned 41 and the restricted area Even if it overlaps with 53, the alarm does not sound. An alarm device for issuing an alarm is installed in the store 50, and the control means 4 of the camera 1 outputs a command indicating the start and end of the alarm (alarm) to the alarm device via the communication means 6 and the network. It is possible.

Therefore, when the movement locus is associated with the ID of the person whose movement locus overlaps with the exclusion zone 53 and there is no entry / exit permission to the exclusion zone 53, the control means 4 sets the alarm start recommendation in the alarm device (step S14). In the next process, an alarm start command is output to the alarm device via the communication means 6. On the other hand, when the access permission to the exclusion zone 53 is recorded in the profile in association with the above-mentioned ID, the control means 4 does not set the alarm command.

The processing by the control means 4 described above is actually a real-time processing, and the detection / recognition means 3 can simultaneously detect the movement loci of a plurality of people, and the detection / recognition means 3 may simultaneously detect a plurality of movement loci. It is possible to output the movement trajectory of the person. On the other hand, the control means 4 updates the movement locus and the profile in real time.

Further, when there are a plurality of cameras 1, the cameras 1 are interlocked with each other, and it is determined based on the movement locus and the face data whether the person being photographed by each camera 1 at the same time is the same person or a different person. In addition, when a person with the same ID is photographed by a plurality of cameras 1, each camera 1 cooperates so that one movement locus can be obtained for one ID.

Therefore, when a person sequentially moves a shooting range in which the plurality of cameras 1 do not overlap each other, the movement loci analyzed by each camera 1 are connected. At this time, in the case of a person whose face is recognized by one camera 1 and whose face is not recognized by the other camera 1, the ID is fixed at the time of face recognition, and the movement locus is also determined by the camera 1 whose face is not recognized. When is connected to the movement locus detected by the camera 1 that recognizes the face, the same ID is attached to the movement locus, and the movement locus detected by each camera 1 is connected.

According to the monitoring information collection system of the present embodiment, a person who has entered the monitoring area is identified by person detection and face recognition, and an ID is assigned to each identified person. Therefore, a person in the monitoring area is assigned an ID. Even if you do not know personal information such as your name, you can manage it with your ID. That is, it is possible to determine whether a person in the monitoring area is the first person to enter the monitoring area, a person who has entered the monitoring area in the past, or a person who frequently enters the monitoring area. It is possible to notify the persons concerned 41 that they have entered the monitoring area or that a person who always comes has entered the monitoring area.

When the monitoring area is set to the store 50, it is possible to notify the store clerk that the regular customer 40 has arrived or the first customer 40 has arrived, which facilitates the response to the customer. In addition, by knowing the movement of each customer 40 in the store 50 and the products purchased by the customer 40, it is possible to refer to the layout in the store 50 such as the arrangement of each product and the arrangement of each product shelf. Is.
Further, as the data for examining the layout in the store 50, the arrangement information indicating the arrangement of the products (including the arrangement of the product shelves) is recorded as the external information input to the recording means 5 from the outside. The movement locus and the arrangement of the products obtained as described above may be analyzed, and the products approached by the customer, the products approached by the customer, and the products stopped may be recorded as attribute information. The product in this case may be a rough product type, for example, a product name that can be identified by a manufacturer or the like, but if it cannot be specified to that extent, a notebook or stationery may be used as product information.
Further, by image recognition, the information of the product in the position where the customer 40 of the product shelves 54 and 55 has stopped, the image of the product when the customer 40 picks up the product, and the position of the product before being picked up are obtained. Information on the goods picked up by the customer may be recorded.

The data recorded in these recording means 5 is used for customer management as data for each customer 40, for example, the repeat rate at which the customer returns to the store, the repeat rate at which the same product is bought, the rate at which the new product is bought, and the same movement. It is required whether to move along the trajectory. In addition, the composition ratio of the types of customers 40 (adults / children, gender, etc.) to be image-recognized by collecting the data of all the customers 40 of the store 50 or the plurality of stores 50, that is, there are many children. It is possible to obtain the number of women, the correlation between the movement locus of each customer 40 and the purchased product, the correlation between the movement locus of each customer 40 and the profile (attribute) of each customer 40, and the like. That is, it is possible to use the movement locus of the customer 40 in collecting information for marketing and the like.

In addition, the surveillance information collection system can search for images of persons involved in a crime after a crime occurs as a general monitoring device, but for example, when there are images such as photographs of persons involved in the crime of the victim or the perpetrator. In addition, the photograph can be taken by the camera 1 to recognize the face, and it is possible to determine whether or not the same person is recorded in the recording means 5 of the monitoring information collecting system. At this time, it is possible to determine whether or not the same person exists based on the recorded face data even if there is not necessarily the data of the captured moving image. Therefore, even if the video data with a large amount of data is not stored for a long period of time, if the face data etc. is stored for a long period of time, it is possible to know whether or not a criminal person has come, the movement trajectory when the criminal person has come, and the purchased product. It becomes possible.

The monitoring area in the monitoring information collection system of the present embodiment is not limited to the store 50, and various facilities such as apartment buildings, tenant buildings, factories, museums, museums, and office buildings may be used as monitoring areas. can. Even when these facilities are used as monitoring areas, it is possible to detect and identify a person and detect a movement locus, so that a suspicious person can be found from the movement locus. For example, in an apartment house, it is usual to move from the entrance to one's room on the shortest course, and a person who draws a movement trajectory other than that may be a person requiring attention.

In addition, even in facilities other than apartment houses, there is a high possibility that it can be determined whether or not the person is a suspicious person based on the movement trajectory. It is possible to determine that. Further, as described above, according to the monitoring information collecting system of the present embodiment, if an image such as a face photograph is obtained, it is possible to automatically determine whether or not the user has entered the monitoring area, and the movement locus thereof. It is possible to obtain the profile, and it is possible to determine whether or not there is a criminal person more easily than a human being analyzes the recorded image of the camera 1. Further, in a facility such as a store 50 where a customer 40 enters and exits, but a facility where non-related persons rarely enter, a suspicious person can be easily found by registering the related persons in advance. Further, in such a facility, it is possible to determine a person who is recognized for the first time other than a person who has already been recognized as a suspicious person.

1 Camera 2 Imaging means 3 Detection / recognition means 31 Person detection means 32 Trajectory analysis means 33 Face recognition means 4 Control means (face data collection means, face determination means, attribute analysis means)
5 Recording means 6 Communication means

Claims (7)

An imaging means for imaging a monitoring area including an exclusion zone and an entrance / exit where no one is allowed to enter,
A person detecting means for detecting a person from an image captured by the imaging means, and a human detecting means.
A locus analysis means for analyzing a person detected by the person detection means on the image to obtain a movement locus,
A face recognition means that recognizes a face by obtaining face data that can be distinguished from another face from a face on the image of a person whose movement locus is obtained by the locus analysis means while being detected by the person detection means.
A recording means for recording the face data and the movement locus corresponding to the face data in association with an identification code that makes the face data of different people distinguishable from each other.
The face data obtained by the face recognition means is compared with the face data recorded in the recording means, and it is determined whether or not the condition that the face data is registered in the recording means is satisfied. Face judgment means and
When the above conditions are satisfied, the movement locus corresponding to the face data obtained by the face recognition means is combined with the identification code associated with the face data recorded in the recording means that matches the face data. In association with the recording means, record
When the above conditions are not satisfied, the face data obtained by the face recognition means and the movement locus corresponding to the face data are associated with the identification code which is not yet associated with the face data to the recording means. Means for collecting face data to be recorded and
With
The imaging means is a first imaging means that includes a face recognition area and a person detection area wider than the face recognition area within the shooting range, and includes the entrance / exit to the monitoring area in the face recognition area. And a monitoring information collecting system comprising the second image pickup means which includes the off-limits area in the person detection area. The first aspect of claim 1, wherein the locus analysis means obtains a movement locus within a range in which the respective shooting ranges of the plurality of imaging means are combined based on the images captured by the plurality of imaging means. Monitoring information collection system. It is provided with an attribute analysis means for obtaining attribute information which is an attribute of the face data from at least the image, the movement locus, or external information input from the outside.
The monitoring information collecting system according to claim 1 or 2, wherein the recording means records the attribute information obtained by the attribute analysis means in association with the identification code. The imaging means is placed in the store where the product is displayed, and the image pickup means is arranged.
In the attribute analysis means, arrangement information indicating the arrangement of the product is input as the external information, and based on the arrangement information and the movement locus, the information indicating the product approached by the person recognized by the face is the attribute information. The monitoring information collection system according to claim 3, wherein the monitoring information collection system is characterized. The imaging means is arranged in a store having a cash register capable of outputting accounting information indicating the accounted product.
The attribute analysis means corresponds to the movement locus in which the accounting information input as the external information is located close to the cash register when the accounting corresponding to the accounting information is performed in the cash register. The monitoring information collection system according to claim 3 or 4, wherein the attribute information is used. Claim 1 is characterized in that the face data for a specific person is identifiablely recorded in the recording means, and the face recognition means identifies a specific person from the face data. -The monitoring information collection system according to any one of claims 5. In the recording means, information on whether or not a person detected by the person detecting means is permitted to enter the restricted area in association with the identification code is recorded .
Further, an alarm means for generating an alarm when the movement locus of the detected person overlaps with the restricted area and the detected person is not permitted to enter the restricted area. The monitoring information collecting system according to any one of claims 1 to 6, wherein the monitoring information collecting system is provided.

Images