JP2018093283A - Monitoring information gathering system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a monitoring information gathering system which records the movement trajectory of a human obtained from an image captured by a monitor camera, and can distinguish from other human subjected to face recognition, by performing face recognition of a human whose movement trajectory is obtained.SOLUTION: A camera 1 of a monitoring information gathering system includes: detection recognition means 3 having human detection means for detecting a human from an image captured by imaging means 2, trajectory analysis means for obtaining the movement trajectory by analyzing a human detected by the human detection means on the image, and face recognition means for recognizing the face by obtaining face data that can be distinguished from other face on the image of a human detected by the human detection means and whose movement trajectory is obtained by the trajectory analysis means; recording means 5 for associating each face data with ID and recording the face data and the movement trajectory corresponding to the face data; and communication means 6 performing data communication with other camera 1 or server.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a monitoring information collecting system for identifying a person photographed by a camera and obtaining a movement locus of the identified person.

A person who recognizes the face of a person photographed with a surveillance camera, for example, a person whose face is recognized from a database in which feature points of a person's face are associated with attribute information such as a person's name, address, and occupation A monitoring system has been developed to extract attribute information. In this case, the person registered in the database may be a wanted crime, for example.
If the feature points of the wanted offender's face are registered in the database, they can be used to find hidden wanted offenders.

  In addition, for example, if the faces of employees and other related persons are registered in the database, the persons registered in the database by face recognition when a person is photographed in a restricted area other than the related persons by the surveillance camera. It has been proposed to reduce false alarms by determining whether or not a person concerned is not alerted (for example, see Patent Document 1).

  In addition, using a laser radar, the movement trajectory of the moving object is obtained based on the distance to the moving object such as a person or a vehicle, and the subsequent movement trajectory is predicted. For example, two moving objects are located at the same position at the same time. It is proposed to judge that there is a possibility that a crime, an accident, or the like will occur when it is predicted to move to (see, for example, Patent Document 2).

JP2013-134675A JP 2014-194686 A

  By the way, even if the face of a person photographed by the surveillance camera is recognized, for example, if it does not match the face registered in the database, there is no way to use the data of the person whose face is recognized. In particular, it is not a facility that is used for the movement of criminals like airports, but the frequency of finding wanted criminals is not so high by face recognition with surveillance cameras in stores such as convenience stores. The role of is to prevent crimes by recording records when accidents or incidents occur, and by indicating that they are monitoring, and is not effectively used.

  In addition, when predicting an accident or the like by monitoring and predicting a moving locus using a laser radar, the laser radar cannot recognize the face of a moving person. Even if an automobile, bicycle, pedestrian, animal, or the like can be determined, the name of the person or the automobile cannot be specified, and the application is limited.

  In addition, the video of the surveillance camera is used to elucidate the truth after an accident or incident occurs, but if a long period of time has passed since the incident or accident occurred, the captured image data will be erased, There is a possibility that it cannot be used to elucidate the truth. In particular, when a plurality of surveillance cameras capture and store (record) moving images with a high number of pixels and a relatively high frame rate, the image data is stored and saved over a long period due to the storage capacity of the storage device. It ’s difficult.

  The present invention has been made in view of the above circumstances, and by obtaining and recording a movement trajectory of a person from an image taken by a surveillance camera, and performing face recognition on the person for whom the movement trajectory is required, It is an object of the present invention to provide a monitoring information collecting system that can be distinguished from other persons whose faces are recognized.

In order to solve the above problems, the monitoring information collection system of the present invention includes an imaging means for monitoring,
Human detection means for detecting a person from an image captured by the imaging means;
Trajectory analysis means for analyzing the person detected by the human detection means on the image to obtain a movement trajectory;
Face recognition means for recognizing a face by obtaining face data that can be distinguished from other faces from the face on the image of the person detected by the person detection means and whose movement locus is obtained by the locus analysis means;
Recording means for recording the face data and the movement trajectory corresponding to the face data in association with identification codes that enable identification of the face data of different people from each other;
A wrinkle determination unit that compares the face data obtained by the face recognition unit with the face data recorded in the recording unit and determines whether or not the face data satisfies a predetermined condition;
When the condition is satisfied, the movement trajectory corresponding to the face data obtained by the face recognition unit is converted to the identification code associated with the face data recorded in the recording unit that matches the face data. Record in association with the recording means,
When the condition is not satisfied, the face data obtained by the face recognition unit and the movement locus corresponding to the face data are associated with the identification code not yet associated with the face data to the recording unit. And a face data collecting means for recording.

  According to such a configuration, it is possible to identify a person who is within the shooting range of the image pickup means by each face data and record the movement trajectory of each person. That is, the face data that can identify a person and the movement trajectory of the person corresponding to the face data can be recorded in association with the identification code. The face data and movement trajectory can be used for verification when an accident or incident occurs. This face data and movement trajectory has less storage capacity than video data, and can be stored for a long time even if the storage capacity of the storage device is limited, which is effective for re-examination of past accidents and incidents. Can be used.

  Further, the movement trajectories of a plurality of persons identifiable by face data can be displayed simultaneously on one screen, for example, and more information can be obtained in a shorter time than when moving image data is viewed. In addition, it is possible to determine whether or not the same person has passed even if the dates are different, and it is possible to monitor the behavior of the same person over a long period. Further, for example, the movement trajectory of the person in the store can be used as a reference when considering the display of goods. The movement trajectory can be used when grasping the flow of people on a road such as a shopping street and determining signal control, traffic regulation, and the like.

  In addition, the identification of a person by wrinkle recognition in the present invention is to identify a wrinkle photographed by the imaging means and another face, in other words, the same person's face is determined to be the same face, and different person's faces are determined. It is not necessary to link personal information such as name and address to face data, so it is not necessary to associate personal information such as name and address with face data. For example, it is often used for the face of a customer who is often at a book store at a convenience store, It only needs to be distinguishable from the face of the customer. As a result, customer management can be performed in the store regardless of personal information, and services according to customer preferences can be provided. In addition, it is easy to manage not only patients but also visitors in nursing homes and hospitals.

  In the configuration of the present invention, the image processing apparatus includes a plurality of the imaging units, and the trajectory analysis unit is within a range obtained by combining the imaging ranges of the plurality of imaging units based on images captured by the plurality of imaging units. It is preferable to obtain the movement locus by

  According to such a configuration, the movement trajectory can be obtained even when the same person moves over the imaging ranges of the plurality of imaging means. In addition, even if a person temporarily deviates from the shooting range and can not track the destination, the movement locus can be obtained again when it becomes possible to recognize the eyelid again. Can be estimated. Accordingly, it is possible to obtain a movement trajectory without losing sight of a person's destination in a complicated store or facility.

In the configuration of the present invention, the apparatus includes attribute analysis means for obtaining attribute information that is an attribute of the face data from at least the image, the movement locus, or external information input from the outside,
The recording unit preferably records the attribute information obtained by the attribute analysis unit in association with the identification code.

  According to such a configuration, it is possible to add attribute information to the face data corresponding to each movement trajectory. For example, by analyzing the height and body shape of a person who is recognized by the image from the image taken by the imaging means. The attribute information may be used, or by adding attribute information indicating whether or not the person is a member such as a staff member to the face data, the person concerned can be excluded from the monitoring target. For example, even if the movement trajectory of a related person overlaps a place where entry is prohibited except for the related person, it can be ignored without issuing an alarm. Further, as information analyzed from the movement trajectory, for example, a specific human behavior pattern, for example, a behavior pattern such as passing a predetermined position at a predetermined time on a predetermined day of the week can be recorded as attribute information.

In the configuration of the present invention, the imaging means is disposed in a store where merchandise is displayed,
The attribute analysis unit receives, as the external information, arrangement information indicating the arrangement of the product, and based on the arrangement information and the movement trajectory, indicates information indicating a product approached by a person who has been recognized as wrinkles. It is preferable that

According to such a configuration, when the information on the arrangement of the product is input and stored as external information, the attribute analysis unit refers to the movement trajectory and the arrangement of the product with respect to the product that the customer has approached. It can be. In addition, you may analyze whether the customer stopped when approaching. In addition, if the product that the customer has approached cannot be identified as a kind, the classification of the same type of product that is placed together may be identified as the product. For example, cup noodles, stationery, etc. may be identified as the product. May be.
Therefore, it is possible to grasp the sales floor frequently used by each customer identified by the face data and use it for customer management, and easily consider the relationship between the customer's movement trajectory and the product arrangement when examining how to display the product. Can grasp. If an image taken at the time of attribute analysis is also used, it is possible to recognize an image of a product picked up by a customer or a product put in a shopping basket and use it as attribute information.

In the above configuration of the present invention, the imaging means is disposed in a store having a cash register capable of outputting accounting information indicating a product for which accounting is performed
The attribute analysis means corresponds to the movement trajectory that is close to the cash register when the accounting corresponding to the accounting information is performed in the cash register. The attribute information is preferably used.

  According to such a configuration, the movement trajectory can be associated with the actually purchased product, and the relationship between the movement trajectory and the purchased product can be analyzed. Furthermore, if the arrangement of the above-mentioned products is known in addition to the movement trajectory and the purchased products, it becomes possible to examine the method for displaying the products in more detail.

  In the configuration of the present invention, the recording means records the face data for a specific person so as to be identifiable as a specific person, and the face recognition means identifies the specific person from the face data. Is preferred.

  According to such a configuration, as a specific person, as described above, the attribute information indicates that the person is a related person, or the related person is excluded from the monitoring target, or the theft experienced person visits the shop as a specific person. It is possible to notify the customer of the customer or the customer as a specific person.

  According to the present invention, each person can be identified and the movement trajectory of each person can be recorded and used.

It is the schematic which shows the case where the monitoring information collection system of embodiment of this invention is installed in the store. It is a block diagram which shows a monitoring information collection system. It is a block diagram which shows the detection / recognition means of the monitoring information collection system. It is a figure explaining the imaging | photography range of the camera of a monitoring information collection system. It is a figure for demonstrating the monitoring information collected by a monitoring information collection system similarly. 3 is a flowchart for explaining a monitoring information collecting method by the monitoring information collecting system.

Embodiments of the present invention will be described below.
As shown in FIG. 1, the monitoring information collection system of the present embodiment has a monitoring camera 1 installed in a monitoring area. In the present embodiment, the camera 1 does not simply shoot the monitoring area, but performs human detection, analysis of the detected human movement trajectory, human face recognition that is detected and analyzed for the movement trajectory, etc. The monitoring information is recorded in association with an ID as an identification code. It is not necessary to provide all of these functions in the camera 1, and functions other than shooting and image output may be provided on the external device side such as a server to which an image output from the camera 1 is input.

  In the present embodiment, the monitoring area where the camera 1 is installed is, for example, a store 50 such as a convenience store. The store 50 has a customer entrance / exit 56, and a customer 40 who is a person enters the store from the customer entrance / exit 56. In the store 50, a cashier counter 52 having a register 51 at one wall is disposed slightly apart from the wall and in parallel with the wall surface. This is a no-entry area 53 in which customers 40 (persons) other than the related person 41 such as a store clerk are prohibited.

  In the store 50, a plurality of merchandise shelves 54 are arranged vertically and horizontally in a matrix at locations away from the wall. Further, a product shelf 55 is arranged on the wall other than the wall where the above-described register counter 52 is located. In the store, three cameras 1 of the present embodiment are arranged for monitoring. In this example, one unit is arranged on the wall on the cashier counter 52 side, and two units are arranged on the wall on the opposite side of the cashier counter 52. The arrangement of the camera 1 can be arbitrarily set, and may be one or more, but it is preferable that as many parts of the store 50 as possible can be photographed.

  Each camera 1 constitutes the monitoring information collection system of the present embodiment. As shown in FIG. 2, for example, the camera 1 is photographed by an imaging means 2 having a lens, a solid-state imaging device, and the like, and the imaging means 2. A detection / recognition means 3 (detection recognition means 3) for recognizing the person's face from which the person is detected from the detected image, the movement locus of the detected person is analyzed, and the movement locus is analyzed; A recording unit 5 that records captured image data and monitoring information obtained by detection and recognition by the detection / recognition unit 3 and a communication unit 6 that communicates with an external device via a network 7 (communication line). And control means 4 for controlling the image pickup means 2, detection / recognition means 3, recording means 5 and communication means 6.

  The imaging unit 2 and the communication unit 6 are not provided in the camera 1 with all the above-described components provided in the camera 1, and other components are arranged in a computer system such as a server outside the camera 1 or other It is good also as what is provided in an apparatus. When a plurality of cameras 1 are arranged in a shooting area such as a store 50, all of the above-described components are arranged in one camera 1, and the remaining cameras 1 basically have an imaging means 2 and Only the communication means 6 may be provided. That is, when there are a plurality of cameras 1, a function as a computer system may be added to one camera 1, and the remaining cameras 1 may be used as a general monitoring camera that outputs images. In any case, the monitoring information collection system can be composed of a plurality of cameras 1.

  The imaging unit 2 has a configuration similar to that of, for example, a general surveillance camera. For example, the imaging unit 2 captures an imaging range corresponding to an angle of view set according to the orientation of the camera 1 and converts the captured image into a moving image. To send as. In this embodiment, a stereo camera capable of calculating the distance / size from the parallax is used as the imaging unit 2. By using this stereo camera, the shape of the subject can be calculated, and when performing various recognitions from the feature points on the image, the feature points can be obtained from the three-dimensional shape, enabling more accurate recognition and facilitating human detection. It is possible to perform face recognition in more detail.

  Further, the change in the distance to the detected person can be calculated from the parallax, and the movement trajectory of the person can be easily analyzed based on the distance from the camera 1 to the detected person. Further, height, body shape, clothes, and the like are required as the attribute information of the detected person based on the stereo camera image, and the following estimation can be performed based on them. That is, the attribute information may be obtained by estimating whether a person photographed with a stereo camera is an adult or a child, whether he / she is fat or thin (body shape), male or female, and the like. The behavior of the customer 40 in the store 50 may be image-recognized to detect criminal acts such as so-called shoplifting. For example, a product taken in the hand is recognized and the product is returned to its original state or shopping It may be determined whether it can be put in a basket or other than that, and the person concerned 41 is notified in other cases. Moreover, it is good also as a structure which judges from the movement of the person on a moving image whether shoplifting was performed by machine learning.

  Further, the camera 1 may be provided with a voice input means for inputting voice in addition to the imaging means 2, and the voice in the monitoring area may be digitally recordable or voice-recognized. In this case, it may be determined whether or not the order in the monitoring area is disturbed.

  Further, the imaging unit 2 is not limited to a stereo camera, and may be a monocular camera. In this case, since the distance cannot be calculated from the parallax, the height, size, and distance compared to the person such as the position of the product shelves 54 and 55 of the store 50 as the background of the person and the height of each shelf board are compared. It is also possible to store the position and shape of the constituent elements that can calculate the background, and to detect the person, analyze the movement trajectory of the person, and determine the height, body shape, etc. of the person based on this. For example, a distance image sensor may be used as the imaging unit 2. As the distance image sensor, for example, TOF (Time Of Flight) can be used. The TOF measures the distance from the time taken for the projected laser to reciprocate to the target.

  If the distance image sensor is used as the imaging means 2, the distance to the subject can be obtained, and the face can be recognized by increasing the resolution, and the distance image sensor can perform human detection, movement trajectory, and face recognition. Become. When the resolution of the distance image is not sufficient and the accuracy of face recognition does not increase, the face recognition may be performed by combining the distance image with an image from a monocular camera.

  The detection / recognition means 3 is a computer system including an arithmetic processing unit and a memory, and performs image recognition. That is, image analysis and image recognition such as human detection, human tracking (movement trajectory), face recognition and the like are performed by image recognition from an image taken by the imaging means 2. In human detection, it is determined by image recognition whether the subject on the image is a person. Here, in the store 50, a person enters from the entrance / exit 56, and a person entering the store is detected. In the human detection, the subject moving from the entrance 56 to the store is a human candidate.

  In human detection, it is determined whether or not a person is present. In face recognition, facial feature points, feature amounts, and the like are obtained by face image recognition and stored as face data. When comparing a plurality of face data, for example, the face data of the same person has many portions where the feature points and feature amounts approximate, and the face data of different people does not approximate as in the case of the same person. Therefore, it is possible to determine whether the two face data compared are those of the same person or different persons according to a known comparison condition, such as conditions such as the number of feature points approximated when comparing face data and the number of feature quantities. it can. That is, face image matching can be performed. Actually, for example, the above-described human detection, human tracking, and face recognition can be performed by using OpenCV (Open Computer Vision Library), which is a library of functions related to image recognition, and recognition by deep learning. . The latest OpenCV library also includes a machine learning function, for example, a deep learning module.

  Here, in the present embodiment, not only the person is tracked on the image, but also the position of the person tracked on the image in the store 50 is determined based on the distance determined by the parallax of the stereo image. The change of this position over time is obtained as a trajectory. Note that the determination of whether or not a person is in human detection and face recognition are also performed based on a three-dimensional image obtained from a stereo image. Therefore, feature points are also obtained on a three-dimensional stereoscopic image. However, as described above, when a monocular camera is used in the imaging unit 2, the positional relationship between the fixed structure as a background and the subject detected and tracked is estimated, and the position in the person's store 50 is estimated. May be determined.

  Further, the position of a person in the store 50 may be obtained based on the distance measured by a distance image sensor such as TOF, and the movement of this position may be recorded as a trajectory. In this case, it is possible to perform person detection, person tracking, and face recognition based on the distance image output from the distance image sensor, and in this case as well, a three-dimensional image is recognized. Further, the distance image of the distance image sensor and the image of the monocular camera may be combined to perform image recognition, and the distance to each pixel by the distance image and the image of the monocular camera having a higher resolution than the distance image are used. Image recognition.

  From the above, the detection / recognition means 3, as shown in FIG. 3, detects the person from the image, tracks the person detected from the image on the image, and analyzes the position of the person. A trajectory analysis means 32 for obtaining a movement trajectory of the person and a face recognition means 33 for recognizing the detected human face. These human detection means 31, trajectory analysis means 32, and face recognition means 33 are connected to the recording means 5 via the control means 4, and the human ID detected by the human detection means 31 is obtained by the trajectory analysis means 32. The obtained movement trajectory and the face data obtained by the face recognition means 33 are stored.

  In extracting feature points such as human eyes, nose, mouth, chin, forehead, and hair, the size of the face in the image becomes a problem, and from the size of the person in the image that can be detected as a person, There is a high possibility that the size of the face in the image capable of face recognition is larger. That is, within the shooting range of the camera 1, the distance from the camera 1 capable of recognizing the face to the person is shorter than the distance from the camera 1 capable of detecting the person. That is, as shown in FIG. 4, the range close to the camera 1 in the shooting range of the camera 1 is the face recognition area 10, and the range wider than the face recognition area 10 is the human detection area 11.

  The human detection area 11 includes a face recognition area 10. In this case, for example, in a store 50 such as a convenience store, if the entire range in the store 50 is to be stored in the face recognition area 10 of the camera 1, the number of cameras 1 is increased or the resolution of the camera 1 is increased. When it is necessary to increase the cost and the cost increases, only a part of the area in the store 50 may be set as the face recognition area 10 as shown in FIG. For example, as shown in FIG. 1, when three cameras 1 are installed so that the entire range in the store is within the shooting range of the camera and the human detection area 11 of the camera 1 is recognized. The area 10 may cover only a part of the store 50.

  In the present embodiment, in the store 50 shown in FIG. 1, almost the entire area of the store 50 falls within the human detection area 11 (shooting range) of the camera 1 on the lower right. The part which is not image | photographed also in the person detection area 11 is disturbed by. Cameras 1 are arranged at the left and right ends of the store 50 so as to cover the portions that cannot be photographed. However, the face recognition area 10 covers only a part of the store 50.

  Even if the face recognition area 10 of the camera 1 arranged at the left end of the store 50 includes the entrance / exit 56 of the store 50 and only a part of the store 50 is the face recognition area 10, the entrance / exit The faces of all the customers 40 entering the store from 56 can be recognized, and the customers 40 are recognized at the time of entering the store, and for example, it is determined whether the customer 40 is an existing customer 40 or a new customer 40. Yes. Further, the camera 1 on the right side in FIG. 1 can photograph the registration counter 52 on the left side in the drawing and the entry prohibition area 53 in the registration counter 52, and the customer 40 who is not the related person 41 can enter the entry prohibition area 53. Can be monitored as well as unauthorized access to the cash register.

  In this embodiment, it is only necessary to be able to identify each customer.For example, facial recognition is used to determine whether the customer who visited yesterday morning and the customer who visited this morning are the same person or different people, There is no need to associate face data with personal information such as name and address. However, as will be described later, accounting data such as purchased items, purchase amount, purchase date and time, cash payment prepaid card payment, etc., output from the register 51 may be linked. That is, each customer identified by an ID is identified as part of customer management, and actions including purchase of each customer are recorded.

  The control unit 4 may control the start and end of shooting by the imaging unit 2 and may control focus adjustment, exposure correction, and the like. Further, the control unit 4 records the data of the image captured by the imaging unit 2 in the recording unit 5. The control unit 4 also includes face data recognized by the detection / recognition unit 3, movement trajectory data for each face data, body information such as height and body shape recognized from the image, Estimated information such as age, sex, race, etc. estimated from information such as clothes read from images and makeup recognized during face recognition is attached to face data and movement trajectory data as attached information that is part of monitoring information Then, it is recorded in the recording means 5 in association with the ID.

  The communication means 6 communicates with other cameras 1 via the network 7 to supply information and take over the movement locus of a person. Inheritance of the movement trajectory basically takes over the movement trajectory of the person whose position is the same in the portion where the shooting areas overlap, but when each camera 1 recognizes the face, each camera of the same person is used. The movement trajectories analyzed in 1 may be joined based on satisfying a predetermined matching condition of face data obtained by face recognition.

  Further, as the information collecting device in the photographing information collecting system, each camera 1 includes the above-described imaging means 2, detection / recognition means 3, recording means 5, communication means 6, and control means 4, but they are collected. An apparatus for viewing information may be provided. For example, a terminal including an input device such as a display, a keyboard, a pointing device, and an arithmetic processing device is connected to the communication means 6 of each camera 1 via the network 7. May be. With this terminal, it is possible to browse and manage data such as images, movement trajectories, and attached information stored in the recording means 5 of the camera 1.

  The image of the camera 1 does not need to be constantly monitored. For example, when a person other than the related person 41 enters the restricted entry area, the camera 1 automatically sounds based on the image recognition. It may be configured to process. Further, it is only necessary that the data stored in the recording means 5 is output to an external computer system so that customer management, product display methods, and the like can be examined by the computer system.

  The communication means 6 of the camera 1 is connected to the cash register 51 via the network 7. The cash register 51 is, for example, a POS cash register for a POS system (point-of-sale information management system). The product name, price, purchase date / time, payment method (cash, card payment, etc.) The point card score is output. In the present embodiment, the above-described various information is transmitted from the cash register 51 to the camera 1.

  When the information corresponding to the purchase of the product is input from the cash register 51, the control means 4 of the camera 1 receives the customer 40 closest to the cash register outside the entry prohibition area 53 of the register counter 52 when the information is input. The information relating to the purchased product transmitted from the cash register 51 is recorded in the recording means 5 in association with the ID. In the recording means 5, the information transmitted from the cash register 51 is recorded as accompanying information in association with the ID.

  As a result, the purchased product of the customer 40 can be understood, and the purchased product and the movement trajectory can be understood, so that it can be used as a basic material when examining the display method of the product. That is, it is possible to analyze which product has passed before or after purchasing the product, and to obtain the relationship between the display of the product and the movement of the customer 40. In addition, analyze the information such as what kind of products the customer whose movement trajectory stopped in front of the magazine department then purchased, and whether the customer who stopped in front of the lunch box department purchased lunch Accordingly, it is possible to obtain a product display method, an optimal replacement time for a product such as a lunch box with a short display time, a suitable display method for a beverage-type product and a lunch-type product, and the like.

  Next, data stored in association with ID (a) in the recording unit 5 will be described. Data stored in association with ID (a) is face data (b), profile (c), purchase history (d), and movement locus (e). As described above, the face data (b) is feature points, feature amounts, and the like, and is data that makes it possible to identify the same person or different persons when there are two face images. The profile (c) is data obtained by analysis (recognition) of an image photographed by the camera 1 and data input from the outside, and is associated with the face data (b) and the movement trajectory (e). Information.

  The profile (c) includes, for example, data based on image recognition such as height, body shape, adult / dwarf, gender, clothes (color, etc.), and a later-mentioned related person (entry / exit permission to enter / exit area 53) Data entered from the outside, such as whether or not a person is). The purchase history (d) is data input from the cash register 51, and includes the product name (product ID) of the purchased product, the quantity, the purchase price, the type of the purchased product, and the like. The movement trajectory (e) basically has position and date / time data in the store 50, and the positions are connected in time order. In the profile (c), information such as a person requiring attention may be registered.

Next, a monitoring information collecting method and a monitoring method by such a monitoring information collecting system will be described with reference to the flowchart of FIG. The processing shown in the flowchart is basically processing in the control means 4.
First, when the customer 40 enters the store 50 from the entrance / exit 56, the customer 40 who has entered the store is photographed by the camera 1, and the customer 40 entering the store from the entrance / exit 56 in the photographed image is detected / recognized by the camera 1. 3 is detected as a person (step S1). The detection recognition means 3 tracks the person detected on the image, analyzes the movement, and detects the movement trajectory until the exit from the store 50, for example, corresponding to the passage of time until the detected person is no longer detected. (Step S2).

  The detection / recognition means detects and recognizes the face of the person who is detected and tracked on the image in order to obtain the movement trajectory, and obtains facial feature points and feature quantities, which are recorded in the recording means 5 as face data. (Step S3). Note that face recognition is performed by, for example, being photographed by the camera 1 when the customer 40 enters the store through the entrance / exit 56. If the face recognition fails here, the next step is performed while obtaining the movement trajectory. Face recognition is performed when the camera enters the face recognition area 10 of any camera 1 and faces the camera 1 side.

  When face recognition is performed, the face data that has already been recognized and stored in association with the ID as described later is compared with the face data that has been acquired through face recognition this time. It is determined whether or not the face data is recorded, that is, whether or not the recognized face data is registered face data (step S4). In this case, when there are few feature points or feature amount matching points of the face data and the matching condition is not satisfied, an unused new ID is registered as a new customer 40 (step S5). Information such as face data and accompanying information associated with the new ID is recognized as a profile (step S6).

  On the other hand, when the newly recognized face data is compared with the face data already recorded in association with the ID, it is determined that there are many feature points and feature quantity matching points and the matching conditions are satisfied. That is, when the recognized face data has already been registered, the face data and accompanying information described above are recognized as a registered ID profile (step S7). Then, the profile recognized in association with the registered ID is updated by the recording unit 5 as the profile of the registered ID (step S8).

As described above, the new profile is stored together with the new ID or the registered ID, and the data of the movement trajectory is recorded in the recording unit 5 as a trajectory profile in association with any of these IDs (step S9). The movement trajectory is basically detected until the customer 40 goes out of the store, and the trajectory profile is periodically updated.
Next, it is determined whether or not the customer 40 is still in the store 50 (in the area) (step S10). When it is determined that the customer 40 is not in the store 50 and is out of the area, the detection of the movement trajectory is completed, and the recording means 5 records information (file or the like) associated with the ID including the information of the movement trajectory. ) Is closed (step S11).

  If there is still a person detected in the store 50 (in the (monitoring) area), it is determined whether the current movement trajectory or the movement trajectory within a predetermined time overlaps the restricted entry area 53 (prohibited area). (Step S12). In other words, it is determined whether or not the detected customer 40 has entered the inaccessible area 53. Next, referring to the profile recorded in the recording means 5 in association with the ID of the invading customer 40, the person in the store 50 such as a store clerk is allowed to enter the off-limits area 53 according to the profile of the ID. It is determined whether or not the person has an ID (step S13).

  In the profile registered for each ID of the recording means 5, in addition to the items recognized from the images and the items estimated from the images and the recognized items, the attached information input from the outside can be stored. In this case, as the attached information (profile), it is recorded in the profile of the store-related person 41 that, for example, the entry / exit permission for the entry prohibition area 53 is granted.

  Therefore, when the face data of the related person 41 is recognized and the ID is read out, there is permission to enter and leave the off-limit area as auxiliary information associated with the ID. The alarm is not sounded even if it overlaps with 53. The store 50 is provided with an alarm device for issuing an alarm, and the control means 4 of the camera 1 outputs a command indicating the start and end of the alarm (alarm) to the alarm device via the communication means 6 and the network. It is possible.

  Therefore, when the movement locus is associated with the ID of the person who overlaps with the entry prohibition area 53 and the entry / exit permission to the entry prohibition area 53 is not permitted, the control means 4 sets an alarm start command in the alarm device (step S14). In the next processing, an alarm start command is output to the alarm device via the communication means 6. On the other hand, when entry / exit permission to the entry prohibition area 53 is recorded in the profile in association with the ID, the control means 4 does not set an alarm command.

  The processing by the control means 4 described above is actually real-time processing, and the detection / recognition means 3 can simultaneously detect the movement trajectories of a plurality of people. The movement trajectory of the person can be output. On the other hand, the control means 4 updates the movement trajectory and profile in real time.

  In addition, when there are a plurality of cameras 1, the cameras 1 are linked to each other, and it is determined based on the movement trajectory and face data whether the people photographed simultaneously by the cameras 1 are the same or different. In addition, when a person with the same ID is photographed by a plurality of cameras 1, the cameras 1 cooperate with each other so that one movement trajectory is obtained for one ID.

  Therefore, when a person sequentially moves the imaging ranges of the plurality of cameras 1 that do not overlap each other, the movement trajectories analyzed by the cameras 1 are connected together. At this time, in the case of a person whose face is recognized by one camera 1 and whose face is not recognized by the other camera 1, the ID is determined when the face is recognized, and the movement trajectory of the camera 1 whose face is not recognized is also determined. Are connected to the movement locus detected by the camera 1 that has recognized the face, the same ID is attached to the movement locus, and the movement locus detected by each camera 1 is connected.

  According to the monitoring information collection system of the present embodiment, the person in the monitoring area is identified by the person detection and the face recognition, and each identified person is assigned an ID. Even if personal information such as name is not known, it can be managed by ID. In other words, it is possible to determine whether a person in the monitoring area has entered the monitoring area for the first time, a person who has previously entered the monitoring area, or a person who frequently enters the monitoring area. It becomes possible to notify the related person 41 that the monitoring area has been entered or that a person who has always come into the monitoring area.

When the monitoring area is the store 50, it is possible to notify the store clerk that the regular customer 40 has arrived or the first customer 40 has come, and it is easy to deal with the customer. In addition, by knowing the movement of each customer 40 in the store 50 and the products purchased by the customer 40, it is possible to refer to the layout in the store 50 such as the arrangement of each product and the arrangement of each product shelf. It is.
In addition, as the data for reviewing the layout in the store 50, in the state where the arrangement information indicating the arrangement of the products (including the arrangement of the product shelves) is recorded as the external information input to the recording unit 5 from the outside, The movement trajectory obtained as described above and the arrangement of the products may be analyzed, and the product that the customer has approached, the product that the customer has approached and stopped, and the like may be recorded as attribute information. The product in this case may be a general product type, for example, a product name that can be identified by the manufacturer or the like, but if it cannot be specified to that extent, a notebook or stationery may be used as product information.
Furthermore, by image recognition, the information on the product at the position where the customer 40 of the product shelves 54 and 55 stops, the image of the product when the customer 40 picks up the product, and the position of the product before being picked up by the hand. You may record the information of the goods which the customer took.

  The data recorded in these recording means 5 is used for customer management as data for each customer 40. For example, the repeat rate at which the customer revisits the store, the repeat rate at which the same product is purchased, the rate at which a new product is purchased, and the same movement It is required to move along the trajectory. In addition, the data of all the customers 40 of the store 50 or a plurality of stores 50 are collected and the composition ratio of the types of customers 40 (adult / dwarf, sex, etc.) that are image-recognized, that is, there are many dwarfs, It can be obtained that there are many women, the correlation between the movement trajectory of each customer 40 and the purchased product, the correlation between the movement trajectory of each customer 40 and the profile (attribute) of each customer 40, and the like. That is, the movement trajectory of the customer 40 can be used in collecting information for marketing or the like.

  In addition, the monitoring information collection system can search for an image of a criminal related person after a crime occurs as a general monitoring device. For example, when there is an image such as a picture of a criminal related to a victim or a perpetrator In addition, it is possible to determine whether or not the same person is recorded in the recording means 5 of the monitoring information collection system by taking the photograph with the camera 1 and recognizing the face. In this case, it is possible to determine whether or not the same person is present in the recorded face data even if there is no captured moving image data. Therefore, even if video data with a large amount of data is not stored for a long period of time, if face data etc. are stored for a long period of time, it is possible to know whether or not a criminal has arrived, and the movement trajectory and product purchased It becomes possible.

  In addition, the monitoring area in the monitoring information collection system of this Embodiment is not restricted to the store 50, Various facilities, such as a housing complex, a tenant building, a factory, a museum, a museum, an office building, may be used as a monitoring area. it can. Even when these facilities are set as monitoring areas, a person can be detected and identified to detect a movement locus, so that a suspicious person can be found from the movement locus. For example, in an apartment house, it is normal to move from the entrance to your room on the shortest course, and the person who draws the other movement trajectory may be watched.

  Also, in facilities other than apartment buildings, there is a high possibility that it is possible to determine whether or not a person is a suspicious person based on the movement trajectory, especially when a person other than the related person enters a place where only a related person cannot enter. Can be determined. Further, as described above, according to the monitoring information collection system of the present embodiment, when an image such as a facial photograph is obtained, it is possible to automatically determine whether or not the person has entered the monitoring area, and the movement locus thereof. The profile can be obtained, and it can be determined whether or not there is a criminal related person more easily than when a human analyzes the recorded image of the camera 1. In addition, it is possible to easily find a suspicious person by registering a related person in advance in a facility where a person other than the related person rarely enters instead of a facility such as the store 50 where the customer 40 enters and exits. In such a facility, it is also possible to determine a person who is recognized for the first time other than a person who has already been recognized as a suspicious person.

DESCRIPTION OF SYMBOLS 1 Camera 2 Imaging means 3 Detection / recognition means 31 Human detection means 32 Trajectory analysis means 33 Face recognition means 4 Control means (Face data collection means, wrinkle determination means, attribute analysis means)
5 Recording means 6 Communication means

Claims (6)

Imaging means for monitoring;
Human detection means for detecting a person from an image captured by the imaging means;
Trajectory analysis means for analyzing the person detected by the human detection means on the image to obtain a movement trajectory;
Face recognition means for recognizing a face by obtaining face data that can be distinguished from other faces from the face on the image of the person detected by the person detection means and whose movement locus is obtained by the locus analysis means;
Recording means for recording the face data and the movement trajectory corresponding to the face data in association with identification codes that enable identification of the face data of different people from each other;
A wrinkle determination unit that compares the face data obtained by the face recognition unit with the face data recorded in the recording unit and determines whether or not the face data satisfies a predetermined condition;
When the condition is satisfied, the movement trajectory corresponding to the face data obtained by the face recognition unit is converted to the identification code associated with the face data recorded in the recording unit that matches the face data. Record in association with the recording means,
When the condition is not satisfied, the face data obtained by the face recognition unit and the movement locus corresponding to the face data are associated with the identification code not yet associated with the face data to the recording unit. A monitoring information collecting system comprising: face data collecting means for recording.   A plurality of the imaging means, and the trajectory analysis means obtains a movement trajectory within a range obtained by combining the imaging ranges of the plurality of imaging means based on images respectively captured by the plurality of imaging means. The monitoring information collection system according to claim 1, wherein Attribute analysis means for obtaining attribute information as an attribute of the face data from at least the image, the movement locus, or external information input from the outside,
The monitoring information collection system according to claim 1, wherein the recording unit records the attribute information obtained by the attribute analysis unit in association with the identification code. The imaging means is disposed in a store where merchandise is displayed,
The attribute analysis unit receives, as the external information, arrangement information indicating the arrangement of the product, and based on the arrangement information and the movement trajectory, indicates information indicating a product approached by a person who has been recognized as wrinkles. The monitoring information collecting system according to claim 3, wherein: The imaging means is disposed in a store having a cash register capable of outputting accounting information indicating a product for which accounting has been performed,
The attribute analysis means corresponds to the movement trajectory that is close to the cash register when the accounting corresponding to the accounting information is performed in the cash register. The monitoring information collection system according to claim 3 or 4, wherein the attribute information is used. The face data for a specific person is recorded in the recording means so as to be identifiable as a specific person, and the face recognition means identifies the specific person from the face data. The monitoring information collection system according to any one of claims 5 to 6.

Images