JP5523900B2 - Person search device, person search method, and person search program - Google Patents

Description

  The present invention relates to a person search device, a person search method, and a person search program, and more particularly, a person search device and a person search method for efficiently and accurately searching a person with a simple operation on a captured image. And a person search program.

  2. Description of the Related Art Conventionally, there has been a technique for performing image processing or the like on a video taken by an imaging means such as a camera and automatically detecting a person included in an image constituting the video. Among them, a technique for detecting a human face from an image has been rapidly developed.For example, in order to improve face discrimination accuracy, a skin color region is extracted from an input image, and based on the skin color region, the top of the head, There is an image processing apparatus that detects the positions of facial feature points such as eyes and mouth and determines whether or not the skin color region is a face from the detection result (see, for example, Patent Document 1).

  In addition, such face detection technology is applied to images obtained by surveillance cameras installed in guarded facilities such as banks, department stores, and convenience stores. It is used to detect crimes and prevent crimes.

  Furthermore, there is a method for searching for a face image having a specified feature by accumulating detected face information together with feature information and specifying a facial feature of a person to be searched as a search condition (for example, (See Patent Document 2).

JP 2004-5384 A JP 2006-318375 A

  By the way, it is preferable that the person detected by the above-described conventional method accumulates various kinds of information that can be read from the camera so that the highly accurate search can be efficiently performed in the image search process. However, there is a limit to the information that can be extracted from only the face information of a person. For example, when determining whether or not the person is a specific person such as a suspicious person, In the case of detection, more detailed person information is required. In addition, since a huge amount is targeted for search, the searcher can efficiently and accurately perform a person search with simple operations for the search conditions and data storage and search, etc. Must be built to output.

  The present invention has been made in view of the above problems, and a person search device, a person search method, and a person search method for efficiently and accurately performing a person search on a captured image with a simple operation, and The purpose is to provide a person search program.

  In order to solve the above problems, the present invention employs means for solving the problems having the following characteristics.

In the present invention, in a person search device that extracts information related to a person included in a video captured by an imaging unit and displays the information on a screen, the storage for storing personal feature information for each time-series image included in the video Means, search means for searching for a person based on conditions preset for the storage means, and information corresponding to the search result obtained by the search means is extracted by the storage means, and the extracted person A screen generating means for generating a screen for displaying a personal feature or action set in advance corresponding to the information, or a patrol position information of a security robot provided with the imaging means, and a screen generated by the screen generating means. and displays, and output means for outputting information displayed on the screen to an external device, with respect to a person included in the image frames constituting the video, the face area Face area detecting means for detecting, human body area detecting means for detecting a human body area for a person included in an image frame constituting the video, detection results obtained by the face area detecting means, and the human body area detecting Identifying means for identifying persons of a plurality of image frames having different times using detection results obtained by the means, tracking means for tracking the movement of the person obtained by the identifying means, and tracking results obtained by the tracking means A suspicious person detecting means for detecting a suspicious person from the image, a person information integrating means for integrating the feature information of the person identified by the identifying means, and an integration result obtained by the person information integrating means to form an image constituting the video Frame information generating means for generating each frame, and the frame information generating means is included in the video identified by the identifying means. For each person, all feature data obtained from the image frame is discretized, the number of appearances is counted, and person feature information obtained by normalizing the discretized data by the number of each image frame is generated. The person characteristic information obtained by the frame information generating means is stored . Thereby, a highly accurate person search can be efficiently performed with a simple operation on the captured image.

  In the present invention, the human body region detecting means extracts color information in a predetermined range of the detected human body, converts the extracted color information into a color space set in advance based on human visual characteristics, and the storage means Stores the color information converted into the color space.

With this embodiment, the searching means, search the search condition, a search request input by the searcher, to convert the feature amount corresponding to the contents of the characteristic of the person, the search request, wherein the maximum value of the normalized histogram And searching for the person information stored by the storage means according to the search request feature quantity, calculating a distance from the feature quantity given to the corresponding person, and scoring the correct answer rate. .

Furthermore, in the present invention, in the person search method for extracting information about a person included in a video photographed by the imaging means and displaying it on the screen, the person feature information is stored in each time-series image included in the video. The storage means extracts the information corresponding to the search result obtained by the search procedure obtained by the storage procedure, the search procedure for searching for a person based on the condition preset for the storage means, and the search procedure. A screen generation procedure for generating a screen for displaying a personal feature or action set in advance corresponding to the extracted person, or a patrol position information of the guard robot provided with the imaging means, and the screen generation procedure and displays the the current screen, and an output procedure to output the information displayed on the screen to an external device, the person included in the image frames constituting the video Then, a face area detection procedure for detecting a face area, a human body area detection procedure for detecting a human body area for a person included in an image frame constituting the video, and a detection result obtained by the face area detection procedure An identification procedure for identifying a person in a plurality of image frames at different times using the detection result obtained by the human body region detection procedure, a tracking procedure for tracking the movement of the person obtained by the identification procedure, and the tracking A suspicious person detection procedure for detecting a suspicious person from a tracking result obtained by the procedure, a person information integration procedure for integrating feature information of a person identified by the identification procedure, and an integration result obtained by the person information integration procedure, A frame information generation procedure that is generated for each image frame constituting the video, and the frame information generation procedure is performed before the identification procedure is identified. For each person included in the video, discretize all feature data obtained from the image frame and count the number of appearances, generate human feature information obtained by normalizing the discretized data by the number of image frames, The storage means stores the person characteristic information obtained by the frame information generation procedure . Thereby, a highly accurate person search can be efficiently performed with a simple operation on the captured image.

  Also, in the present invention, the human body region detection procedure extracts color information of a predetermined range of the detected human body, converts the extracted color information into a color space set in advance based on human visual characteristics, and stores the storage procedure. Stores the color information converted into the color space.

With this embodiment, the search procedure, search the search condition, a search request input by the searcher, to convert the feature amount corresponding to the contents of the characteristic of the person, the search request, wherein the maximum value of the normalized histogram And searching for the person information stored by the storage means according to the search request feature quantity, calculating a distance from the feature quantity given to the corresponding person, and scoring the correct answer rate. .

Further, according to the present invention, in a person search program in a person search apparatus that extracts information related to a person included in a video photographed by an imaging unit and displays the information on a screen, a computer is used for each time-series image included in the video. Storage means for storing person feature information, search means for searching for a person based on conditions preset for the storage means, and information corresponding to the search results obtained by the search means by the storage means A screen generation unit that generates a screen for displaying a personal feature or action set in advance corresponding to the extracted person, or a patrol position information of a security robot provided with the imaging unit; and displays the generated screen, output means for outputting the information displayed on the screen to an external device, images forming the image A face area detecting means for detecting a face area for a person included in a frame, a human body area detecting means for detecting a human body area for a person included in an image frame constituting the video, and the face area detecting means. Identification means for identifying a person in a plurality of image frames having different times using the detection result obtained and the detection result obtained by the human body region detection means; a tracking means for tracking the movement of the person obtained by the identification means; Suspicious person detecting means for detecting a suspicious person from the tracking result obtained by the tracking means, personal information integrating means for integrating the feature information of the person identified by the identifying means, and the integrated result obtained by the personal information integrating means and to function as a frame information generating means for generating for each image frames constituting the video, the frame information generating means, the same For each person included in the video identified by the means, discretize all feature data obtained from the image frame and count the number of appearances, and normalize the discretized data by the number of each image frame The information is generated, and the storage means stores the person characteristic information obtained by the frame information generation means . Thereby, a highly accurate person search can be efficiently performed with a simple operation on the captured image. Further, by installing the program, the person search process according to the present invention can be easily realized by a general-purpose personal computer or the like.

  In the present invention, the human body region detecting means extracts color information in a predetermined range of the detected human body, converts the extracted color information into a color space set in advance based on human visual characteristics, and the storage means Stores the color information converted into the color space.

With this embodiment, the searching means, search the search condition, a search request input by the searcher, to convert the feature amount corresponding to the contents of the characteristic of the person, the search request, wherein the maximum value of the normalized histogram And searching for the person information stored by the storage means according to the search request feature quantity, calculating a distance from the feature quantity given to the corresponding person, and scoring the correct answer rate. .

  According to the present invention, a highly accurate person search can be efficiently performed with respect to a captured image with a simple operation.

It is a figure which shows an example of schematic structure of the person search device in this embodiment. It is a figure of an example for demonstrating the additional function in a face area | region detection means and a human body area | region detection means. It is a figure which shows an example of the frame information in this embodiment. It is a figure for demonstrating the outline | summary until it acquires the person characteristic for a search in this embodiment. It is a figure of an example for demonstrating the search method in this embodiment. It is a figure which shows the example of a screen transition in this embodiment. It is a figure which shows an example of the initial screen in this embodiment. It is a figure which shows the date specification screen in this embodiment. It is a figure which shows the feature and action condition designation | designated screen in this embodiment. It is a figure which shows an example of a color map. It is a figure which shows the search result display screen in this embodiment. It is a figure for demonstrating a search result display area concretely. It is a figure which shows the video reproduction screen in this embodiment. It is a figure which shows the shared screen in this embodiment. It is a figure which shows an example of the hardware constitutions which can implement | achieve the person search process in this embodiment. It is a flowchart which shows an example of the accumulation | storage process procedure in this embodiment. It is a flowchart which shows an example of the search processing procedure in this embodiment.

<About the present invention>
The present invention is an image of a time when a person close to a person image intended by a searcher is photographed from a huge amount of images stored in advance by a photographing means such as a camera or a real-time video from the photographing means. Can be easily searched and confirmed. Specifically, when searching for a video, the video is searched based on the characteristics and actions of the person. For example, a search is performed by combining various types of personal features and actions that could not be combined so far, such as numerical values such as height, information such as the presence or absence of sunglasses, and the color of clothes, which depend on subjectivity. In addition, as an imaging means in this invention, there exist a fixed-type thing installed in the wall or ceiling etc. which were preset, for example, and a movable type mounted in the security robot etc. An appropriate person detection method corresponding to any of them is provided.

  Hereinafter, embodiments in which a person search device, a person search method, and a person search program according to the present invention are suitably implemented will be described with reference to the drawings.

<Schematic configuration example of person search device>
FIG. 1 is a diagram illustrating an example of a schematic configuration of a person search apparatus according to the present embodiment. 1 includes an input unit 11, an output unit 12, a storage unit 13, a face region detection unit 14, a human body region detection unit 15, an identification unit 16, a tracking unit 17, and a suspicious unit. Person detection means 18, person information integration means 19, frame information generation means 20, search means 21, screen generation means 22, notification means 23, transmission / reception means 24, and control means 25. ing. Note that the transmission / reception means 24 is connected to an imaging means 26 such as a camera for photographing a person for extracting person information, and acquires each time-series image included in the video imaged by the imaging means 26. be able to. Further, the imaging unit 26 may be configured integrally with the person search device 10.

  The input means 11 includes a face area detection instruction, a human body area detection instruction, an identification instruction, a tracking instruction, a suspicious person detection instruction, a person information integration instruction, a frame information generation instruction, a search instruction, a screen generation instruction, and a notification instruction from a user or the like Various instructions such as transmission / reception instructions are accepted. Note that the input unit 11 includes, for example, a keyboard, a pointing device such as a mouse, a voice input device such as a microphone, and the like.

  The output unit 12 includes the instruction content input by the input unit 11, control data generated based on each instruction content, the face region detection unit 14, the human body region detection unit 15, the identification unit 16, the tracking unit 17, and the suspicious person. Contents of various information obtained by progress or result executed by each component such as detection means 18, person information integration means 19, frame information generation means 20, search means 21, screen generation means 22, notification means 23, transmission / reception means 24, etc. Is displayed and the sound is output. The output unit 12 has a screen display function such as a display, a sound output function such as a speaker, and the like.

  Further, the output unit 12 outputs a search result obtained by the search unit 21 described later, information displayed on the screen generated by the screen generation unit 22, and the like to an external device. That is, the output means 12 outputs to an external device, for example, to a printer, generates a file and outputs it to a recording device such as a database or a recording medium, or generates a file based on a search result for a security robot Output control signals, switch on / off of sensors in the facility to be guarded, switch lights on / off, or related information based on search results for mobile terminals held by guards (there was an anomaly A printing / output function for outputting a control signal for displaying the location and contents). Further, the output unit 12 can output simultaneously to one or more external devices described above.

  The storage unit 13 can store various information for realizing the above-described embodiment, and reading and writing are performed as necessary. Specifically, the accumulating unit 13 includes various feature amount data used for face authentication, gender and age estimation, face area detection results in the face area detecting unit 14, and human body area detecting unit 15. Human body region detection result, person identification result in identification means 16, tracking result in tracking means 17, suspicious person detection result in suspicious person detection means 18, person information integration result in person information integration means 19, frame information in frame information generation means 20 Information such as the generation result, transmission / reception information in the transmission / reception means 24, information controlled by the control means 25, error information at the time of error occurrence, log information, a program for realizing the present invention, and the like are accumulated.

  The face area detection unit 14 includes a person included in an image in the video for a real-time video captured by the imaging unit 26 such as a camera or a huge amount of monitoring video already stored in the storage unit 13. If it is determined that the face area of the person is detected, the face area of the person is detected. That is, the face area detection unit 14 acquires a video captured by the imaging unit 26 such as a camera via the transmission / reception unit 24, and among the time-series images included in the acquired video, a predetermined image One or a plurality of human faces are detected for each frame image or an image with an interval of several frames.

  Specifically, the face area detection unit 14 acquires the feature amount of the face from position information such as eyes, nose, and mouth in the face included in the photographed image, for example, and detects it as a preset face. For example, a human face is detected by performing a matching process using a feature amount matching pattern. Further, the face area detection unit 14 is not limited to the above-described face detection processing, and for example, face detection by edge detection or shape pattern detection, face detection by hue extraction or skin color extraction, or the like can be used. Note that the face area detection unit 14 detects a rectangular face area having a vertical and horizontal width of the face in the image after detecting the face.

  The face area detecting means 14 detects the center coordinates (position information) of the face area and the size (size) of the area on the image, and synthesizes the face area with the original image with a predetermined shape. Various information for screen display is acquired and stored in the storage means 13 so as to be clearly understood. In the present invention, the shape of the face region may be a rectangle, a circle, an ellipse, another polygon, a silhouette shape enlarged from the outer shape of a human face at a predetermined magnification, or the like. The face area detecting means 14 has some additional functions in addition to the above-described contents, which will be described later.

  The human body region detection means 15 includes a person included in an image in the video for a real time video taken by the imaging means 26 such as a camera or a huge amount of monitoring video already stored in the storage means 13. If it is determined that the human body area of the person is detected. That is, the human body region detection unit 15 acquires a video captured by the imaging unit 26 such as a camera via the transmission / reception unit 24, and among the time-series images included in the acquired video, a predetermined image One or a plurality of human body regions are detected for each frame image or an image with an interval of several frames.

  Specifically, the human body region detection unit 15 compares, for example, successive image frames, and there is a place where the color information (luminance, chromaticity, etc.) changes within a predetermined time, and is further surrounded by the place. A human body region is detected when the region is equal to or larger than the predetermined region or when the moving range with time is within the predetermined range. The human body detection method is not limited to this in the present invention.

  Further, the human body region detection means 15 detects the center coordinates of the human body region and the size of the human body region on the image, and synthesizes the human body region with the original image with a predetermined shape so that the human body region can be clearly understood. Various information for screen display is acquired and stored in the storage means 13. The shape of the human body region may be a rectangle, a circle, an ellipse, another polygon, a silhouette shape enlarged at a predetermined magnification from the outer shape of a person, and the like, similar to the face region described above. Further, the human body region detection means 15 has some additional functions in addition to the above-described contents, which will be described later.

  The identification unit 16 performs an identification process as to whether or not the same person is included in one or a plurality of persons detected from two images extracted from videos taken at different times by the same camera. . Specifically, for example, the identification unit 16 determines whether the person included in the shooting is a person who has been shot or a new person, for example, whether there is a person in the previous video. If there is a person, it is determined whether or not the person is included in the movable range set in advance. Further, the same person can be determined based on the presence or absence of similarity by comparing the front and back face feature amounts. It is possible to determine whether or not a message is included.

  The identification unit 16 also includes the face pattern detected by the face region detection unit 14 and personal information stored in the storage unit 13 in advance (name, gender, age, which celebrity (person) is similar). It is possible to determine who the face is, that is, who the person is. Furthermore, the identification means 16 can also acquire a similarity such as how much the face is similar to whom. Thereby, in the screen generation means 22 etc. which will be described later, it is possible to display a predetermined number of images in which a person with a high degree of similarity to a certain person is displayed in descending order of the degree of similarity.

  Note that the identification unit 16 may identify a person using an appropriate parameter by a method of selecting a feature amount according to the state of the image. For example, the identification unit 16 determines the state of the person (stopped, walking, whether there was a change in sunshine, etc.), and based on the result, various feature amounts (gait (step length, walking speed, posture, etc.) are determined. Including), body type, gender, clothes color, hair color, etc.) can be used for person identification. Furthermore, the identification unit 16 calculates the degree of separation only for the feature amount to be used, performs weighting based on the degree of separation, and changes the contribution rate for each feature amount with respect to person identification. Thereby, highly accurate person determination can be performed using the feature-value according to a condition.

  In the identification unit 16, the human body detection result in the human body region detection unit 15 may be different from the face detection result in the face region detection unit 14. In such a case, the identification unit 16 may not be able to determine the same person by identification because the detection information selection unit 16 does not match even if the comparison is performed when the process is switched halfway. Conceivable. Therefore, the identification unit 16 performs both the face detection by the face region detection unit 14 and the human body detection by the human body region detection unit 15 on the acquired video before and after switching, for example, and the detection results are described above. In this way, it is associated with whether or not they are the same person. Thereby, when the identification unit 16 performs the identification process, the person identification process can be easily performed even if the detection information is switched.

  The tracking unit 17 tracks the behavior of the same person identified by the identifying unit 16 over time using the captured video. The tracking unit 17 can track a plurality of persons included in the video over time in parallel. In addition, when tracking the person included in the video over time, the tracking unit 17 may estimate the next movable range from the orientation, posture, behavior, and the like of the person acquired from the image. it can. In this case, the tracking unit 17 can also generate information for combining the transferable range with the image and output the information to the screen generation unit 22 described later.

  Here, the tracking means 17 tracks the person from the position of the foot and head of the human body region included in the image and its characteristics, and when the size of the person that can be determined to be the same changes, It is judged whether or not it is hidden by etc. The part of the person to be concealed is, for example, the lower body, the upper body, the head, arms, and the torso. In other words, in this embodiment, when information about the height of the human body from the toes of the feet to the head can be acquired at least once on the screen, if both the head and feet are not simultaneously hidden, The person can be estimated and tracked.

  The suspicious person detection means 18 collates the tracking result by the tracking means 17 with preset suspicious behavior patterns and behavior data stored in the storage means 13, so that 1 taken by the imaging means 26. Or, a person corresponding to a suspicious person among a plurality of persons is detected.

  In other words, the suspicious person detecting means 18 has at least one suspicious action such as the person to be tracked hidden behind the shield, the camera being worried about for a predetermined time or longer, or staying for a long time. In that case, the person is detected as a suspicious person. Further, the suspicious person detecting means 18 detects a suspicious person even when the person to be tracked is wearing a mask or wearing sunglasses to hide his face.

  Further, when the suspicious person detecting means 18 detects a suspicious person, the suspicious person detecting means 18 accumulates all the contents including information on which suspicious action is applicable in the accumulating means 13. As a result, when “what corresponds to a suspicious person” is designated as a keyword at the time of search, it is easily extracted by a simple operation of the searcher. Furthermore, the suspicious person detecting means 18 may generate a screen for notifying the screen generating means 22 of the contents to that effect when the suspicious person is detected.

  The person information integration means 19 associates the face area and the human body area as the same person, and integrates the characteristics of the person. Specifically, the person information integration unit 19 acquires the barycentric coordinates of the face area in the image, and when there is a human body area including the acquired coordinates, the face area and the human body area are from the same person. Associating as there is.

  Furthermore, when one human body region includes two or more face regions, the person information integration unit 19 associates the same human body region with all the face regions. Thereby, even when a plurality of persons temporarily overlap on the screen during the tracking process or the like, it can be continuously tracked without interruption. Persons are managed using identification information such as IDs. Note that the processing for associating the human body region and the face region as those of the same person is not limited to this in the present invention. For example, the posture and orientation of the person are extracted and the extracted information is used. May be associated with each other.

  The frame information generation unit 20 generates frame information in which the person information for each person integrated by the person information integration unit 19 described above is stored for each frame of the image included in the video, and the generated search person feature data Is stored in the storage means 13 as a database or the like. Thereby, the characteristics of a person can be managed in units of frames. In addition, since the time information of the shot is given for each frame, various searches can be performed with high accuracy using various search keywords such as what kind of person at what time during the search. can do. Note that a specific configuration of frame information in the present embodiment will be described later.

  The search means 21 can be selected by the searcher according to the present embodiment for various types of information such as real-time video captured by the imaging means 26 such as a camera and surveillance video already stored in the storage means 13. A predetermined keyword, an extraction condition, or the like is set, or a suspicious person detection condition for detecting a suspicious person is set in advance, and a search result such as an image including a corresponding person is extracted.

  The search means 21 automatically sets features such as a face and clothes (including not only clothes but also hats, masks, glasses, etc.), action extraction conditions, and the like for the search results. It is also possible to perform a process of performing a similar search for searching for persons similar to, and outputting a predetermined number of search results in descending order of similarity.

  In addition, when the search unit 21 searches for a lost child or the like in real time using a monitoring robot as will be described later, a condition set in advance for an image from the imaging unit 26 provided in the monitoring robot is used. It also has a function of retrieving an image on which a person satisfying (height, color of clothes, location information, etc.) is displayed and displaying the result in real time by the screen generation means 22.

  The search screen in the search means 21 and the output results displayed as a result thereof are generated by the screen generation means 22 and output by the output means 12. Thereby, in this embodiment, the search which satisfy | fills a predetermined condition can be performed based on the image | video and the real-time image | video accumulated beforehand, and the search result can be displayed. The search method in the search means 21 will be described later.

  The screen generation unit 22 performs a video search performed by the imaging unit 26, a face region detected by the face region detection unit 14, a human body region detected by the human body region detection unit 15, and a person search in this embodiment. A screen for outputting a notification result or the like in the notification means 23 such as a menu screen, an input screen used in the search by the search means 21, or a search result screen is generated. At this time, the screen generation means 22 can also display, for example, digitized data such as position information corresponding to the area of the photographed person.

  The screen generation means 22 generates and displays a screen so that various data such as a movable range used when tracking a person included in the video and a tracking route obtained by the tracking means 18 can be easily monitored. Can do. Furthermore, when the screen generation means 22 corresponds to a preset suspicious person's behavior pattern, the screen generation means 22 generates a screen for notifying the content to that effect.

  Further, when displaying the search results obtained by the search means 21, the screen generation means 22 may temporarily stop the display of the search results when displaying the search results in real time. it can. Thereby, for example, when the search result is updated and displayed as needed in real time, the necessary information can be displayed without forgetting to confirm the information by pausing. Further, the screen generation means 22 can continuously display the display content suspended as described above.

  In addition, the screen generation means 22 not only displays the characteristics or actions of the person included in the video, but also, for example, when one or more security robots provided with the imaging means 26 circulate in the security target facility. Can display the patrol position information (three-dimensional coordinates, latitude, longitude, etc.) transmitted together with the monitoring video for each security robot. Thereby, it is possible to accurately grasp the current position for each security robot.

  Various kinds of information necessary for screen generation can be used by appropriately reading out necessary information from information stored in the storage unit 13 in advance. Further, the screen generation unit 22 can display the generated screen or the like on a display as the output unit 12 or output sound or the like through a speaker or the like.

  The notification means 23 generates an image detected as a suspicious person obtained by the suspicious person detection means 18 and information related to the image (detection date and time, detection location, video for a predetermined time, etc.) by the screen generation means 22. And displayed by the output means 12. In addition, the notification means 23 is an alert that notifies the security officer, the supervisor, the representative manager, the security robot, etc. in charge of the building in the manager or the security company in response to the problem occurrence signal in such suspicious behavior detection. It has a function.

  For example, when notifying the security robot or the like, the notification means 23 is in a position where the security robot is facing the target person or close enough that the target person is photographed by the imaging means 26 of the security robot. Therefore, it is possible to output a control message for the security robot that causes the security robot to output a voice message to the target person, or to perform a process of calling attention to the surroundings by an alarm lamp or emergency sound. .

  The transmission / reception means 24 receives monitoring video from one or a plurality of imaging means 26 via a communication network such as a LAN (Local Area Network) or the Internet. Note that the transmission / reception unit 24 does not have to receive the monitoring video directly from the imaging unit 26. For example, the video acquired by the imaging unit 26 is temporarily stored somewhere and the stored information is used. In this embodiment, human body detection may be performed.

  Further, the transmission / reception means 24 can be used as a communication interface for transmitting to other terminals constituting the person search apparatus 10 and receiving various data from other terminals.

  The control means 25 controls the entire functional configuration of the person search apparatus 10. Specifically, the control unit 25 performs face area detection, human body area detection, identification processing, and tracking processing based on input information from the user input by the input unit 11. Various processes such as performing suspicious person detection processing, performing personal information integration processing, generating frame information, performing search processing, generating screens, performing notification control, performing transmission / reception control, etc. Take control.

  Further, for example, when the security robot is patroling the inside of the security target facility, the control unit 25 controls the security robot with respect to the tour route and the operation when the target person is found. Specifically, the control unit 25 causes the notification unit 23 to output a voice message, a warning, or the like for the target person and its surroundings when the target person is discovered, or forcibly terminates patrol of the security robot after the target person is found, Control such as setting a return path. That is, when the security robot finds the target person, the patrol can be ended and the route setting can be made to guide the target person to a predetermined place such as a management room.

  Further, the control means 25 generates a control signal for causing the security robot to perform an emergency operation in response to a warning signal (for example, fire, water leakage, person detection, etc.) obtained from the security robot. Can be transmitted to the security robot or the like via the output means 12, the transmission / reception means 24, etc., and the security robot can be controlled.

<Additional functions in face area detection means 14 and human body area detection means 15>
Next, additional functions in the face area detection unit 14 and the human body area detection unit 15 will be described with reference to the drawings. FIG. 2 is a diagram illustrating an example of additional functions in the face area detecting unit and the human body area detecting unit. 2A shows an example of the functional configuration of the face area detecting means 14, and FIG. 2B shows an example of the functional configuration of the human body area detecting means 15.

  As shown in FIG. 2A, the face area detection unit 14 includes a tracking unit 31, a face authentication unit 32, a gender / age estimation unit 33, and a face-hiding determination unit 34. The tracking unit 31 performs person tracking using the detected face area, and assigns identification information (tracking ID) to identify and accumulate each of the plurality of face areas included in the image frame. In addition, the tracking unit 31 outputs, for example, four types of statuses “unused”, “frame-in”, “frame-out”, and “tracking” for the assigned tracking ID, depending on the tracking status.

  The face authentication unit 32 acquires a face feature amount including each piece of arrangement information such as eyes, nose, and mouth from the detected face area, and stores the acquired feature amount in advance as a database (DB) or the like in the storage unit 13 or the like. Face authentication processing is performed using the accumulated facial features of persons such as registered persons (registered persons) and entertainers, and the identification information (ID) of the authenticated person is output. Further, the face authentication unit 32 may convert the ID of the face image stored in the storage unit 13 into the name of a person (registered person / celebrity) and output it.

  The gender / age estimation means 33 acquires the facial feature quantity from the detected face area, estimates the gender and age using the acquired facial feature quantity and the storage means 13, and outputs the result.

  The face hiding determination unit 34 detects whether or not a process for hiding the face such as a mask or sunglasses is performed on the detected face area. Specifically, the face hiding determination unit 34 acquires color information of the eye area and mouth area estimated from the face area, and the color information of the eye area is assumed to be wearing sunglasses (for example, black). System), it is determined that the face is hidden by sunglasses, and when the mouth area is a color assumed to be wearing a mask (for example, white), the face is hidden by the mask. And the determination result is output.

  Further, the human body region detection means 15 has a height estimation means 35, a color information extraction means 36, and a person position estimation means 37 as shown in FIG.

  The height estimation means 35 calculates the height of the person for each of the detected human body area and face area. For example, the height estimating means 35 detects one or a plurality of persons included in the image, and determines whether or not the person is a new person when the person is present. The position of the person in real space is calculated from the tip of the foot and the vanishing point corresponding to the image, and the height of the actual person is estimated together with the apparent size on the image. The vanishing point may be included in the image, but may not be included in the image depending on the camera angle or the like. In this case, a vanishing point is set on a virtual space and the vanishing point is used. Note that the height estimation method is not limited to this in the present invention.

  The color information extraction unit 36 estimates the head region, skin region, upper body region, and lower body region of a person using the positional relationship between the detected region information of the human body region and the face region, and color information of each region To extract. Specifically, color information such as hair, upper garment, and lower garment is extracted, and a representative color (average color, etc.) of the extracted color information is determined as color information such as hair, upper garment, and lower garment.

  In the above-described calculation of the color information of each area, the total number of pixels and the appearance frequency of each area are counted, and, for example, RGB values for the top 10 colors are output. Here, each of the above-described areas is specifically the head area from the top of the face area, the cheek (skin) area from the left and right of the face area, and the upper body area from the bottom of the face area above the center of the human body area. The lower body region is extracted below the center of the human body region or above the ground. Each area described above may be set based on, for example, the position and size of the face area (for example, one side length when the face area is represented by a rectangular area). In addition, the setting content of the area | region extracted is not limited to this in this invention. Furthermore, the color information extraction means 36 can cut out and output the image data of at least one of the face area, human body area, head area, skin area, upper body area, and lower body area as it is.

  The person position estimation means 37 calculates the position coordinates of the person in the real space for each of the detected human body area and face area. In this case, the person position estimating means 37 calculates the position coordinates (X, Y, Z) in the three-dimensional real space by a conversion formula set in advance for the two-dimensional coordinates obtained from the image taken by the camera. get.

<Configuration Example of Frame Information in Frame Information Generation Unit 20>
Next, a configuration example of frame information in the above-described frame information generation unit 20 will be described. FIG. 3 is a diagram illustrating an example of frame information in the present embodiment. The frame information generating means 20 extracts information shown in FIG. 3 for the person detected by performing processing for each frame. 3 includes, for example, “file name” and “detection date / time” as common items, and “person position coordinates (X, Y, Z)” as information extracted from the face area. "Height information", "various color information", "registrant information", "similar celebrity information", "age information", "sex information", "face orientation (PAN, TILT)", "mask" “Sunglasses”, “acquired face image information”, “personal position coordinates (X, Y, Z)”, “height information”, “various color information”, and the like.

  The “file name” stores the file name of the video file currently being processed. In addition, “date and time of detection” stores the date and time when a person was detected. As “detection date and time”, since the video file name corresponds to the actual recording start time, the actual time is stored by adding the shooting start time and the time in the video.

  Further, the “person position coordinates (X, Y, Z)” extracted from the face area stores the XY coordinates of the camera position and the XYZ coordinates of the person's feet in the real space with the ground as the origin. That is, the position coordinates in the real space of the person calculated from the face area are stored in “person position coordinates (X, Y, Z)”. Also, the height calculated from the face area is stored in the “height information”. The “various color information” stores the total number of pixels of the head and upper body, the top 10 colors, and the frequency calculated from the face area.

  As the head color information, RGB numerical values and frequencies from the first to the tenth frequency for the head calculated from the face area are stored. Further, the total number of pixels of the head region calculated from the face region is stored as the total number of pixels of the head. Further, as the upper body color information, RGB numerical values and frequencies of the upper body calculated from the face area from the 1st frequency to the 10th frequency are stored. Further, the total number of pixels of the upper body area calculated from the face area is stored as the total number of pixels of the upper body.

  In “registrant information”, the closest person name and authentication score are set as a result of collation with the registered person DB. Specifically, when authentication is performed as a registrant (exceeding the authentication threshold) by collation with the registered person DB, the names of the top 10 registrants are stored. If not authenticated, “unregistered person” is stored. Further, the authentication result score is indicated by an integer value from 0 to 1000 for the top 10 authentication scores, and -1 is stored if the result of collation is “unregistered”.

  Further, in “similar celebrity information”, the closest person name and authentication score are stored as a result of collation with a DB in which celebrities are registered. Specifically, the name of the entertainer is stored when the authentication score is the highest and is above the threshold value by collation with the entertainer DB. If not authenticated, “similar” is stored. The authentication result score is the same as described above.

  The “age information” stores the age and reliability of the detected person. The “sex information” stores the sex and reliability of either gender. Further, the “face orientation” stores a face orientation angle with respect to camera work such as pan and tilt as the origin. Specifically, the angle of the face viewed from the camera is stored with the camera as the origin, and the upward direction and the left direction viewed from the camera are +.

  Further, “mask” stores whether or not the periphery of the mouth is hidden by a mask or the like, and specifically stores “Yes”, “No”, or “Unknown”. Further, “Sunglasses” stores whether or not the vicinity of the eyes is hidden by sunglasses or the like, and specifically, “Yes”, “No”, or “Unknown” is stored.

  The “acquired face image information” stores the file path to the detected face image and the reliability, and specifically, the file path to the location where the face image detected in this frame is stored. And the length of one side of the detected face area, the authentication result name having the highest authentication score equal to or higher than the threshold, the reliability, and the like are stored.

  Further, in the “person position coordinates (X, Y, Z)” extracted from the human body region, the XY coordinates of the camera position and the XYZ coordinates of the human foot in the real space with the ground as the origin are stored. The position coordinates of the person in the real space calculated from the above are stored.

  Also, the height calculated from the human body area / face area is stored in the “height information”. Further, the “various color information” stores the total number of pixels of the head, upper body, and lower body, the top 10 colors, and the frequency calculated from the human body region. Specifically, the frequency for the head is ranked from 1st to 10th. RGB values and frequencies up to, the total number of pixels in the head region, the RGB numbers and frequencies from the first to the 10th frequency for the upper body, the total number of pixels in the upper body region, the frequencies from the first to the 10th in the lower body At least one of the RGB numerical value and frequency, and the total number of pixels in the lower body region are stored.

  In the present embodiment, the above-described information is output in units of frames as a result of various processes. If neither a person area nor a face area is detected, nothing is output. Some information may not be output depending on the detection state of the person area and the face area. In that case, a value indicating “unknown” is output.

  In the present embodiment, the above-described information is generated for each frame, and a tag is assigned to each data for which the frame information is output. This makes it possible to clarify what information each data corresponds to. In addition, when a plurality of persons are detected in one frame, different tracking IDs (identification information) are allocated for each person, and the same date and time are assigned to each tracking ID. Is output.

  FIG. 4 is a diagram for explaining the outline until the retrieval person feature is acquired in the present embodiment. The diagram shown in FIG. 4 is a diagram for generating search person feature data from the person features obtained by the above-described face region detection and human body region detection. The person characteristic data is generated by the frame information generation means 20 and the like, and is stored in the storage means 13 by a database or the like.

  Specifically, in FIG. 4A, in frame information generation, characteristics (for example, height, presence / absence of sunglasses, etc.) for the person (for example, person A, person B, etc.) detected in each frame (F). Is acquired, and these person features are collected for each person as shown in FIG. 4B, and the person features are counted. That is, in FIG. 4B, all data are discretized and the number of appearances is counted. Specifically, in FIG. 4B, the person A is extracted in three frames in the video, of which When the height is estimated to be 170 cm, there are 2 times, when the height is estimated to be 175 cm, once when the wearing of sunglasses is detected, twice, when there is no detection of wearing, etc. Is counted. In the above example, the height is discretized every 5 cm, but the present invention is not limited to this.

  Next, as shown in FIG. 4C, normalization is performed with the number of frames, and this data is used for retrieval. Specifically, when the search person feature data is created, in order to search for a person with a combination of various features and actions, for example, it is output in various forms such as height, sex, and HSV value of clothes color. According to the following formula, describe the person information in a unified manner.

For example, when creating data about a feature X of a person A, when the feature X is a digital value (gender (male / female), sunglasses (presence / absence), mask (presence / absence), etc.), The total frequency F _AX for the person A having the frequency value X is calculated by the equation (1) shown below.

また、特徴Ｘがアナログ値（身長、認証スコア、服の色のＨＳＶ値等）の際には、以下に示す（２）式により連続値Ｘ内特定範囲ｄの人物Ａについての総頻度を算出する。

When the feature X is an analog value (height, authentication score, clothes color HSV value, etc.), the total frequency of the person A in the specific range d within the continuous value X is calculated by the following equation (2). To do.

これにより、人物の全ての情報をヒストグラムへと変換することができる。なお、このままでは人物間の比較に用いることができないため、各人物の検出フレーム数で正規化することで、正規化ヒストグラムへと変換し、以下に示す（３）式により人物Ａの特徴Ｘについての信頼度Ｐ_ＡＸを算出する。

Thereby, all the information of the person can be converted into a histogram. Since it cannot be used for comparison between persons as it is, it is converted into a normalized histogram by normalizing with the number of detected frames of each person, and the characteristic X of the person A is expressed by the following equation (3). The reliability P _AX is calculated.

これにより、図４（ｃ）に示すように検索用人物特徴データを取得することができる。

Thereby, as shown in FIG.4 (c), the person characteristic data for a search can be acquired.

<Search method in this embodiment>
Next, the search method in the present invention in the search means 21 described above will be described with reference to the drawings. In the search method according to the present embodiment, it is possible to search for a person shown in a video with a combination of various person characteristics and actions by the above-described search scoring method using person feature data. In any combination, search results can be output in descending order of matching with the search request.

  Here, FIG. 5 is a diagram illustrating an example of a search method according to the present embodiment. In this embodiment, as shown in FIG. 5A, the search request “175 cm person wearing sunglasses” input by the searcher is converted into the above-described feature amount, and the maximum value ( = 1) is set as the search request feature amount (FIG. 5B). Further, a feature corresponding to the search request feature value is extracted with reference to the feature (FIG. 5C), a distance from the feature is calculated, and the distance is set as a search score (FIG. 5D). The search results are output in descending order of the number of persons.

  It should be noted that the color information suitable for search in the present embodiment is particularly important for the human characteristics, such as the color of clothes and hair, but the expression of the color changes depending on the subjectivity. To absorb. Specifically, in order to adapt the search result to a color that a person feels, for example, processing is performed in a color space that is set in advance based on human visual characteristics, such as an HSV space. The color space described above is not limited to HSV in the present invention, and for example, HLS, LUV, LAB, or the like can be used. Furthermore, the subjectivity variation is absorbed by reducing the color resolution.

Further, when performing a search using free words, the input word / phrase is converted into a search request and a search process is performed. Conventionally, when a search keyword such as “red jacket blue pants” is entered, there is a problem that it is impossible to understand which adjectives and which nouns are in a modification relationship. Therefore, in the present embodiment, the search accuracy can be further improved by converting into a search request in consideration of the modification relationship of the input words.
In this embodiment, when inputting a search request, it is possible to select from a free word input representing a person or a searchable person feature list.

  Thereby, it is possible to easily search for a video at a time when a person close to a person image intended by the searcher is taken from a huge amount of videos stored, and to obtain the search result. In the present embodiment, when the video is confirmed, in addition to the person video, the appearance characteristics and behavior of the person are written in characters. The desired person can be output to various media as suspicious person information. Thereby, a highly accurate person search can be efficiently performed with a simple operation on the captured image.

<Example of Screen Generated by Screen Generation Unit 22>
Next, an example of a screen generated by the screen generation unit 22 will be described with reference to the drawings. FIG. 6 is a diagram showing an example of screen transition in the present embodiment. The screen example shown in FIG. 6 includes an initial screen 41, a date / time designation screen 42, a feature / action condition designation screen 43, a search result display screen 44, a video playback screen 45, and a shared screen 46. ing. When the system according to the present embodiment is started, an initial screen 41 is first displayed. Further, the transition to each screen shown in FIG. 6 is performed by setting a transition button for transitioning to a predetermined screen in advance as shown in FIG. 6 and selecting the transition button so that each screen is output means such as a display. 12 is output. Next, each of the above-described screens will be described in detail below. However, screen examples generated in the present invention are not limited to this, and the screen layout and display contents shown below are also limited thereto. It is not a thing.

<Initial screen 41>
FIG. 7 is a diagram illustrating an example of an initial screen in the present embodiment. The search method from the initial screen 41 shown in FIG. 7 includes a store visitor search area 51, a suspicious person search area 52, a keyword search area 53), a condition-designated search (search from person characteristics) area 54-1, and a real-time search. There are five types of areas 54-2.

  In FIG. 7, the date and time designation screen is displayed by pressing the “visitor search button” in the visitor search area 51 in the target facility where the above-described imaging means 26 is installed (for example, when confirming a person who has visited a store) In addition, when searching for a suspicious person on the initial screen 41, the “Date Search” button in the suspicious person search area 52 is pressed to change to the date and time designation screen 42.

  In addition, when performing a keyword search on the initial screen 41, a keyword to be searched is entered in the edit box of the keyword search area 53, and a “date search” button is pressed to set a preset date and time or place. A search is performed by the search means 21 using information or the like and a search character entered in the edit box as a keyword (search request conversion process), and at the same time, the screen transitions to the search result display screen 44. Search results are displayed.

  Further, when searching from the characteristics of a person on the initial screen 41, a transition to the characteristic / behavioral condition specifying screen 43 is made by pressing a “search from person characteristics” button in the condition specifying search area 54-1.

  Furthermore, when performing a real-time search for a lost child or the like by the monitoring robot on the initial screen, a transition to the feature / action condition designation screen 43 is made by pressing the “real-time search button” in the real-time search area 54-2.

<Date designation screen 42>
FIG. 8 is a diagram showing a date and time designation screen in the present embodiment. In the initial screen 41 described above, when each button in “Search” is pressed, the date / time condition is designated on the date / time designation screen 42. In addition, on the date and time designation screen 42 shown in FIG. 8, a search location condition can also be designated. Thus, for example, even when an image obtained by a mobile imaging means such as a security robot is targeted, an image obtained by a target location based on the patrol position information of the security robot obtained together with the image Can be efficiently searched for only the target. This location designation is effective because, for example, the video to be searched can be limited even when the video from the fixed imaging means is targeted.

  Next, the specified date and time conditions and location conditions are passed to the search process, and when the search process is completed, the search result is called and the screen shifts to the search result display screen 44.

  The date / time designation screen 42 shown in FIG. 8 includes a radio button / date / time designation box 55-1, a location designation box 55-2, a search execution button group 56, and a hyperlink display area 57. In the radio button / date / time designation box 55-1, the initial state is “from today's image”, and the date / time designation box is in a state of being semi-transparent and not operable. Here, when “specify time” is selected with the radio button, the date / time designation box becomes operable. Also, a place designation box 55-2 designates a preset place or range. The designated place may be, for example, a place name, a zip code, a name of a store or a commercial facility, the number of floors, and the like, and is further designated as an area range within a radius of 100 m from the security robot that shot the video. You can also.

  The search execution button group 56 has an “OK button” and a “cancel button”, and the “OK button” portion is displayed by the transition source information (the pressed button) passed by the initial screen 41. Change. The respective displays are “store visitor search”, “suspicious person search”, and “keyword search”. When each button is pressed, the date / time information and the location information are stored in the storage variables, and the search processing function is called using the date / time specification information and the location specification information as arguments. Also, as soon as the search process is completed (sequential process), the screen transitions to the search result display screen 44. If “from today's image” is selected in the radio button / date / time designation box 55-1, the flag is designated by the argument and the search processing function is called. Further, in the hyperlink display area 57, by selecting a portion where characters are displayed, the screen is changed to the first screen (initial screen 41).

<Characteristic / action condition designation screen 43>
FIG. 9 is a diagram showing a feature / action condition designation screen in the present embodiment. When “search from person feature” or “real-time search” is selected on the initial screen 41 described above, the search condition is designated on the feature / action condition designation screen 43. Call the storage function for all conditions. Further, when the search button is pressed, a search processing function is called and a transition is made to the search result display screen 44.

  The feature / action condition designation screen 43 includes a date designation area 58, a place designation area 59-1, a height designation area 59-2, a color information designation area 60, a gender designation area 61, a mask designation area 62, and a sunglasses designation area. 63, an age designation area 64, a character designation area 65, a similar entertainer designation area 66, a search execution button group 67, and a hyperlink display area 68.

  In the date designation area 58, the search target date and time is set. The start and end times are checked for large and small errors, and the error is notified to the searcher when an error occurs. In the place designation area 59-1, a place for searching for a target person is set. Specifically, the location information to be searched is specified based on the position information of the image pickup means 26 where the video was shot. If the image pickup means 26 is a fixed type, the position information can be easily obtained from the identification information of the image pickup means 26 included in the video data, and if the position information is a moving type such as a monitoring robot, The patrol position information of the monitoring robot included in the video data is used.

  In the height designation area 59-2, the height of a person to be searched is set. A threshold value is set for a minimum value (for example, 100) and a maximum value (for example, 200), and a search is performed when a numerical value not included in the range is specified or when a character other than the numerical value is input. Notify the person to that effect.

  Further, in the color information designation area 60, the color of the head of the person to be searched, the color of the upper garment (upper clothing), and the color of the lower garment (lower clothing) are set. In the present embodiment, designation is performed using a color map as shown in FIG. Here, FIG. 10 is a diagram illustrating an example of a color map. In the present embodiment, RGB values (0 to 255) are converted into HSV values using a color map, and H (0 to 360) is plotted on the outer ring and SV (0 to 1) is plotted on the inner rectangle. . In addition, the outer ring acquires only angle information only by pasting BMP. Further, the H value is selected by the outer circle button, and the SV map according to the value of H is displayed in the internal rectangle. That is, the color information (RGB value) to be searched is output by converting the value of H and the position of the circle button (SV value) inside the rectangle into RGB. The currently selected color is displayed on the inner and outer circle buttons. By using the color map described above, the searcher can easily set various kinds of color information. When color information is not designated, the HSV value is set to “−1”.

  In the gender designation area 61, “male”, “female”, or “neither” is set. Also, in the mask designation area 62, “doing”, “not doing” or “both” is set for wearing the mask. In the sunglasses designation area 63, “doing”, “not doing”, or “both” is set for wearing sunglasses.

  Further, in the age designation area 64, for example, “-9 years old”, “10-19 years old”, “20-39 years old”, “40-59 years old”, “60 years old”, “60 years old” Set one of them. In the character designation area 65, a registered person name is set from the list in accordance with registered person information stored in advance. In the similar entertainer designation area 66, the entertainer name is set from the list in accordance with entertainer information accumulated in advance.

  The search execution button group 67 has a “search button for a person who meets this condition” and a “clear button”. When the “search button for a person who satisfies this condition” is pressed, “ The storage function for all conditions corresponding to “search from person characteristics” or “real-time search” is sequentially called. Finally, the search function is called to execute the search process, and after the process is completed, the screen transitions to the search result display screen 44. If there is an input that does not meet the conditions, for example, a message “Please input again” is notified. When the “clear button” of the search execution button group 67 is pressed, the settings of various conditions on the feature / action condition designation screen 43 are cleared (initialized). Further, in the hyperlink display area 68, by selecting a portion where characters are displayed, the screen is changed to the first screen (initial screen 41).

<Search result display screen 44>
FIG. 11 is a diagram showing a search result display screen in the present embodiment. Various search conditions are designated on the initial screen 41, the date / time designation screen 42, and the feature / action condition designation screen 43 described above, and the search results are displayed on the search result display screen 44. Also, a predetermined search result output function is called to display various information. In the example of FIG. 11, the upper limit number of displays per page is five. However, the present invention is not limited to this and can be arbitrarily set as necessary.

  Further, the search result display screen 44 shown in FIG. 11 includes various search areas 69, a search result number display area 70, an image display area 71 when the corresponding person appears, a face image 72 of the corresponding person, a search result display area 73, and a video confirmation. A button 74, a page selection icon 75, and a hyperlink display area 76 are included.

  In various search areas 69, according to the event selected on the initial screen 41 or the like, the search process and screen transition are performed by pressing various buttons corresponding to the event. The various search areas 69 are provided with a “pause button”. For example, this button is used to temporarily stop the output of the real-time search result without updating the real-time search. It is. Specifically, for example, in the case of real-time search, the search result display area 73 is displayed as needed from the one with a high similarity to a person to be searched such as a lost child, or has a certain degree of similarity. Since the result is displayed at any time, the display content is updated, which may cause the content displayed on the screen to be missed. Therefore, the search result can be confirmed accurately by pausing the update display of the search result.

  In the search result number display area 70, the number of search results and the number of currently displayed results are displayed (for example, “1-5 of 62 search results”). In the image display area 71 when the corresponding person appears, the human body image for thumbnail when the corresponding person appears is displayed. Further, in the face image 72 of the corresponding person, the face image for the thumbnail of the corresponding person is displayed. In the image display area 71 and the face image 72 of the corresponding person when the corresponding person appears, the image is not displayed if there is no display target image (“NO IMAGE” display or the like).

  In addition, the search result display area 73 displays icons corresponding to information (appearance characteristics / behavior) of the corresponding person, detection by the security robot, or the like as a result of the search. Here, FIG. 12 is a diagram for specifically explaining the search result display area. In the search result display area in FIG. 12, shooting time / person information 73-1, icon display area 73-2, shooting location information 73-3, and search score information 73-4 are displayed for each search result.

  In the shooting time / person information 73-1, the shooting time of the video extracted by the search, the name of the registrant if the person is a registrant, and the unregistered person if the person is not a registrant. Is displayed. In addition, eight types of icons are displayed in the icon display area 73-2 as an example. From the left, “age”, “height”, “with / without sunglasses”, “similar entertainer”, “with / without mask” Icons that are easy to visually recognize conditions such as “slowly”, “stay for a long time”, “check camera”, “detection by security robot” are fixedly arranged at predetermined positions.

  In addition, the icon of the security robot shown in the icon display area 73-2 indicates the icon of the security robot that is traveling in cooperation with the imaging unit 26. Therefore, the icon display area 73-2 may display the position information of the guard robot in addition to the icon display of the guard robot. Further, the types and number of icons displayed in the icon display area 73-2 are not limited to this, and icons of various sensors and security equipment that cooperate with the imaging unit 26 may be displayed. .

  The icons described above are set in advance corresponding to the above-described conditions, and can be displayed separately in preset colors such as “red”, “black”, and “gray”. Here, if it is displayed in “red”, it indicates that it was included in the search condition, and “black” indicates that it is not included in the search condition but is an image that meets the above condition. “Gray” indicates that the search condition and the analysis result of the video are not applicable. Note that the type of color is not particularly limited in the present invention, and other colors may be used. Alternatively, the colors may be distinguished and highlighted by blinking, diagonal lines, or the like. In the icon display area 73-2, not the icon but the character itself can be displayed. In this case, for example, the font type, size, thickness, underline, etc. of the character can be changed, It can be distinguished by blinking or changing the color.

  The shooting location information 73-3 displays the shooting location of the video extracted by the search, and the search score information 73-4 displays the search scoring result obtained by the search means 21 described above. That is, in the search result display area 73, it is possible to display from the highest search score. As a result, it is possible to easily confirm the requesting person by outputting the search results under various conditions in the order of coincidence with the search request. Note that the present invention is not limited to the method of displaying in the order of search score, and for example, it is possible to display in the order of shooting date and time.

  Further, when the cursor is moved to the icon display area 73-2 of the search result display area 73 and stopped for a predetermined time (for example, 0.5 seconds), additional information 73-5 corresponding to a preset icon is displayed as a pop-up. Is done.

  On the video confirmation button 74, when the button is clicked, a video playback process is called, and a transition is made to the video playback screen 45. Also, the page selection icon 75 jumps to each page when the search result covers a plurality of pages. Further, in the hyperlink display area 76, the screen is changed to the first screen (initial screen 41) by selecting the portion where the character is displayed. The above-described screen allows the searcher to easily grasp which condition the extracted search result corresponds to.

  In other words, on the search result display screen 44, by displaying information such as a person's characteristics and actions, devices linked to the imaging means 26, and the like with icons, the requested person can be easily confirmed.

<Video playback screen 45>
FIG. 13 is a diagram showing a video playback screen in the present embodiment. When the video confirmation button 74 is pressed on the search result display screen 44 described above, the corresponding video is displayed on the video playback screen 45. Here, the video playback screen 45 shown in FIG. 13 has a video display area 77, a text display area 78, a feature display area 79, and a button area 80.

  In the video display area 77, video is played back and displayed using the input file path, playback start time, and playback end time as arguments of video playback processing. In addition, the video display area 77 includes a slider, a playback button, a stop button, and the like, so that it is possible to easily view the video from a desired place and to play or stop the video. In addition to the AVI file, the data displayed in the video display area 77 includes MPEG2, MPEG4, H.264. H.264, WMV, MP4, and other file format data can be displayed, and image data such as JPEG, GIF, TIFF, and BMP can also be displayed.

  Also, in the text display area 78, when the video playback time is the time in the text, the action text corresponding to the video is displayed with highlighted characters such as red letters and bold letters. Further, past information is displayed as it is, and when new information is displayed, the past information is changed to black characters. Thereby, it is possible to easily grasp the current action and the action history so far.

  In the feature display area 79, feature information of a person included in the displayed video is displayed. When a plurality of people are photographed, the respective feature information is displayed.

  In the button area 80, a “share button” and a “similar search button” are provided. When the “share button” is pressed, the screen transitions to the share screen 46. In this case, the video reproduction in the video display area is stopped to reduce the load on the apparatus. Also, when the “similarity search button” is pressed, a similarity search from another video is performed based on the characteristics of the displayed person, etc., and it is checked whether the image is taken by another camera, and the result is the search described above. Output after transitioning to the result display screen 44.

  That is, in the video playback screen 45, since the behavior and time of the person in the video are described in the text passed from the search result display screen 44, the video is displayed in the video display area 77 while the video is being displayed. Is the time corresponding to the text, the text display area 78 displays the text in red (highlighted). Thereafter, when there is new information, the past information is displayed in black characters.

<Share screen 46>
FIG. 14 is a diagram showing a shared screen in the present embodiment. When the “Share button” in the button area 80 is pressed on the video playback screen 45 described above, the person information is displayed on the share screen 46.

  Here, the shared screen 46 has an image display area 81, a text display area 82, and a comment input area 83. In the image display area 81, an image when the corresponding person appears and a face image of the corresponding person are displayed. In the text display area 82, suspicious information is displayed with highlighted characters such as red characters, and time information (date, time of entry, time of closing, etc.) and characteristics (height, sex, year, etc.) are also displayed. Is done. In the comment input area 83, a comment is input by the input unit 11. The comment input area 83 displays “Please input a comment” as a default display, and the input means 11 can accept input of characters by a searcher or the like. Thereby, the searcher can set and manage personal information from a unique point of view.

  Furthermore, it has a print / output function for outputting various contents displayed on the shared screen 46 to a printer, generating a file and sending it to a predetermined mobile phone by e-mail. This makes it possible to check information about a person from both text and video on the video display screen, and to output it to an external device in the form of printed matter, etc. (Sharing).

<Hardware configuration example>
Here, the person search apparatus 10 described above can be controlled by a dedicated apparatus configuration having the functions described above, but generates an execution program (person search program) that can cause a computer to execute each function. For example, the person search processing in the present invention can be realized by installing the execution program in a general-purpose personal computer, server, or the like.

  A hardware configuration example of a computer capable of realizing person search processing in the present embodiment will be described with reference to the drawings. FIG. 15 is a diagram illustrating an example of a hardware configuration capable of realizing the person search process according to the present embodiment.

  15 includes an input device 91, an output device 92, a drive device 93, an auxiliary storage device 94, a memory device 95, a CPU (Central Processing Unit) 96 that performs various controls, and a network connection device. 97 are connected to each other by a system bus B.

  The input device 91 has a pointing device such as a keyboard and a mouse operated by a user or the like, and inputs various operation signals such as execution of a program from the user or the like.

  The output device 92 has a monitor that displays various windows and data necessary for operating the computer main body for performing the processing according to the present invention, and displays the execution progress and results of the program by the control program of the CPU 96. can do.

  The input device 91 and the output device 92 may be integrated input / output means such as a touch panel. In this case, the input device 91 and the output device 92 are predetermined using a finger of a user, a pen-type input device, or the like. The position can be touched to input.

  Here, the execution program installed in the computer main body in the present invention is provided by a portable recording medium 98 such as a USB (Universal Serial Bus) memory or a CD-ROM, for example. The recording medium 98 on which the program is recorded can be set in the drive device 93, and the execution program included in the recording medium 68 is installed from the recording medium 98 to the auxiliary storage device 94 via the drive device 93.

  The auxiliary storage device 94 is a storage means such as a hard disk, and can store an execution program according to the present invention, a control program provided in a computer, and the like, and can perform input / output as necessary.

  The memory device 95 stores an execution program or the like read from the auxiliary storage device 94 by the CPU 96. The memory device 95 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  Based on a control program such as an OS (Operating System) and an execution program read and stored by the memory device 95, the CPU 96 performs various operations and inputs / outputs data to / from each hardware component. Each process can be realized by controlling the process. Various information necessary during the execution of the program can be acquired from the auxiliary storage device 94, and the execution result and the like can also be stored.

  The network connection device 97 acquires an execution program from another terminal connected to the communication network by connecting to a communication network or the like, or an execution result obtained by executing the program, or The execution program itself can be provided to other terminals. With the hardware configuration as described above, the person search process according to the present invention can be executed. Further, by installing the program, the person search process according to the present invention can be easily realized by a general-purpose personal computer or the like. Next, specific contents of the person search process will be described.

<Person search processing>
Next, a person search processing procedure executed by the execution program (person search program) according to the present invention will be described with reference to a flowchart. In the following description, an accumulation process procedure until the person characteristic information for search is accumulated and a search process procedure using the accumulated data will be described.

<Person search processing: accumulation processing procedure>
FIG. 16 is a flowchart illustrating an example of an accumulation processing procedure in the present embodiment. In FIG. 16, first, an image taken by an imaging means such as a camera is input (S01). Next, when one or more persons are present in each image included in the video, face area detection processing (S02) and human body area detection processing (S03) are performed. In the process of S03, whether or not there is a person can be determined based on, for example, whether or not the area where the color information is changing is greater than or equal to a predetermined size in comparison between time-series image frames. In addition, it can also be determined based on whether or not the face area has been detected in conjunction with the process of S02.

Further, after the processes of S02 and S03 are completed, the identification process is performed as described above (S04), the identified person is further tracked (S05), and the suspicious person is detected by the action of the person obtained from the tracking result. (S06)
Next, after performing the above-described processing, person information integration processing is performed (S07), frame information is generated from the integration information (S08), and the generated information is accumulated (S09). Note that the above-described accumulation processing procedure may not be performed in the case of searching in real time.

<Person search processing: Search processing procedure>
FIG. 17 is a flowchart illustrating an example of a search processing procedure in the present embodiment. In FIG. 17, first, the system is activated to display a search screen (S11). Here, it is determined whether or not there is a search instruction (S12). If there is a search instruction (YES in S12), the video or video stored in advance using the search method described above according to the input search condition or A search is performed on the real-time video (S13), and the output content corresponding to the search result is extracted from the storage means (S14). Further, a screen to be displayed to the searcher described above is generated from the extracted contents (S15), and the generated screen is output (S16). Here, it is determined whether or not the search is to be ended (S17). If the search is not ended (NO in S17), the process returns to S12 to perform the subsequent processing. If the search is to be ended (YES in S17), the search screen is closed and the person search process is ended.

  As described above, according to the present invention, a highly accurate person search can be efficiently performed with a simple operation on a captured image.

  As for the person search, there are a case where a person is searched for a video stored in advance, or a case where a person is searched for a video obtained in real time. There are fixed types installed on set walls, ceilings, etc., and mobile types installed on security robots, etc. According to the present invention, appropriate person search corresponding to any of them is realized. can do.

  Further, according to the present invention, for example, by the search scoring method described above, it is possible to search for a person shown in an accumulated video or a real-time video by a combination of various person characteristics and actions. In any combination, search results are output in descending order of matching with the search request. As a result, the search result can be output effectively, and the work of checking the video can be greatly reduced. Furthermore, according to the present invention, it is possible to perform a similarity search of persons between videos and display the result.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

DESCRIPTION OF SYMBOLS 10 Person search device 11 Input means 12 Output means 13 Accumulation means 14 Face area detection means 15 Human body area detection means 16 Identification means 17 Tracking means 18 Suspicious person detection means 19 Person information integration means 20 Frame information generation means 21 Search means 22 Screen generation Means 23 Notification means 24 Transmission / reception means 25 Control means 26 Imaging means 31 Tracking means 32 Face authentication means 33 Gender / age estimation means 34 Face hiding judgment means 35 Height estimation means 36 Color information extraction means 41 Initial screen 42 Date / time designation screen 43 Features / Action condition designation screen 44 Search result display screen 45 Video playback screen 46 Shared screen 51 Visitor search area 52 Suspicious person search area 53 Keyword search area 54-1 Condition designation search (search from person characteristics) area 54-2 Real-time search area 55 -1 radio button / date Fixed box 55-2 Location designation box 56 Search execution buttons 57 Hyperlink display area 58 Date designation area 59-1 Location designation area 59-2 Height designation area 60 Color information designation area 61 Gender designation area 62 Mask designation area 63 Sunglasses designation Area 64 Period designation area 65 Character designation area 66 Similar entertainer designation area 67 Search execution button group 68 Hyperlink display area 69 Various search areas 70 Search result number display area 71 Image display area when relevant person appears 72 Face image 73 Search result display area 74 Video confirmation button 75 Page selection icon 76 Hyperlink display area 77 Video display area 78 Text display area 79 Feature display area 80 Button area 81 Image display area 82 Text display area 83 Comment input area 91 Input Device 92 Output device 93 Drive device 94 Auxiliary storage device 95 Memory device 96 CPU (Central Processing Unit)
97 Network connection device 98 Recording medium

Claims (9)

In a person search device that extracts information about a person included in a video imaged by an imaging means and displays it on a screen,
Storage means for storing person feature information for each time-series image included in the video;
Search means for searching for a corresponding person based on conditions preset for the storage means;
Information corresponding to the search result obtained by the search means is extracted by the storage means, and personal features or actions set in advance corresponding to the extracted person, or patrol position information of the security robot provided with the imaging means Screen generation means for generating a screen for display;
An output unit that displays the screen generated by the screen generation unit and outputs information displayed on the screen to an external device ;
Face area detection means for detecting a face area for a person included in an image frame constituting the video;
Human body region detecting means for detecting a human body region for a person included in an image frame constituting the video;
Identification means for identifying a person in a plurality of image frames having different times using the detection result obtained by the face area detection means and the detection result obtained by the human body area detection means;
Tracking means for tracking the movement of the person obtained by the identification means;
A suspicious person detecting means for detecting a suspicious person from the tracking result obtained by the tracking means;
Person information integration means for integrating the feature information of the person identified by the identification means;
Frame information generating means for generating an integration result obtained by the person information integrating means for each image frame constituting the video,
The frame information generation means discretizes all feature data obtained from the image frame and counts the number of appearances for each person included in the video identified by the identification means. Generate human feature information normalized by the number of image frames,
The person search apparatus characterized in that the storage means stores the person feature information obtained by the frame information generation means . The human body region detecting means extracts color information of a predetermined range of the detected human body, converts the extracted color information into a color space set in advance based on human visual characteristics,
The person search apparatus according to claim 1 , wherein the storage unit stores the color information converted into the color space. The search means includes
Examples search condition, the input search request by the searcher, to convert the feature amount corresponding to the contents of the characteristic of the person, the maximum value of the normalized histogram and search request feature quantity, by the search request feature quantity 3. The person search according to claim 1 or 2 , wherein the person information stored by the storage means is searched, a distance from the feature amount given to the person concerned is calculated, and scoring of the correct answer rate is performed. apparatus. In a person search method for extracting information about a person included in an image captured by an imaging unit and displaying the information on a screen,
An accumulation procedure for accumulating human characteristic information in the accumulation means for each time-series image included in the video;
A search procedure for searching for a corresponding person based on conditions preset for the storage means;
Information corresponding to a search result obtained by the search procedure is extracted by the storage means, and personal features or behaviors set in advance corresponding to the extracted person, or patrol position information of a security robot provided with the imaging means is obtained. A screen generation procedure for generating a screen for display;
An output procedure for displaying the screen generated by the screen generation procedure and outputting information displayed on the screen to an external device ;
A face area detection procedure for detecting a face area for a person included in an image frame constituting the video;
A human body region detection procedure for detecting a human body region for a person included in an image frame constituting the video;
An identification procedure for identifying a person in a plurality of image frames at different times using a detection result obtained by the face region detection procedure and a detection result obtained by the human body region detection procedure;
A tracking procedure for tracking the movement of the person obtained by the identification procedure;
A suspicious person detection procedure for detecting a suspicious person from the tracking result obtained by the tracking procedure;
A person information integration procedure for integrating feature information of the person identified by the identification procedure;
A frame information generation procedure for generating an integration result obtained by the person information integration procedure for each image frame constituting the video;
For each person included in the video identified by the identification procedure, the frame information generation procedure discretizes all feature data obtained from the image frame and counts the number of appearances. Generate human feature information normalized by the number of image frames,
The person search method characterized in that the storage means stores the person feature information obtained by the frame information generation procedure . The human body region detection procedure extracts color information of a predetermined range of the detected human body, converts the extracted color information into a color space set in advance based on human visual characteristics,
The person search method according to claim 4 , wherein the storing procedure stores the color information converted into the color space. The search procedure is as follows:
Examples search condition, the input search request by the searcher, to convert the feature amount corresponding to the contents of the characteristic of the person, the maximum value of the normalized histogram and search request feature quantity, by the search request feature quantity 6. The person search according to claim 4 or 5 , wherein the person information stored by the storage means is searched, a distance from the feature amount assigned to the person concerned is calculated, and the correct answer rate is scored. Method. In a person search program in a person search apparatus that extracts information about a person included in a video imaged by an imaging means and displays the information on a screen,
Computer
Storage means for storing person feature information for each time-series image included in the video;
Search means for searching for a corresponding person based on conditions preset for the storage means;
Information corresponding to the search result obtained by the search means is extracted by the storage means, and personal features or actions set in advance corresponding to the extracted person, or patrol position information of the security robot provided with the imaging means Screen generation means for generating a screen for display;
An output unit that displays the screen generated by the screen generation unit and outputs information displayed on the screen to an external device ;
Face area detecting means for detecting a face area for a person included in an image frame constituting the video;
Human body region detection means for detecting a human body region for a person included in an image frame constituting the video;
Identification means for identifying persons of a plurality of image frames having different times using the detection result obtained by the face area detection means and the detection result obtained by the human body area detection means;
Tracking means for tracking the movement of the person obtained by the identification means;
A suspicious person detecting means for detecting a suspicious person from the tracking result obtained by the tracking means;
Person information integration means for integrating the feature information of the person identified by the identification means; and
The integration result obtained by the person information integration unit is caused to function as a frame information generation unit that generates each image frame constituting the video ,
The frame information generation means discretizes all feature data obtained from the image frame and counts the number of appearances for each person included in the video identified by the identification means. Generate human feature information normalized by the number of image frames,
The storage unit stores the person feature information obtained by the frame information generation unit . The human body region detecting means extracts color information of a predetermined range of the detected human body, converts the extracted color information into a color space set in advance based on human visual characteristics,
The person search program according to claim 7 , wherein the storage unit stores the color information converted into the color space. The search means includes
Examples search condition, the input search request by the searcher, to convert the feature amount corresponding to the contents of the characteristic of the person, the maximum value of the normalized histogram and search request feature quantity, by the search request feature quantity 9. The person search according to claim 7 or 8 , wherein the person information stored by the storage means is searched, the distance from the feature amount assigned to the person concerned is calculated, and the correct answer rate is scored. program.

Images