JP2010244090A - Person information extraction device, person information extraction method, and person information extraction program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To efficiently obtain various information related to a person included in a photographed image. <P>SOLUTION: A person information extraction device for extracting information related to a person included in a video photographed by an image pickup means includes: a face region detection means for detecting a face region with respect to one or more persons from a prescribed image in each of time-series images included in the video; a body region detection means for detecting a body region by using the face region obtained by the face region detection means; and a person information integration means for integrating the detection result obtained by the face region detection means and the detection result obtained by the body region detection means. The body region detection means includes a height estimation means for estimating the height of the person based on the installation position of the image pickup means, the angle of view, and a horizontal distance, based on the face region, between the image pickup means and the person. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to a person information extraction apparatus, a person information extraction method, and a person information extraction program, and more particularly to a person information extraction apparatus and person information for efficiently acquiring a variety of information related to a person included in a photographed image. The present invention relates to an extraction method and a person information extraction program.

  2. Description of the Related Art Conventionally, there has been a technique for performing image processing or the like on a video taken by an imaging means such as a camera and automatically detecting a person included in an image constituting the video. Among them, a technique for detecting a human face from an image has been rapidly developed.For example, in order to improve face discrimination accuracy, a skin color region is extracted from an input image, and based on the skin color region, the top of the head, There is an image processing apparatus that detects the positions of facial feature points such as eyes and mouth and determines whether or not the skin color region is a face from the detection result (see, for example, Patent Document 1).

  In addition, such face detection technology is applied to images obtained by surveillance cameras installed in guarded facilities such as banks, department stores, and convenience stores. It is used to detect crimes and prevent crimes.

JP 2004-5384 A

  By the way, it is preferable that the person detected by the above-described conventional method accumulates various kinds of information that can be read from the camera so that the highly accurate search can be efficiently performed in the image search process. However, there is a limit to the information that can be extracted from only the face information of a person. For example, when determining whether or not the person is a specific person such as a suspicious person, In the case of detection, more detailed person information is required.

  The present invention has been made in view of the above-described problems, and a person information extraction device, a person information extraction method, and a person for efficiently acquiring various types of information about a person included in a captured image An object is to provide an information extraction program.

  In order to solve the above problems, the present invention employs means for solving the problems having the following characteristics.

  In the present invention, in a person information extraction device that extracts information about a person included in a video captured by an imaging unit, a face for one or more persons from a predetermined image among time-series images included in the video A face area detecting means for detecting an area; a human body area detecting means for detecting a human body area using the face area obtained by the face area detecting means; a detection result obtained by the face area detecting means; and the human body area detection. Human information integration means for integrating the detection results obtained by the means, wherein the human body region detection means includes the imaging means and the person based on the installation position, the angle of view, and the face area of the imaging means. And a height estimating means for estimating the height of the person based on the horizontal distance between the person and the person. Thereby, it is possible to efficiently acquire various kinds of information related to the person included in the photographed image.

  In the present invention, the human body region detecting means includes color information extracting means for extracting color information relating to the person included in the human body region on the basis of the size of the face region.

  In the present invention, the color information extraction unit extracts at least one color information or image data from a head region, a skin region, an upper body region, and a lower body region on the basis of the size of the face region. And

  Further, the present invention is characterized in that the person information integration means includes frame information generation means for generating person information for each integrated person for each image frame in the video.

  Furthermore, in the present invention, in a person information extraction method for extracting information about a person included in a video taken by an imaging unit, one or more persons from a predetermined image among time-series images included in the video are extracted. A face area detection procedure for detecting a face area, a human body area detection procedure for detecting a human body area using the face area obtained by the face area detection procedure, a detection result obtained by the face area detection procedure, and the human body area A human information integration procedure that integrates the detection results obtained by the detection procedure, and the human body region detection procedure includes the imaging unit based on the installation position, the angle of view, and the face region of the imaging unit; It has a height estimation procedure for estimating the height of the person based on the horizontal distance to the person. Thereby, it is possible to efficiently acquire various kinds of information related to the person included in the photographed image.

  According to the present invention, the human body region detection procedure includes a color information extraction procedure for extracting color information related to the person included in the human body region on the basis of the size of the face region.

  In the present invention, the color information extraction procedure extracts at least one color information or image data from a head region, a skin region, an upper body region, and a lower body region based on the size of the face region. And

  In the present invention, the person information integration procedure includes frame information generation means for generating individual information for each integrated person for each image frame in the video.

  Further, according to the present invention, in a person information extraction program for extracting information about a person included in a video captured by an imaging unit, the computer is configured to select one or more from a predetermined image among time-series images included in the video. A face area detection means for detecting a face area for a person, a human body area detection means for detecting a human body area using the face area obtained by the face area detection means, and a detection result obtained by the face area detection means; The human body region detection unit functions as a person information integration unit that integrates the detection result obtained by the human body region detection unit, and the human body region detection unit is configured to use the imaging unit based on the installation position, the angle of view, and the face region. And a height distance estimating means for estimating the height of the person based on the horizontal distance from the person. Thereby, it is possible to efficiently acquire various kinds of information related to the person included in the photographed image. Also, by installing the program, the personal information extraction process of the present invention can be easily realized by a general-purpose personal computer or the like.

  In the present invention, the human body region detecting means includes color information extracting means for extracting color information relating to the person included in the human body region on the basis of the size of the face region.

  In the present invention, the color information extraction unit extracts at least one color information or image data from a head region, a skin region, an upper body region, and a lower body region on the basis of the size of the face region. And

  Further, the present invention is characterized in that the person information integration means includes frame information generation means for generating person information for each integrated person for each image frame in the video.

  ADVANTAGE OF THE INVENTION According to this invention, the various information regarding the person contained in the image | photographed image can be acquired efficiently.

It is a figure which shows an example of schematic structure of the person information extraction apparatus in this embodiment. It is a figure for demonstrating the example of height estimation in this embodiment. It is a figure for demonstrating the height estimation in case a some person exists in the image | photographed image. It is a figure which shows the specific example of the color information acquired in this embodiment. It is a figure which shows the example of a setting of the color information acquired when there exist several persons in the image | photographed image. It is a figure which shows an example of the frame information in this embodiment. It is a figure which shows an example of the hardware constitutions which can implement | achieve the person information extraction process in this embodiment. It is a flowchart which shows an example of the person information extraction process procedure in this embodiment.

<About the present invention>
The present invention acquires information about a person included in each image for each time-series image included in an image captured by an imaging unit such as a single camera. At that time, the face area of the person included in the image is first detected. Next, based on the detected face area, a human body area including various kinds of personal information such as height, hair, and clothes color information is detected. In the present invention, the images obtained by the above-described face area detection and human body area detection are integrated.
As a result, it is possible to extract various kinds of person information with high accuracy and realize high speed.

  Hereinafter, embodiments in which a person information extraction device, a person information extraction method, and a person information extraction program according to the present invention are suitably implemented will be described with reference to the drawings.

<Schematic configuration example of person information extraction device>
FIG. 1 is a diagram illustrating an example of a schematic configuration of a person information extraction device according to the present embodiment. 1 includes an input unit 11, an output unit 12, a storage unit 13, a face region detection unit 14, a human body region detection unit 15, a person information integration unit 16, and a frame information generation unit. Means 17, transmission / reception means 18, and control means 19 are provided. Note that the transmission / reception means 18 is connected to an imaging means 20 such as a camera for photographing a person for extracting person information, and acquires each time-series image included in the video photographed by the imaging means 20. be able to. Further, the imaging unit 20 may be configured integrally with the person information extraction device 10.

  The input unit 11 receives various instructions such as a face area detection instruction, a human body area detection instruction, a person information integration instruction, a frame information generation instruction, and a transmission / reception instruction from a user or the like. Note that the input unit 11 includes, for example, a keyboard, a pointing device such as a mouse, a voice input device such as a microphone, and the like.

  The output unit 12 includes the instruction content input by the input unit 11, control data generated based on each instruction content, the face region detection unit 14, the human body region detection unit 15, the person information integration unit 16, and the frame information generation unit. 17. The contents of various information obtained by the progress or result executed by each component such as the transmission / reception means 18 are displayed and the sound is output. The output unit 12 has a screen display function such as a display, a sound output function such as a speaker, and the like.

  The storage unit 13 can store various information for realizing the above-described embodiment, and reading and writing are performed as necessary. Specifically, the accumulating unit 13 includes various feature amount data used for face authentication, gender and age estimation, face area detection results in the face area detecting unit 14, and human body area detecting unit 15. Human body region detection result, person information integration result in person information integration means 16, frame information result in frame information generation means 17, information controlled by transmission / reception means 18 and control means 19, error information at the time of error occurrence, log information, book Each piece of information such as a program for realizing the invention is stored.

  When it is determined that the person is included in the image, the face area detecting unit 14 detects the face area of the person. The face area detection unit 14 acquires a video imaged by the imaging unit 20 such as a camera, and among the time-series images included in the acquired video, a predetermined image (for each frame image or several frames). The face of one or a plurality of persons is detected with respect to an image or the like at intervals.

  Specifically, the face area detection unit 14 acquires the feature amount of the face from position information such as eyes, nose, and mouth in the face included in the photographed image, for example, and detects it as a preset face. For example, a human face is detected by performing a matching process using a feature amount matching pattern. Further, the face area detection unit 14 is not limited to the above-described face detection processing, and for example, face detection by edge detection or shape pattern detection, face detection by hue extraction or skin color extraction, or the like can be used. Note that the face area detection unit 14 detects a rectangular face area having a vertical and horizontal width of the face in the image after detecting the face.

  The face area detecting means 14 detects the center coordinates (position information) of the face area and the size (size) of the area on the image, and synthesizes the face area with the original image with a predetermined shape. Accumulate various information that can be displayed on the screen so that can be clearly understood. In the present invention, the shape of the face region may be a rectangle, a circle, an ellipse, another polygon, a silhouette shape enlarged from the outer shape of a human face at a predetermined magnification, or the like.

  The face area detection unit 14 includes a tracking unit 21, a face authentication unit 22, a gender / age estimation unit 23, and a face hiding determination unit 24. The tracking means 21 performs person tracking using the detected face area, and assigns identification information (tracking ID) in order to identify and accumulate each of the plurality of face areas included in the image frame. Further, the tracking means 21 outputs four types of statuses of “unused”, “frame-in”, “frame-out”, and “tracking” for the allocated tracking ID depending on the tracking status.

  The face authentication unit 22 acquires a facial feature amount including each piece of arrangement information such as eyes, nose, and mouth from the detected face area, and stores the acquired feature amount in advance as a database (DB) or the like in the storage unit 13 or the like. Face authentication processing is performed using the accumulated facial features of persons such as registered persons (registered persons) and entertainers, and the identification information (ID) of the authenticated person is output. Further, the face authentication means 22 may convert the face image ID stored in the storage means 13 into the name of a person (registered person / celebrity) and output it.

  The gender / age estimation means 23 acquires a facial feature quantity from the detected face area, estimates the gender and age using the acquired facial feature quantity and the storage means 13, and outputs the result.

  The face hiding determination unit 24 detects whether or not a process for hiding the face such as a mask or sunglasses is performed on the detected face area. Specifically, the face hiding determination means 24 acquires color information of the eye area and mouth area estimated from the face area, and the color information of the eye area is assumed to be wearing sunglasses (for example, black). System), it is determined that the face is hidden by sunglasses, and when the mouth area is a color assumed to be wearing a mask (for example, white), the face is hidden by the mask. And the determination result is output.

  When the face area is detected by the face area detecting means 14, the human body area detecting means 15 is based on the face area information from a predetermined image (such as each frame image or an image spaced several frames apart). One or more human body regions are detected.

  Specifically, the human body region detection unit 15 compares, for example, successive image frames, and there is a place where the color information (luminance, chromaticity, etc.) changes within a predetermined time, and is further surrounded by the place. A human body region is detected when the region is equal to or larger than the predetermined region or when the moving range with time is within the predetermined range. The human body detection method is not limited to this in the present invention.

  The human body region detecting means 15 detects the center coordinates of the human body region and the size of the region on the image, and combines the face region with the original image with a predetermined shape so that the human body region can be clearly seen. Accumulate various information that can be displayed. The shape of the human body region may be a rectangle, a circle, an ellipse, another polygon, a silhouette shape enlarged at a predetermined magnification from the outer shape of a person, and the like, similar to the face region described above.

  The human body region detection unit 15 includes a height estimation unit 25, a color information extraction unit 26, and a person position estimation unit 27.

  The height estimation means 25 calculates the height of the person for each of the detected human body area and face area. A specific height estimation method in the present embodiment will be described later.

  The color information extraction unit 26 estimates the head region, skin region, upper body region, and lower body region of the person using the positional relationship between the detected region information of the human body region and the face region, and color information of each region To extract. Specifically, color information such as hair, upper garment, and lower garment is extracted, and a representative color (average color, etc.) of the extracted color information is determined as color information such as hair, upper garment, and lower garment.

  In the above-described calculation of the color information of each area, the total number of pixels and the appearance frequency of each area are counted, and, for example, RGB values for the top 10 colors are output. Here, each of the above-described areas is specifically the head area from the top of the face area, the cheek (skin) area from the left and right of the face area, and the upper body area from the bottom of the face area above the center of the human body area. The lower body region is extracted below the center of the human body region or above the ground. In addition, the setting content of the area | region extracted is not limited to this in this invention. Further, the color information extracting means 26 can cut out and output the image data of at least one of the face area, the human body area, the head area, the skin area, the upper body area, and the lower body area as it is. Specific examples of the color information acquired in this embodiment will be described later.

  The person position estimating means 27 calculates the position coordinates of the person in the real space for each of the detected human body area and face area. In this case, the person position estimating means 27 calculates the position coordinates (X, Y, Z) in the three-dimensional real space by a conversion formula set in advance for the two-dimensional coordinates obtained from the image taken by the camera. get.

  The person information integration unit 16 associates the face area and the human body area as the same person, and integrates the characteristics of the person. Specifically, the person information integration unit 16 acquires the center of gravity coordinates of the face area in the image, and when there is a human body area that includes the acquired coordinates, the face area and the human body area are from the same person. Associating as there is.

  Furthermore, when one human body region includes two or more face regions, the person information integration unit 16 associates the same human body region with all the face regions. Thereby, even when a plurality of persons temporarily overlap on the screen during the tracking process or the like, it can be continuously tracked without interruption. Persons are managed using identification information such as IDs. Note that the processing for associating the human body region and the face region as those of the same person is not limited to this in the present invention. For example, the posture and orientation of the person are extracted and the extracted information is used. May be associated with each other.

  The frame information generation unit 17 generates frame information in which the person information for each person integrated by the person information integration unit 16 described above is stored for each frame of the image included in the video. As a result, the person can be managed in units of frames, and the time information of the shots is given for each frame. Various searches can be realized with high accuracy using search keywords. Note that a specific configuration of frame information in the present embodiment will be described later.

  The transmission / reception means 18 receives a monitoring video from a single imaging means 20 via a communication network such as a LAN (Local Area Network) or the Internet. Note that the transmission / reception means 18 does not have to receive the monitoring video directly from the imaging means 20. For example, the transmission / reception means 18 temporarily stores the video acquired in advance by the imaging means 20 somewhere and uses the saved information. The person information extraction in this embodiment may be performed.

  Moreover, the transmission / reception means 18 can be used as a communication interface for transmitting to other terminals constituting the person information extracting apparatus 10 and receiving various data from other terminals.

  The control means 19 controls the entire functional configuration of the person information extraction apparatus 10. Specifically, the control unit 19 performs face area detection, human body area detection, person information integration, or frame information generation based on input information from the user input by the input unit 11. And various controls such as performing transmission / reception control.

<Example of height estimation in this embodiment>
Here, an example of height estimation in the present embodiment will be specifically described. FIG. 2 is a diagram for explaining an example of height estimation in the present embodiment. As shown in FIG. 2, a camera that is the imaging means 20 is installed at a predetermined location, and the height is estimated for a person 30 existing within the angle of view θ of the camera. In the example of FIG. 2, the face area shape 31 and the human body area shape 32 are shown because there are all of the human face and human body in the angle of view θ.

  Specifically, in the present embodiment, the height H is estimated by, for example, the following expression (1).

ここで、上述した（１）式において、ＣａｍＺはカメラを設置した地上からの高さを示し、Ｄは現在のカメラと人物３０との水平距離を示しており、αはＣａｍＺの高さを基準とした水平線に対する顔領域の中心までの角度を示している。また、上述のＤは、以下に示す（２）式で算出される。

Here, in the above-described equation (1), CamZ indicates the height from the ground where the camera is installed, D indicates the horizontal distance between the current camera and the person 30, and α is the height of CamZ. The angle to the center of the face area with respect to the horizontal line is shown. The above D is calculated by the following equation (2).

ここで、上述した（２）式において、Ｓは検出された顔領域３１のサイズ（１辺の長さ（ｐｉｘｅｌｓ））を示し、θ_Ｈは、現在のカメラの水平画角を示し、θ_Ｖは現在のカメラの垂直画角を示し、Ｄ_ｂは基準値測定時のカメラと人物の水平距離を示し、Ｓ_ｂは基準値測定時に検出された顔領域のサイズ（１辺の長さ（ｐｉｘｅｌｓ））を示し、θ_Ｈｂは基準値測定時のカメラの水平画角を示し、θ_Ｖｂは基準値測定時のカメラの垂直画角を示している。

Here, in the above equation (2), S indicates the size of the detected face region 31 (length of one side (pixels)), θ _H indicates the horizontal angle of view of the current camera, and θ _V Indicates the vertical angle of view of the current camera, D _b indicates the horizontal distance between the camera and the person at the time of measuring the reference value, and S _b indicates the size of the face area (length of one side (pixels) detected at the time of measuring the reference value. )), Θ _Hb represents the horizontal angle of view of the camera when measuring the reference value, and θ _Vb represents the vertical angle of view of the camera when measuring the reference value.

  That is, in the present embodiment, the distance D to the camera is estimated from the size of the face in the image using a perspective method, and the height is estimated as described above based on the distance D between the camera and the person. In the present embodiment, it is necessary to measure the reference value in advance in order to estimate the height. The reference values are the camera angle of view when the distance between the camera and the person is known, and the horizontal distance between the camera and the person.

  FIG. 3 is a diagram for explaining the height estimation when there are a plurality of persons in the photographed image. In FIG. 3, two persons 30-1 and 30-2 are photographed as a plurality of examples, and the face regions 31-1 and 31-2 can be extracted from the photographed images. Since the whole body is not included in the angle of view θ of the camera as the means 20, the human body region shape is not extracted. However, even in such a case, the height can be estimated.

  Specifically, for the photographed persons 30-1 and 30-2, the horizontal distances D1 and D2 between the camera and the person obtained from the size of the face area in the image, and the camera height (CamZ). Height can be estimated as angles α1 and α2 to the centers of the face areas 31-1 and 31-2 with respect to the reference horizontal line.

  Various estimation methods other than the above-described height estimation method can be used. For example, when one or a plurality of persons included in an image are detected and the person is present, whether or not the person is a new person is determined. Judging, if it is a new person, calculate the position of the person in real space from the tip of the foot of the human body region and the vanishing point corresponding to the image, together with the apparent size on the image, You can also estimate the actual height of a person. The vanishing point may be included in the image, but may not be included in the image depending on the camera angle or the like. In this case, a vanishing point is set on a virtual space and the vanishing point is used.

<Specific example of color information acquired in this embodiment>
Next, a specific example of color information acquired in this embodiment will be described. FIG. 4 is a diagram illustrating a specific example of the color information acquired in the present embodiment. For example, when the person image shown in FIG. 4 can be acquired from the captured image, the head region 41 for acquiring the hair color is acquired from the head 30h of the person. As shown in FIG. 4, the head region 41 is at the top with respect to the position of the face region 31, and at least one within a range of d / 2 with respect to the length d in the vertical direction (one side). It is set (one in the example of FIG. 4). Further, as shown in FIG. 4, the upper body area 42 is at the lower part with respect to the position of the face area 31, and at least one of the upper body 30 b of the person is set within a range of d with respect to the length d. (One in the example of FIG. 4).

  That is, the area shown in FIG. 4 can be determined based on the size of the face area detected by the face area detecting means 14. In addition, the setting content of each area | region is not limited to the said content in this invention, For example, it can also set with the height from the ground on the basis of the human body area | region mentioned above. In this case, since the area information is in real space, it is necessary to convert it into position coordinates on the image plane.

  FIG. 5 is a diagram illustrating an example of setting color information acquired when there are a plurality of persons in a captured image. In the screen 50 of FIG. 5, two persons 30-1 and 30-2 are photographed as a plurality of examples, and face areas 31-1 and 31-2 are extracted from the photographed images, respectively. In this case, the head regions 41-1 and 41-2 are set based on the positions of the face regions 31-1 and 31-2, and the upper body regions 42 are further set. Get the color information of the area. In the example of FIG. 5, in the case of the person 30-1, upper body areas 42-1a, 42-1b, and 42-1c are set, and in the case of the person 30-2, the upper body areas 42-2a and 42. -2b is set. In addition, the area | region and number set in this embodiment are not limited to FIG. 5, For example, a skin color and a lower body area | region can also be set. Further, not only the color information of each area but also the image of the area itself may be cut out and stored.

  As described above, the color information can be acquired for each area by, for example, the RGB value of the average color. However, the color information is not converted to the RGB value but is converted into an HSV value that is a color space in consideration of visual characteristics and is output. May be.

<Specific Frame Information Configuration Example in the Present Embodiment>
Next, a configuration example of frame information in the above-described frame information generation unit 17 will be described. FIG. 6 is a diagram illustrating an example of frame information in the present embodiment. The frame information generation unit 17 extracts information shown in FIG. 6 for the person detected by performing processing for each frame. Here, the frame information shown in FIG. 6 includes, for example, “file name” and “detection date / time” as common items, and “person position coordinates (X, Y, Z)” as information extracted from the face area, "Height information", "various color information", "registrant information", "similar celebrity information", "age information", "sex information", "face orientation (PAN, TILT)", "mask" “Sunglasses”, “acquired face image information”, “personal position coordinates (X, Y, Z)”, “height information”, “various color information”, and the like.

  The “file name” stores the file name of the video file currently being processed. In addition, “date and time of detection” stores the date and time when a person was detected. As “detection date and time”, since the video file name corresponds to the actual recording start time, the actual time is stored by adding the shooting start time and the time in the video.

  Further, the “person position coordinates (X, Y, Z)” extracted from the face area stores the XY coordinates of the camera position and the XYZ coordinates of the person's feet in the real space with the ground as the origin. That is, the position coordinates in the real space of the person calculated from the face area are stored in “person position coordinates (X, Y, Z)”. Also, the height calculated from the face area is stored in the “height information”. The “various color information” stores the total number of pixels of the head and upper body, the top 10 colors, and the frequency calculated from the face area.

  As the head color information, RGB numerical values and frequencies from the first to the tenth frequency for the head calculated from the face area are stored. Further, the total number of pixels of the head region calculated from the face region is stored as the total number of pixels of the head. Further, as the upper body color information, RGB numerical values and frequencies of the upper body calculated from the face area from the 1st frequency to the 10th frequency are stored. Further, the total number of pixels of the upper body area calculated from the face area is stored as the total number of pixels of the upper body.

  In “registrant information”, the closest person name and authentication score are set as a result of collation with the registered person DB. Specifically, when authentication is performed as a registrant (exceeding the authentication threshold) by collation with the registered person DB, the names of the top 10 registrants are stored. If not authenticated, “unregistered person” is stored. Further, the authentication result score is indicated by an integer value from 0 to 1000 for the top 10 authentication scores, and -1 is stored if the result of collation is “unregistered”.

  Further, in “similar celebrity information”, the closest person name and authentication score are stored as a result of collation with a DB in which celebrities are registered. Specifically, the name of the entertainer is stored when the authentication score is the highest and is above the threshold value by collation with the entertainer DB. If not authenticated, “similar” is stored. The authentication result score is the same as described above.

The “age information” stores the age and reliability of the detected person. The “sex information” stores the sex and reliability of either gender. Further, the “face orientation” stores a face orientation angle with respect to camera work such as pan and tilt as the origin. Specifically, the angle of the face viewed from the camera is stored with the camera as the origin, and the upward direction and the left direction viewed from the camera are +.
Further, “mask” stores whether or not the periphery of the mouth is hidden by a mask or the like, and specifically stores “Yes”, “No”, or “Unknown”. Further, “Sunglasses” stores whether or not the periphery of the eyes is hidden by sunglasses or the like, and specifically, “Yes”, “No”, or “Unknown” is stored.

  The “acquired face image information” stores the file path to the detected face image and the reliability, and specifically, the file path to the location where the face image detected in this frame is stored. And the length of one side of the detected face area, the authentication result name having the highest authentication score equal to or higher than the threshold, the reliability, and the like are stored.

  Further, in the “person position coordinates (X, Y, Z)” extracted from the human body region, the XY coordinates of the camera position and the XYZ coordinates of the human foot in the real space with the ground as the origin are stored. The position coordinates of the person in the real space calculated from the above are stored.

  Also, the height calculated from the human body area / face area is stored in the “height information”. Further, the “various color information” stores the total number of pixels of the head, upper body, and lower body, the top 10 colors, and the frequency calculated from the human body region. Specifically, the frequency for the head is ranked from 1st to 10th. RGB values and frequencies up to, the total number of pixels in the head region, the RGB numbers and frequencies from the first to the 10th frequency for the upper body, the total number of pixels in the upper body region, the frequencies from the first to the 10th in the lower body At least one of the RGB numerical value and frequency, and the total number of pixels in the lower body region are stored.

  In the present embodiment, the above-described information is output in units of frames as a result of various processes. If neither a person area nor a face area is detected, nothing is output. Some information may not be output depending on the detection state of the person area and the face area. In that case, a value indicating “unknown” is output.

  In the present embodiment, the above-described information is generated for each frame, and a tag is assigned to each data for which the frame information is output. This makes it possible to clarify what information each data corresponds to. In addition, when a plurality of persons are detected in one frame, different tracking IDs (identification information) are allocated for each person, and the same date and time are assigned to each tracking ID. Is output.

<Hardware configuration example>
Here, the person information extraction apparatus 10 described above can be controlled by a dedicated apparatus configuration having the functions described above, but generates an execution program (person information extraction program) that can cause a computer to execute each function. For example, by installing the execution program in a general-purpose personal computer, server, or the like, the personal information extraction processing in the present invention can be realized.

  A hardware configuration example of a computer capable of realizing the person information extraction process in the present embodiment will be described with reference to the drawings. FIG. 7 is a diagram illustrating an example of a hardware configuration capable of realizing the person information extraction process according to the present embodiment.

  7 includes an input device 61, an output device 62, a drive device 63, an auxiliary storage device 64, a memory device 65, a CPU (Central Processing Unit) 66 for performing various controls, and a network connection device. 67, which are connected to each other by a system bus B.

  The input device 61 has a pointing device such as a keyboard and a mouse operated by a user or the like, and inputs various operation signals such as execution of a program from the user or the like.

  The output device 62 has a monitor for displaying various windows and data necessary for operating the computer main body for performing the processing in the present invention, and displays the program execution progress, results, etc. by the control program of the CPU 66. can do.

  The input device 61 and the output device 62 may be integrated input / output means such as a touch panel. In this case, the input device 61 and the output device 62 are predetermined using a finger of a user, a pen-type input device, or the like. The position can be touched to input.

  Here, the execution program installed in the computer main body in the present invention is provided by, for example, a portable recording medium 68 such as a USB (Universal Serial Bus) memory or a CD-ROM. The recording medium 68 on which the program is recorded can be set in the drive device 63, and the execution program included in the recording medium 68 is installed from the recording medium 68 to the auxiliary storage device 64 via the drive device 63.

  The auxiliary storage device 64 is a storage means such as a hard disk, and can store an execution program according to the present invention, a control program provided in a computer, and the like, and can perform input / output as necessary.

  The memory device 65 stores an execution program read from the auxiliary storage device 64 by the CPU 66. The memory device 65 includes a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU 66 performs various operations and data input / output with each hardware component based on a control program such as an OS (Operating System) and an execution program read and stored by the memory device 65. Each process can be realized by controlling the process. Various information necessary during the execution of the program can be acquired from the auxiliary storage device 64, and the execution result and the like can also be stored.

  The network connection device 67 is connected to a communication network or the like to acquire an execution program from another terminal connected to the communication network or the execution result obtained by executing the program, or in the present invention. The execution program itself can be provided to other terminals. With the above-described hardware configuration, the person information extraction process according to the present invention can be executed. Also, by installing the program, the personal information extraction process of the present invention can be easily realized by a general-purpose personal computer or the like. Next, specific contents of the person information extraction process will be described.

<Person information extraction process>
Next, a person information extraction processing procedure executed by the execution program (person information extraction program) according to the present invention will be described with reference to a flowchart. FIG. 8 is a flowchart showing an example of a person information extraction processing procedure in the present embodiment.

  In FIG. 8, first, an image taken by an imaging means such as a camera is input (S01). Next, it is determined whether or not one or more persons included in the video are detected (S02). When a person is detected (YES in S02), the face area detection process (S03) and the human body described above are performed. An area detection process (S04) is performed. Note that the process of S02 can be determined based on, for example, whether or not the area where the color information is changed is greater than or equal to a predetermined size in the comparison between time-series image frames, and is combined with the process of S03. Thus, the determination can be made based on whether or not the face area has been detected.

  Further, after the processes of S03 and S04 are completed, a person information integration process is performed (S05), frame information is generated from the integrated information (S06), and the generated information is stored (S07). Further, after the process of S07 is completed or when no person is detected in the process of S02 (NO in S02), it is next determined whether or not the video is finished (S08), and the video is not finished. (NO in S08), the process of S02 is returned, the video is continuously reproduced, and the subsequent process is performed.

  As described above, according to the present invention, it is possible to efficiently acquire various types of information related to a person included in a captured image.

  Specifically, for example, it becomes possible to extract the characteristics of a person appearing in the video of a surveillance camera, etc., and estimate the height of the criminal at the time of the crime, the color of clothes worn, etc. It is possible to provide prompt information. In addition, when searching for a person in the stored video, it is possible to search using the height and color information as keys. Moreover, it becomes possible to detect a specific person by combining the estimated information and other information (age, gender, etc.). Furthermore, by taking statistical information such as hair color and clothing color, it can be applied to marketing and the like.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

DESCRIPTION OF SYMBOLS 10 Person information extraction apparatus 11 Input means 12 Output means 13 Accumulation means 14 Face area detection means 15 Human body area detection means 16 Person information integration means 17 Frame information generation means 18 Transmission / reception means 19 Control means 19 Person information extraction apparatus 20 Imaging means 21 Tracking Means 22 Face authentication means 23 Gender / age estimation means 24 Face hiding judgment means 25 Height estimation means 26 Color information extraction means 27 Human position estimation means 30 Person 31 Face area 32 Human body area 41 Head area 42 Upper body area 50 Screen 61 Input device 62 Output Device 63 Drive Device 64 Auxiliary Storage Device 65 Memory Device 66 CPU (Central Processing Unit)
67 Network connection device 68 Recording medium

Claims (12)

In a person information extraction device that extracts information about a person included in a video imaged by an imaging means,
A face area detecting means for detecting a face area for one or a plurality of persons from a predetermined image among time-series images included in the video;
Human body region detecting means for detecting a human body region using the face region obtained by the face region detecting unit;
Human information integration means for integrating the detection result obtained by the face area detection means and the detection result obtained by the human body area detection means;
The human body region detection means includes a height estimation means for estimating the height of the person based on an installation position of the imaging means, an angle of view, and a horizontal distance between the imaging means and the person based on the face area. A person information extracting device characterized by comprising: The human body region detecting means includes
The person information extracting apparatus according to claim 1, further comprising color information extracting means for extracting color information relating to the person included in the human body area on the basis of the size of the face area. The color information extracting means includes
3. The person information extracting apparatus according to claim 2, wherein at least one color information or image data is extracted from the head region, the skin region, the upper body region, and the lower body region on the basis of the size of the face region. .   4. The person according to claim 1, further comprising a frame information generation unit configured to generate person information for each integrated person for each image frame in the video. Information extraction device. In a person information extraction method for extracting information about a person included in a video photographed by an imaging means,
A face area detection procedure for detecting a face area for one or a plurality of persons from a predetermined image among time-series images included in the video;
A human body region detection procedure for detecting a human body region using the face region obtained by the face region detection procedure;
A human information integration procedure for integrating the detection result obtained by the face region detection procedure and the detection result obtained by the human body region detection procedure;
The human body region detection procedure includes a height estimation procedure for estimating a height of the person based on an installation position, an angle of view of the imaging unit, and a horizontal distance between the imaging unit and the person with reference to the face region. A person information extraction method characterized by comprising: The human body region detection procedure includes:
6. The person information extraction method according to claim 5, further comprising a color information extraction procedure for extracting color information related to the person included in the human body area on the basis of the size of the face area. The color information extraction procedure includes:
7. The person information extraction method according to claim 6, wherein at least one color information or image data is extracted from the head region, the skin region, the upper body region, and the lower body region with reference to the size of the face region. .   8. The person according to claim 5, further comprising: frame information generation means for generating person information for each person integrated in the person information integration procedure for each image frame in the video. Information extraction method. In a person information extraction program for extracting information about a person included in a video shot by an imaging means,
Computer
A face area detecting means for detecting a face area for one or a plurality of persons from a predetermined image among time-series images included in the video;
Human body region detecting means for detecting a human body region using the face region obtained by the face region detecting unit; and
Function as human information integration means for integrating the detection result obtained by the face area detection means and the detection result obtained by the human body area detection means;
The human body region detection means includes a height estimation means for estimating the height of the person based on an installation position of the imaging means, an angle of view, and a horizontal distance between the imaging means and the person based on the face area. A personal information extraction program characterized by comprising: The human body region detecting means includes
The person information extraction program according to claim 9, further comprising color information extraction means for extracting color information about the person included in the human body area on the basis of the size of the face area. The color information extracting means includes
The person information extraction program according to claim 10, wherein at least one color information or image data is extracted from a head region, a skin region, an upper body region, and a lower body region on the basis of the size of the face region. .   12. The person according to claim 9, further comprising a frame information generation unit configured to generate the person information for each person integrated for each image frame in the video. Information extraction program.

Images