JP6090847B2 - Information processing apparatus and determination method - Google Patents

Description

  The present invention relates to a technique for handling a three-dimensional shape of a determination target indicated by three-dimensional information.

  There has been proposed a method for determining the legitimacy of a person to be determined using the characteristics of the three-dimensional shape of a human face. In Patent Document 1 below, a three-dimensional face feature amount that is information on the shape of a predetermined number of three-dimensional lines on the three-dimensional shape of the entire face of the person to be authenticated is calculated, and the three-dimensional face feature amount and the comparison face are calculated. A method for comparing with feature amounts has been proposed. Non-Patent Documents 1 and 2 below propose a three-dimensional face authentication technique that uses partial structure information of a face instead of the structure information of the entire face.

JP 2008-123216 A

Akihiro Hayasaka et al., “Examination of three-dimensional face recognition using partial facial structure”, 22nd Signal Processing Symposium, November 7-9, 2007 (Sendai) Akihiro Hayasaka et al., “Study of 3D face recognition algorithm robust to facial expression changes”, 21st Circuit and System Karuizawa Workshop, April 21-22, 2008

  Each of the proposed methods described above is not suitable for high-speed processing because it determines the legitimacy of the person to be judged based on the distance between the 3D point cloud data and the similarity between the feature vectors (features). There is sex. For example, it is difficult to apply the above-described proposed method to the one-to-N authentication with a large parameter N or a function for detecting a specific person from a moving image in which a plurality of people are captured.

  The present invention has been made in view of such circumstances, and provides a technique for rapidly determining the suitability of the determination condition in the determination target using three-dimensional information.

  Each aspect of the present invention employs the following configurations in order to solve the above-described problems.

  A 1st aspect is related with information processing apparatus. The information processing apparatus according to the first aspect includes the information acquisition unit that acquires the depth information and the two-dimensional image forming the three-dimensional information of at least a part of the determination target, and the two-dimensional image acquired by the information acquisition unit. Based on the two-dimensional feature extraction unit that extracts the two-dimensional feature information of the determination target and the depth information acquired by the two-dimensional feature information and the information acquisition unit, the feature classification of the three-dimensional shape of the determination target is shown A three-dimensional feature extraction unit for extracting feature classification information, a condition acquisition unit for acquiring two-dimensional feature information and feature classification information as determination conditions, and a two-dimensional feature information and a three-dimensional feature extracted by the two-dimensional feature extraction unit A determination unit that determines suitability of the determination condition in the determination target object by comparing the feature classification information extracted by the extraction unit with the two-dimensional feature information and the feature classification information as the determination condition. That.

  The second aspect relates to a determination method executed by at least one computer. In the determination method according to the second aspect, the depth information and the two-dimensional image forming at least a part of the three-dimensional information of the determination target are acquired, and the two-dimensional feature information of the determination target is obtained from the acquired two-dimensional image. Extracting the feature classification information indicating the feature classification of the three-dimensional shape of the object to be determined based on the two-dimensional feature information and the depth information, and acquiring the two-dimensional feature information and the feature classification information as a determination condition; The extracted two-dimensional feature information and the extracted feature classification information are compared with the two-dimensional feature information and the feature classification information as the determination conditions, and by this comparison, the suitability of the determination conditions in the determination target is determined. Including judging.

  As another aspect of the present invention, a computer program that causes at least one computer to execute the determination method according to the second aspect may be used, or a computer-readable recording medium that records such a program. There may be. This recording medium includes a non-transitory tangible medium.

  According to each of the above aspects, it is possible to provide a technique for rapidly determining the suitability of the determination condition in the determination object using the three-dimensional information.

It is a figure which shows notionally the structural example of the information processing apparatus which concerns on embodiment of this invention. It is a figure which shows notionally the hardware structural example of the face authentication apparatus in 1st Embodiment. It is a figure which shows notionally the process structural example of the face authentication apparatus in 1st Embodiment. It is a flowchart which shows the operation example of the face authentication apparatus in 1st Embodiment. It is a figure which shows notionally the hardware structural example of the monitoring image | video analysis apparatus in 2nd Embodiment. It is a figure which shows notionally the process structural example of the monitoring video analyzer in 2nd Embodiment. It is a flowchart which shows the operation example of the monitoring video analyzer in 2nd Embodiment.

  Embodiments of the present invention will be described below. In addition, embodiment mentioned below is an illustration and this invention is not limited to the structure of the following embodiment.

  FIG. 1 is a diagram conceptually illustrating a configuration example of an information processing apparatus 100 according to an embodiment of the present invention. As illustrated in FIG. 1, the information processing apparatus 100 is acquired by an information acquisition unit 101 that acquires depth information and a two-dimensional image that form three-dimensional information of at least a part of a determination target, and an information acquisition unit 101. Based on the two-dimensional feature extraction unit 102 that extracts the two-dimensional feature information of the determination target object from the two-dimensional image and the depth information acquired by the two-dimensional feature information and the information acquisition unit 101. A three-dimensional feature extraction unit 103 that extracts feature classification information indicating a feature classification of a three-dimensional shape, a condition acquisition unit 104 that acquires two-dimensional feature information and feature classification information serving as determination conditions, and a two-dimensional feature extraction unit 102 By comparing the extracted two-dimensional feature information and the feature classification information extracted by the three-dimensional feature extraction unit 103 with the two-dimensional feature information and the feature classification information as the determination conditions, It includes a determination unit 105 determines the suitability of the determination condition in the determination target object, the.

  The information processing apparatus 100 has, for example, the same hardware configuration as the face authentication apparatus 10 and the analysis apparatus 20 in the detailed embodiment described later, and the program is processed in the same manner as the face authentication apparatus 10 and the analysis apparatus 20. Thus, each processing unit described above is realized.

  The determination method according to the embodiment of the present invention is executed by at least one computer. The determination method acquires depth information and a two-dimensional image forming at least a part of the three-dimensional information of the determination target, extracts the two-dimensional feature information of the determination target from the acquired two-dimensional image, Based on the two-dimensional feature information and the depth information, feature classification information indicating the feature classification of the three-dimensional shape of the object to be determined is extracted, two-dimensional feature information and feature classification information serving as determination conditions are acquired, and the extracted 2 Including comparing the two-dimensional feature information and the feature classification information as the determination condition with the two-dimensional feature information and the extracted feature classification information, and determining the suitability of the determination condition in the determination target by this comparison. . However, each step included in the present determination method may be executed sequentially in random order, or may be executed simultaneously.

  In the present embodiment, the two-dimensional feature information of the object to be determined is extracted from the two-dimensional image forming the three-dimensional information, and the feature classification of the object to be determined from the two-dimensional feature information and the depth information forming the three-dimensional information. Information is extracted. Here, the two-dimensional feature information means information indicating the feature of the object to be determined in the two-dimensional space, such as two-dimensional position information or color information of a characteristic part of the object to be determined. When the determination target is a person, examples of the characteristic part include eyes, nose, mouth, and the like, and positional information such as the corner of the eye, the top of the eye, and the center of the eye is exemplified as the two-dimensional feature information. Further, when the object to be determined is a vehicle, examples of the characteristic part include a headlight and a windshield, and positional information of the center point of the headlight and the windshield is exemplified as two-dimensional feature information.

  The feature classification information means information indicating a classification related to the shape feature of the determination target in the three-dimensional space. When the object to be determined is a person, the feature classification information is, for example, a suspended eye, a hanging eye, a single, a double, a cut length, or the like. When the determination target is a vehicle, the feature classification information includes a sedan, a coupe, a 1 BOX wagon, a station wagon, an SUV, and the like. However, specific forms of the two-dimensional feature information and feature classification information are not limited. Further, the two-dimensional feature information and the feature classification information may be extracted from the entire object to be determined, or may be extracted from a part (characteristic part) of the object to be determined.

  In the present embodiment, the two-dimensional feature information and the feature classification information extracted as described above are compared with the two-dimensional feature information and the feature classification information acquired as the determination conditions. The suitability of the conditions is determined. For example, if the determination condition is the target two-dimensional feature information and feature classification information to be found, whether or not the determination target obtained from the two-dimensional image and depth information is the target is determined. In addition, when the determination condition is two-dimensional feature information and feature classification information of a person (a legitimate person) registered in advance, whether or not the person to be determined obtained from the two-dimensional image and the depth information is a legitimate person The suitability of is determined.

  As described above, according to the present embodiment, the suitability of the determination condition in the determination target object is based on both the two-dimensional feature information obtained from the two-dimensional image and the feature classification information indicating the feature of the three-dimensional shape. Since the determination is made, the determination accuracy can be improved as compared with the method of performing the determination only with the two-dimensional information or only with the three-dimensional information. Furthermore, according to the present embodiment, since feature classification information obtained by categorizing the shape features in the three-dimensional space is used for the determination, the distance between the three-dimensional point cloud data and the similarity between the feature vectors are calculated. Compared with the technique, the determination process can be simplified, and thus the determination process can be speeded up.

  Hereinafter, the details of the above-described embodiment will be described. Each detailed embodiment shown below is an example when the configuration of the information processing apparatus and the determination method in the above-described embodiment are applied to a face authentication apparatus and a monitoring video analysis apparatus. The above-described embodiment is not limited to application to target detection from images or biometric authentication, and can be applied to various determination techniques using three-dimensional information such as an object or a living thing.

[First Embodiment]
〔Device configuration〕
FIG. 2 is a diagram conceptually illustrating a hardware configuration example of the face authentication device 10 according to the first embodiment. The face authentication device 10 according to the first embodiment is a so-called computer, and includes, for example, a CPU (Central Processing Unit) 2, a memory 3, an input / output interface (I / F) 4, and the like that are connected to each other via a bus 5. . The memory 3 is a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk, a portable storage medium, or the like.

  As shown in FIG. 2, the input / output I / F 4 is connected to the three-dimensional sensor 7 and the like. The first embodiment does not limit devices connected to the input / output I / F 4, but includes an input device that accepts user operation input such as a keyboard and a mouse, a display device such as a display, a printer, You may connect with the communication apparatus etc. which communicate with another computer via a network. The hardware configuration of the face authentication device 10 is not limited.

  The three-dimensional sensor 7 detects at least three-dimensional information of the face of the person to be determined. The detected three-dimensional information includes information on the two-dimensional image (image) of the subject obtained by visible light and information on the distance from the three-dimensional sensor 7 (depth information). The three-dimensional sensor 7 is realized by a visible light camera and a distance image sensor like Kinect (registered trademark), for example. A distance image sensor, also called a depth sensor, irradiates a subject with a near-infrared light pattern from a laser and captures the pattern with a camera that detects the near-infrared light. The distance (depth) to the person is calculated.

  The method for realizing the three-dimensional sensor 7 is not limited, and the three-dimensional sensor 7 may be realized by a three-dimensional scanner method using a plurality of visible light cameras. In FIG. 2, only one three-dimensional sensor 7 is illustrated. However, as the three-dimensional sensor 7, a plurality of visible light cameras that capture a two-dimensional image of the subject and a depth sensor that detects a distance to the subject are used. May be provided.

[Processing configuration]
FIG. 3 is a diagram conceptually illustrating a processing configuration example of the face authentication apparatus 10 in the first embodiment. The face authentication device 10 according to the first embodiment includes an information acquisition unit 11, a two-dimensional feature extraction unit (hereinafter referred to as a 2D extraction unit) 12, a three-dimensional feature extraction unit (hereinafter referred to as a 3D extraction unit) 13, It has a condition acquisition unit 14, a determination unit 15, a feature information storage unit 16, and the like. Each of these processing units is realized, for example, by executing a program stored in the memory 3 by the CPU 2. Further, the program may be installed from a portable recording medium such as a CD (Compact Disc) or a memory card or another computer on the network via the input / output I / F 4 and stored in the memory 3. Good.

  The information acquisition unit 11 acquires three-dimensional information of the face of the person to be determined from the three-dimensional sensor 7. This three-dimensional information includes a two-dimensional image in which the face of the person to be determined is at least partially captured, and depth information regarding the subject of the two-dimensional image. Hereinafter, 3D may be expressed as 3D, and 2D may be expressed as 2D.

  The 2D extraction unit 12 extracts 2D feature information related to the target portion of the person to be determined from the 2D image acquired by the information acquisition unit 11. For example, the 2D extraction unit 12 extracts 2D feature information related to at least one part of the face of the person to be determined, such as eyes, nose, and mouth. In order to increase the determination accuracy, it is desirable to extract 2D feature information of a plurality of parts. The 2D extraction unit 12 extracts, as the 2D feature information of the eyes, the positions of the eye corners, the eyes and the center of the eyes (2D positions) for both eyes. Further, the 2D extraction unit 12 extracts both edges of the nose wing and the nose tip that determine the nose width as 2D feature information of the nose. In the first embodiment, the 2D extraction unit 12 does not limit the face part from which the 2D feature information is extracted or the specific content of the 2D feature information. A known method may be used as a method for extracting 2D feature information from a 2D image.

  By the way, when the 2D feature information indicates the 2D position, the 2D feature information of the person to be determined and the 2D feature information of the registrant stored in the feature information storage unit 16 described later are obtained from the 2D image in the same 2D coordinate space. Extracted based on The unification of the 2D coordinate space is, for example, that each 2D image that is the basis of the 2D feature information is captured while the distance from the three-dimensional sensor 7 to the subject (determination person or registrant) is constant. It is realized with. In addition, for example, each 2D image includes a common marker (measure) along with the subject, and each 2D feature information obtained from each 2D image is normalized based on the 2D distance indicated by the common marker. Can be realized.

  Based on the 2D feature information extracted by the 2D extraction unit 12 and the depth information acquired by the information acquisition unit 11, the 3D extraction unit 13 extracts feature classification information of the person to be determined. Specifically, the 3D extraction unit 13 superimposes the 2D position indicated by the 2D feature information and the coordinates of the depth information, thereby obtaining the 3D shape information of the target part of the face of the person to be determined indicated by the 2D feature information. Extract. The 3D extraction unit 13 extracts feature classification information of the target part based on the extracted 3D shape information. For example, the 3D extraction unit 13 detects the boundary between the eyelid and the eyeball based on the extracted 3D information of the eye, recognizes the shape of the eye based on the detected boundary, and features relating to the shape of the eye Extract classification information. The 3D extraction unit 13 recognizes the direction and shape of the nose tip (nose tip), the shape and height of the back of the nose, the shape of the nose wing, and the like based on the extracted 3D information of the nose. Feature classification information is extracted. The 3D extraction unit 13 can recognize the lip thickness and the like regarding the mouth of the person to be determined.

  The 3D extraction unit 13 specifies at least one shape feature classification among the plurality of shape feature classifications regarding the target part of the face of the person to be determined as the feature classification information. For example, the 3D extraction unit 13 relates to the eye shape of the person to be judged, and is at least one of a plurality of shape feature classifications such as a suspended eye, a leaned eye, a single, a double, and a cut length (for example, a leaned eye). Is identified as feature classification information. In addition, the 3D extraction unit 13 relates to the nose shape of the person to be judged, and is at least one of a plurality of shape feature classifications such as turn-up type, wrinkle type, lion type, Roman type, Greek type, and Nubian type. One (for example, a shin type) is specified as the feature classification information.

The 3D extraction unit 13 determines a point indicating a degree corresponding to the shape feature classification for each of the plurality of shape feature classifications related to the target part of the face of the person to be determined, and from such shape feature classification and points Such information may be determined as the feature classification information. For example, the 3D extraction unit 13 can determine the following feature classification information regarding the shape of the person's nose. In the following example, the maximum point of each shape feature classification is 10 points.
Turn-up type: 4 points Horsetail type: 1 point Sushi type: 6 points Roman type: 8 points Greek type: 0 points Nubian type: 1 point

  The feature information storage unit 16 stores valid 2D feature information and valid feature classification information that can prove the legitimacy of the person to be determined. Hereinafter, a person in which such a pair of valid feature information (2D feature information and feature classification information) is stored in the feature information storage unit 16 is referred to as a registrant. That is, the feature information storage unit 16 can also store such a pair of valid feature information for each registrant to be authenticated. Such a pair of valid feature information is information extracted by the 2D extraction unit 12 and the 3D extraction unit 13 from the registrant's own 3D information, and is stored in the feature information storage unit 16 at an arbitrary timing.

  The condition acquisition unit 14 extracts a pair of valid feature information targeted by the determination unit 15 from the feature information storage unit 16. Specifically, the condition acquisition unit 14 extracts one set of 2D feature information and feature classification information when one-to-one authentication is performed, and when one-to-N authentication is performed, the condition acquisition unit 14 2D feature information and feature classification information are extracted. The condition acquisition unit 14 can also acquire the ID of the person to be determined and extract a set of 2D feature information and feature classification information associated with the ID from the feature information storage unit 16.

  The determination unit 15 compares the 2D feature information of the person to be determined with the legitimate 2D feature information acquired by the condition acquisition unit 14, and is acquired by the feature classification information of the person to be determined and the condition acquisition unit 14. By comparing with the valid feature classification information, it is determined whether or not the person to be judged is a registrant, that is, the legitimacy of the person to be judged. Specifically, the determination unit 15 matches or approximates the 2D feature information of the person to be determined and the 2D feature information of the registrant, and matches the feature classification information of the person to be determined and the feature classification information of the registrant. Or, in the case of approximation, the person to be judged is judged to be valid.

  For example, when the distance between 2D feature information is less than or equal to a predetermined value, the determination unit 15 can determine that both 2D feature information matches or approximates. Further, in the form in which the feature classification information indicates at least one shape feature classification, the determination unit 15 registers the feature classification information of the person to be determined when at least one shape feature classification of both feature classification information is the same. It can be determined that the feature classification information of the person matches or approximates. In the form in which the feature classification information is indicated by the combination of the shape feature classification and the point, the determination unit 15 determines that both the features are the same when the total or average or maximum value of the differences in the same shape feature classification is equal to or less than a predetermined value. It can be determined that the classification information matches or approximates.

  In addition, the determination unit 15 points the comparison result between the 2D feature information and the comparison result between the feature classification information, calculates a total point therefrom, and determines the legitimacy of the person to be determined using the total point. You may make it do. Specifically, the determination unit 15 determines a 2D fitness value indicating the degree of matching between 2D feature information, determines a 3D fitness value indicating the degree of matching between feature classification information, and determines the 2D fitness value and the 3D fitness value. The total fitness value is calculated by multiplication, and the legitimacy of the person to be judged is determined using the total fitness value. For example, in a form in which a larger 2D fitness value and 3D fitness value are set as the degree of approximation between 2D feature information and feature classification information is higher, the determination unit 15 determines that the overall fitness value is greater than a predetermined threshold. It is determined that the person to be determined is valid.

  In addition, the determination unit 15 determines a 2D conformance value and a 3D conformance value for each of a plurality of target parts of the object to be determined, and multiplies the 2D conformance value and the 3D conformance value of the same target part to thereby obtain a partial conformance value May be further determined, and the total fitness value may be calculated by adding the partial fitness values of each target region. In this case, the determination unit 15 may calculate the total fitness value by adding a value obtained by multiplying the partial fitness value of each target site by a weighting coefficient corresponding to each target site. For example, the weighting factor is set to a large value for a target part having a shape that easily changes due to a shape or expression that is difficult to be erroneously detected. Set to a small value.

[Operation example]
Hereinafter, the face authentication method in the first embodiment will be described. FIG. 4 is a flowchart illustrating an operation example of the face authentication apparatus 10 according to the first embodiment. In the following description, the face authentication device 10 is an execution subject of each process, but each of the processing units included in the face authentication device 10 may be an execution subject.

  The face authentication device 10 acquires 3D information of the face of the person to be determined (S41). This 3D information may be acquired from the information detected by the 3D sensor 7 as described above, or may be acquired from another device.

  The face authentication device 10 extracts 2D feature information of the person to be determined from the 2D image included in the acquired 3D information (S42). For example, the face authentication device 10 extracts 2D feature information related to each of a plurality of target portions of the person to be determined from the 2D image. As described above, the 2D feature information of each target part is determined for each target part. 2D positions such as the left mouth corner, right mouth corner, upper lip point, and lower lip point are extracted as 2D feature information of the mouth, and 2D positions such as both edges of the nose wing and nose tip are extracted as 2D feature information of the nose.

  The face authentication device 10 extracts the feature classification information of the person to be determined based on the 2D feature information extracted in (S42) and the depth information included in the 3D information (S43). Specifically, the face authentication device 10 superimposes the 2D position indicated by the 2D feature information and the coordinates of the depth information, thereby obtaining the 3D shape information of the target part of the face of the person to be determined indicated by the 2D feature information. Based on the extracted 3D shape information, the feature classification information of the target part is specified.

  For example, the face authentication apparatus 10 may extract at least one shape feature classification from among a plurality of shape feature classifications related to the target portion of the person to be determined as the feature classification information. Further, the face authentication device 10 determines a point indicating a degree corresponding to each shape feature classification for each of the plurality of shape feature classifications regarding the target part of the person to be determined, and sets a plurality of pairs of the shape feature classification and the points. You may make it extract as feature classification information. For example, the face authentication apparatus 10 may specify at least one of a plurality of shape feature classifications such as thin lips, thick lips, candy mouths, duck mouths, and receptacles as mouth feature classification information.

  The face authentication device 10 acquires valid 2D feature information and valid feature classification information (a pair of valid feature information) that can prove the validity of the person to be determined from the feature information storage unit 16 (S44). The face authentication device 10 may acquire all valid pairs of feature information stored in the feature information storage unit 16, or one valid feature information if it can be specified by the ID of the person to be determined. You may get a pair.

  The face authentication device 10 compares the 2D feature information of the person to be determined extracted in (S42) with the valid 2D feature information acquired in (S44), and the person to be determined extracted in (S43) Is compared with the legitimate feature classification information acquired in (S44) (S45). Here, the face authentication apparatus 10 may determine a 2D fitness value indicating a comparison result between 2D feature information, and may determine a 3D fitness value indicating a comparison result between feature classification information.

  The face authentication device 10 determines the legitimacy of the person to be determined based on the comparison result of (S45) (S46). In other words, the face authentication apparatus 10 determines whether or not the person to be determined is a registrant who has already stored a valid feature information pair in the feature information storage unit 16. Specifically, the face authentication device 10 determines that the person to be determined is valid when the 2D feature information matches or approximates and the feature classification information matches or approximates. The degree of approximation between feature information is determined as the 2D conformance value and the 3D conformance value as described above, and the person to be judged is determined by whether or not the total conformance value calculated from the 2D conformance value and the 3D conformance value exceeds a predetermined threshold value. May be determined.

[Operation and Effect in First Embodiment]
As described above, in the first embodiment, the 3D information of the person to be determined detected by the three-dimensional sensor 7 is acquired, and 2D feature information regarding the target part of the person to be determined is extracted from the 2D image of the 3D information. Feature classification information related to the target part of the person to be determined is extracted from the depth information and the 2D feature information of the 3D information. Then, the legitimacy of the person to be judged is determined by comparing the person's 2D feature information and feature classification information with the legitimate 2D feature information and legitimate feature classification information stored in the feature information storage unit 16. The Therefore, according to the first embodiment, the legitimacy of the person to be judged is determined in consideration of not only the two-dimensional information about the target part of the person to be judged but also three-dimensional information about the target part. The determination accuracy can be improved.

  Furthermore, according to the first embodiment, the feature classification information of the three-dimensional shape is indicated by at least one of the plurality of shape feature classifications or a plurality of pairs of the shape feature classification and its points. Comparison processing between classification information can be simplified and speeded up.

  In the first embodiment, a 2D matching value indicating a degree of matching between the 2D feature information of the person to be judged and the legitimate 2D feature information is determined, and the feature classification information of the person to be judged and the legitimate feature classification information are 3D matching values indicating the degree of matching between the two are determined, and the legitimacy of the person to be determined is determined based on the total matching value calculated from them. As described above, since the collation result of the two-dimensional information and the collation result of the three-dimensional information are each pointed, according to the first embodiment, the determination process of the validity of the person to be determined is simplified and speeded up. can do.

  In the first embodiment, a partial fitness value may be calculated for each target region of the person to be determined from the 2D fitness value and the 3D fitness value, and the total fitness value may be calculated by summing the partial fitness values. According to the first embodiment, since the partial fitness value is calculated for each target part in this way, the weight for each target part is changed in accordance with the degree of influence on the validity determination of the person to be determined. As a result, the determination accuracy can be improved.

[Second Embodiment]
Hereinafter, the monitoring video analysis device according to the second embodiment will be described focusing on the content different from the face authentication device 10 according to the first embodiment. In the following description, the same contents as those in the first embodiment are omitted as appropriate. The monitoring video analysis apparatus in the second embodiment detects a target (target person in this embodiment) that matches the determination condition from the monitoring video.

〔Device configuration〕
FIG. 5 is a diagram conceptually illustrating a hardware configuration example of a monitoring video analysis device (hereinafter simply referred to as an analysis device) 20 in the second embodiment. The analysis device 20 in the second embodiment is a so-called computer and has the same configuration as the face authentication device 10 in the first embodiment. In the second embodiment, the input / output I / F 4 is connected to the input device 8, the display device 9, and the like. The input device 8 is a device that receives input of a user operation such as a keyboard and a mouse, and the display device 9 is a device that performs display output in accordance with an image signal sent from an analysis device 20 such as a display. The input / output I / F 4 may be connected to a surveillance video device (not shown) or may be connected to a communication device (not shown) that communicates with other devices. The hardware configuration of the analyzer 20 is not limited.

[Processing configuration]
FIG. 6 is a diagram conceptually illustrating a processing configuration example of the analysis apparatus 20 in the second embodiment. The analysis apparatus 20 in the second embodiment includes an information acquisition unit 21, a two-dimensional feature extraction unit (hereinafter referred to as a 2D extraction unit) 22, a three-dimensional feature extraction unit (hereinafter referred to as a 3D extraction unit) 23, a condition It has the acquisition part 24, the determination part 25 grade | etc.,. Each of these processing units is realized, for example, by executing a program stored in the memory 3 by the CPU 2 as in the first embodiment.

  The information acquisition unit 21 acquires monitoring video data including 3D information from a monitoring video device, a portable recording medium, or another device. The monitoring video data may be moving image data or still image data. When the moving image data is acquired as the monitoring video, the information acquisition unit 21 gives the still image frame constituting the moving image data to the other processing unit as a 2D image, and the depth information included in the monitoring video data to the other processing unit. give.

  The 2D extraction unit 22 and the 3D extraction unit 23 are the same as the 2D extraction unit 12 and the 3D extraction unit 13 of the first embodiment. The to-be-determined person in the second embodiment is an unspecified person that appears in the 2D image of the monitoring video. When a plurality of unspecified persons are reflected in the 2D image, the 2D extraction unit 22 and the 3D extraction unit 23 extract 2D feature information and feature classification information for each unspecified person, respectively.

  The condition acquisition unit 24 acquires determination condition information of a target person to be detected. For example, the condition acquisition unit 24 causes the display device 9 to display an input screen for allowing the operator to input the determination condition of the target person, and acquires determination condition information input using the input device 8 via the input screen. To do. The determination condition information may be acquired from a portable recording medium or another device via the input / output I / F 4. The determination condition information is 2D feature information and feature classification information regarding the target person. Such determination condition information may be 2D feature information and feature classification information extracted by a 2D extraction unit 22 and a 3D extraction unit 23 from a 2D image of a target person such as a portrait or a photograph.

  The determination unit 25 determines whether or not the determination target person is the target person by the same processing as the determination unit 15 of the first embodiment described above. However, in the second embodiment, the determination unit 25 compares the 2D feature information and feature classification information of the person to be determined with the 2D feature information and feature classification information indicated by the determination condition information in this determination. When a plurality of persons to be determined (unspecified persons) are shown in the 2D image, the determination unit 25 determines whether each person to be determined is a target person. In the second embodiment, the purpose is to detect the target person from the monitoring video, so the degree of compatibility (total fitness value) between the 2D feature information and the feature classification information is lower than that of the first embodiment. May be.

[Operation example]
Hereinafter, a monitoring video analysis method according to the second embodiment will be described. FIG. 7 is a flowchart illustrating an operation example of the analyzer 20 according to the second embodiment. In the following description, the analysis device 20 is an execution subject of each process, but each of the above-described processing units included in the analysis device 20 may be an execution subject.

  The analysis apparatus 20 acquires monitoring video data including 3D information (S71). As described above, the monitoring video data may be acquired from the monitoring video device, or may be acquired from a portable recording medium, another device, or the like. Hereinafter, processing is executed on a 2D image in which at least one unspecified person is captured among 2D images obtained from the monitoring video data. An unspecified person appearing in a 2D image is described as a person to be judged.

  The analysis device 20 extracts the 2D feature information of the person to be determined from the 2D image of the monitoring video data (S72). Subsequently, the analysis apparatus 20 extracts feature classification information of the person to be determined based on the 2D feature information and the depth information of the monitoring video data (S73). Specific processing in (S72) and (S73) is the same as (S42) and (S43) shown in FIG.

  The analyzer 20 acquires determination condition information regarding the target person to be detected (S74). The determination condition information is 2D feature information and feature classification information regarding the target person, and is input from an input device, for example. Further, 2D feature information and feature classification information extracted by the same process as (S72) and (S73) from a portrait or photograph of the target person may be used as the determination condition information.

  The analysis apparatus 20 compares the 2D feature information of the person to be determined extracted in (S72) with the 2D feature information of the target person acquired in (S74), and the person to be determined extracted in (S73). Is compared with the feature classification information of the target obtained in (S74) (S75). Here, the analysis apparatus 20 may determine a 2D fitness value indicating a comparison result between 2D feature information, and may determine a 3D fitness value indicating a comparison result between feature classification information.

  Based on the comparison result of (S75), the analyzer 20 determines whether the person to be determined is the target person (S76). In other words, the analysis apparatus 20 determines whether or not the unspecified person shown in the 2D image of the monitoring video data is the target person indicated by the determination condition information. Specifically, the analysis apparatus 20 determines that the determination target person is the target person when the 2D feature information matches or approximates and the feature classification information matches or approximates. The degree of approximation between feature information is determined as the 2D conformance value and the 3D conformance value as described above, and the person to be judged is determined by whether or not the total conformance value calculated from the 2D conformance value and the 3D conformance value exceeds a predetermined threshold value. It may be determined whether or not is a target person.

[Operations and effects in the second embodiment]
As described above, in the second embodiment, 2D feature information related to the target part of the person to be determined is extracted from the 2D image included in the monitoring video data, and from the depth information and 2D feature information included in the monitoring video data, Feature classification information regarding the target part of the judge is extracted. Then, it is determined whether or not the determination target person is the target person by comparing the determination target person's 2D feature information and feature classification information with the target person's 2D feature information and feature classification information to be detected. Therefore, according to the second embodiment, the target person is detected in consideration of not only the two-dimensional information about the target part of the person to be determined but also the three-dimensional information about the target part. Detection accuracy for detecting a person can be improved.

  Furthermore, in the second embodiment, the feature classification information of the three-dimensional shape is indicated by at least one of the plurality of shape feature classifications or a plurality of pairs of the shape feature classification and its points. The determination condition information for specifying can be easily specified by the operator, and the comparison processing between the feature classification information can be simplified and speeded up.

  In the second embodiment, a 2D fitness value indicating the degree of matching between the 2D feature information of the person to be judged and the 2D feature information of the target person is determined, and the feature classification information of the person to be judged and the feature classification of the target person A 3D matching value indicating the degree of matching with the information is determined, and it is determined whether or not the determination target person is the target person based on the total matching value calculated therefrom. As described above, since the comparison result of the two-dimensional information and the comparison result of the three-dimensional information are each pointed, according to the second embodiment, the determination process of whether or not the determination target person is the target person is performed. It can be simplified and speeded up. As a result, according to the second embodiment, it is possible to quickly detect the target person from the monitoring video data in which a plurality of unspecified persons appear irregularly.

[Modification]
In each of the embodiments described above, the part in the face of the person to be determined is exemplified as the target part, but the target part from which the feature classification information is extracted is not limited to the above example. For example, feature classification information indicating features of the shape of the head of the person to be determined may be extracted, or feature classification information indicating characteristics of the person to be determined, such as height and body shape, may be extracted. By using the three-dimensional information, it is possible to extract shape feature classifications such as a slim shape, a stubborn body shape, and a solid body shape. Further, the height of the person in a 2D image in which an arbitrary person located at a predetermined depth from the three-dimensional sensor 7 and the height of the person in the real world are measured in advance, and the predetermined depth, the height in the image, and the real world are measured. From the height relationship, the height of the person to be judged in the acquired 2D image, and the depth of the person to be judged indicated by the acquired depth information, the height of the person to be judged in the real world can be calculated. . By using various types of feature classification information in this way, the determination accuracy can be improved.

  In the second embodiment described above, the determination condition information includes 2D feature information and feature classification information. However, the determination condition information may indicate only feature classification information. In this case, the determination unit 15 may determine whether or not the determination target person is the target person by performing comparison only between the feature classification information without performing comparison between the 2D feature information. In this case, although the determination accuracy is lowered, it is possible to easily detect the target person only by specifying the feature classification information indicating the feature of the three-dimensional shape.

  In addition, in the some flowchart used by the above-mentioned description, although several process (process) is described in order, the execution order of the process performed by each embodiment is not restrict | limited to the order of the description. In each embodiment, the order of the illustrated steps can be changed within a range that does not hinder the contents.

2 CPU
3 Memory 4 Input / output I / F
7 3D sensor 8 Input device 9 Display device 10 Face authentication device 11, 21, 101 Information acquisition unit 12, 22, 102 Two-dimensional feature extraction unit (2D extraction unit)
13, 23, 103 3D feature extraction unit (3D extraction unit)
14, 24, 104 Condition acquisition unit 15, 25, 105 Determination unit 16 Feature information storage unit 20 Surveillance video analysis device (analysis device)
100 Information processing apparatus

Claims (9)

An information acquisition unit for acquiring depth information and a two-dimensional image forming three-dimensional information of at least a part of the determination object;
A two-dimensional feature extraction unit that extracts two-dimensional feature information of the object to be determined from a two-dimensional image acquired by the information acquisition unit;
Based on the two-dimensional feature information and the depth information acquired by the information acquisition unit, a three-dimensional feature extraction unit that extracts feature classification information indicating a feature classification of the three-dimensional shape of the determination object;
A condition acquisition unit for acquiring two-dimensional feature information and feature classification information as determination conditions;
By comparing the two-dimensional feature information extracted by the two-dimensional feature extraction unit and the feature classification information extracted by the three-dimensional feature extraction unit with the two-dimensional feature information and the feature classification information as the determination condition, A determination unit for determining suitability of the determination condition in the determination object;
An information processing apparatus comprising: The two-dimensional feature extraction unit extracts two-dimensional feature information related to a target part of the determination object,
The three-dimensional feature extraction unit specifies at least one shape feature classification among a plurality of shape feature classifications related to the target part as the feature classification information.
The information processing apparatus according to claim 1. The two-dimensional feature extraction unit extracts two-dimensional feature information related to a target part of the determination object,
The three-dimensional feature extraction unit determines, as the feature classification information, a point indicating a degree corresponding to the shape feature classification for each of a plurality of shape feature classifications related to the target part;
The information processing apparatus according to claim 1. The determination unit determines a two-dimensional fitness value by comparing the two-dimensional feature information of the determination object and the two-dimensional feature information as the determination condition, and determines the feature classification information of the determination object and the A three-dimensional fitness value is determined by comparison with the feature classification information as a determination condition, and a total fitness value is calculated using the two-dimensional fitness value and the three-dimensional fitness value. , Determining suitability of the determination condition in the object to be determined;
The information processing apparatus according to any one of claims 1 to 3. The determination unit determines the two-dimensional fitness value and the three-dimensional fitness value for each of the plurality of target regions of the determination target, and determines the two-dimensional fitness value and the three-dimensional fitness value of the same target region. Further determining a partial fitness value, and calculating the total fitness value from the partial fitness value of each target region,
The information processing apparatus according to claim 4. The to-be-determined object is a to-be-determined person reflected in a two-dimensional image acquired by the information acquisition unit,
The condition acquisition unit acquires the two-dimensional feature information and feature classification information as the determination condition from a feature storage unit that stores legitimate two-dimensional feature information and feature classification information of an individual,
The determination unit determines the legitimacy of the determination target person as the suitability determination,
The information processing apparatus according to any one of claims 1 to 5. The to-be-determined object is a to-be-determined person reflected in a two-dimensional image acquired by the information acquisition unit,
The condition acquisition unit acquires two-dimensional feature information and feature classification information about the target person detected from the three-dimensional information acquired by the information acquisition unit,
The determination unit determines whether the determination target person is the target person as the determination of the suitability;
The information processing apparatus according to any one of claims 1 to 5. In a determination method executed by at least one computer,
Obtaining depth information and 2D image forming 3D information of at least a part of the determination object;
Extracting the two-dimensional feature information of the object to be determined from the acquired two-dimensional image;
Based on the two-dimensional feature information and the depth information, the feature classification information indicating the feature classification of the three-dimensional shape of the determination object is extracted,
Obtain two-dimensional feature information and feature classification information as judgment conditions,
Comparing the extracted two-dimensional feature information and the extracted feature classification information with the two-dimensional feature information and the feature classification information as the determination condition;
The suitability of the determination condition in the determination object is determined by the comparison.
A determination method including the above. In a program for causing at least one computer to execute a determination method,
The determination method is:
Obtaining depth information and 2D image forming 3D information of at least a part of the determination object;
Extracting the two-dimensional feature information of the object to be determined from the acquired two-dimensional image;
Based on the two-dimensional feature information and the depth information, the feature classification information indicating the feature classification of the three-dimensional shape of the determination object is extracted,
Obtain two-dimensional feature information and feature classification information as judgment conditions,
Comparing the extracted two-dimensional feature information and the extracted feature classification information with the two-dimensional feature information and the feature classification information as the determination condition;
The suitability of the determination condition in the determination object is determined by the comparison.
A program that includes that.

Images