JP7240193B2 - Face orientation determination device, computer program, and storage medium - Google Patents

Description

The present invention relates to a device for determining the orientation of an occupant's face in a vehicle interior. The invention also relates to a computer program executed by a processor provided in the device and a storage medium on which the computer program is stored.

2. Description of the Related Art As disclosed in Patent Literature 1, there is known a device for determining the orientation of an occupant's face in a vehicle interior. The device determines the orientation of the occupant's face based on the distance between the eyes and the nose included in the image of the face facing the front of the camera.

JP 2012-022579 A

SUMMARY OF THE INVENTION An object of the present invention is to make it possible to determine the direction of the face of an occupant whose face is not facing the camera.

A first aspect for achieving the above object is a face direction determination device,
an input interface that accepts a position data set that includes a plurality of point elements each associated with a position in three-dimensional space;
a processor for determining the face orientation of an occupant in the vehicle interior based on the position data set;
and
The processor
extracting from the position data set a position data subset corresponding to a region in three-dimensional space where the occupant's head is likely to exist;
A facial orientation of the occupant is determined based on the number of point elements included in the position data subset.

A second aspect for achieving the above object is a computer that causes a processor to determine the orientation of the face of an occupant in the vehicle compartment based on a position data set that includes a plurality of point elements each associated with a three-dimensional position. a program,
When the computer program is executed, the processor:
extracting from the position data set a position data subset corresponding to a spatial region where the occupant's head is likely to exist;
A face orientation of the occupant is determined based on the number of the point elements included in the position data subset.

According to the configuration as described above, the direction of the face of the occupant can be determined based on the amount of information regarding the position of the occupant's head in the three-dimensional space. Since it is not necessary to rely only on image information, the orientation of the face of a passenger whose face is not facing the camera can also be determined.

A third aspect for achieving the above object is a face direction determination device,
an input interface that accepts a position data set that includes a plurality of point elements each associated with a position in three-dimensional space;
a processor for determining the face orientation of an occupant in the vehicle interior based on the position data set;
and
The processor
extracting from the position data set a position data subset corresponding to a region in three-dimensional space where the occupant's head is likely to exist;
estimating a center position of the head based on the position data subset;
identifying the radius of a sphere approximating the head centered at the center location;
A face orientation of the occupant is determined based on the radius.

A fourth aspect for achieving the above object is a computer that causes a processor to determine the orientation of the face of an occupant in a vehicle compartment based on a position data set that includes a plurality of point elements each associated with a three-dimensional position. a program,
When the computer program is executed, the processor:
extracting from the position data set a position data subset corresponding to a spatial region where the occupant's head is likely to exist;
estimating a center position of the head based on the position data subset;
identify the radius of a sphere approximating the head centered at the center position;
The orientation of the occupant's face is determined based on the radius.

According to the above configuration, the estimated position of the center of the head of the occupant and the radius of the sphere that approximates the head centered on the position are obtained from the information related to the position of the occupant's head in the three-dimensional space, The orientation of the occupant's face can be determined based on the value of the radius. Since it is not necessary to rely only on image information, the orientation of the face of a passenger whose face is not facing the camera can also be determined.

The face direction determination device according to the third aspect can be configured as follows.
The processor
calculating a representative value of the distance from the center position to each of the plurality of point elements included in the position data subset;
Based on the comparison between the representative value and the radius, the orientation of the occupant's face is determined.

A computer program according to the fourth aspect can be configured as follows.
to the processor;
calculating a representative value of distances from the center position to each of the plurality of point elements included in the position data subset;
The orientation of the occupant's face is determined based on the comparison between the representative value and the radius.

According to the configuration as described above, it is possible to more precisely determine the orientation of the occupant's face using the radius of the sphere that approximates the head.

The face direction determination device according to the third aspect can be configured as follows.
The processor determines the orientation of the occupant's face based on the positional relationship between the center position and the barycentric positions of the plurality of positions in the three-dimensional space associated with the plurality of point elements included in the position data subset. determine.

A computer program according to the fourth aspect can be configured as follows.
The processor determines the direction of the occupant's face based on the positional relationship between the centroid position of the plurality of three-dimensional positions associated with the plurality of point elements included in the position data subset and the center position. Let

According to the configuration as described above, even when the occupant's face is not facing the camera, it is possible to determine the specific orientation of the occupant's face.

The face direction determination devices according to the first aspect and the third aspect can be configured as follows.
The input interface receives the position data set from a TOF (Time of Flight) camera,
Each of the plurality of point elements corresponds to one of the plurality of pixel data output from the TOF camera.

A computer program according to the second aspect and the fourth aspect can be configured as follows.
The position data set is obtained from a TOF (Time of Flight) camera.

According to the configuration as described above, it is possible to efficiently acquire the position information related to the occupant's head together with the image information.

A fifth aspect for achieving the above object is a face direction determination device,
an input interface for receiving image data including a plurality of pixel data corresponding to an image including a captured vehicle interior;
Using a discriminator that has learned, as training data, an image including at least the captured head in order to discriminate the direction of the face of the occupant captured based on the feature amount related to the luminance of the image, based on the image data, a processor that determines the orientation of the captured face of the occupant included in the image;
and
The processor
identifying a first detection region in the image data that may include the imaged head of the occupant;
inputting a feature quantity based on the brightness of each pixel included in the first detection region into the discriminator;
Acquiring a first discrimination result for the orientation of the captured face of the occupant based on the discrimination result of the discriminator for the first detection area,
setting at least one second detection area having the same size as the first detection area at a position partially overlapping the first detection area;
inputting a feature amount based on the brightness of each pixel included in the second detection area into the discriminator;
Acquiring at least one second determination result for the orientation of the captured face of the occupant based on the identification result of the classifier for the second detection area;
If the at least one second determination result matches the first determination result, the first determination result is validated.

In a sixth aspect for achieving the above object, an image including at least the captured head is used as training data to determine the direction of the face of the passenger captured based on the feature value related to the brightness of the image. A computer program that uses a discriminator and causes a processor to determine the direction of the face of an occupant captured in an image based on image data that corresponds to an image that includes a captured vehicle interior and includes a plurality of pixel data. hand,
When the computer program is executed, the processor:
specifying a first detection area in the image data that may include the imaged head of the occupant;
causing the discriminator to input a feature value based on the brightness of each pixel included in the first detection region;
Acquiring a first discrimination result for the orientation of the captured face of the occupant based on the discrimination result of the discriminator for the first detection area;
setting at least one second detection area having the same size as the first detection area at a position partially overlapping the first detection area;
causing the discriminator to input a feature value based on the brightness of each pixel included in the second detection region;
Acquiring at least one second determination result for the orientation of the captured face of the occupant based on the result of identification by the classifier for the second detection area;
If the at least one second determination result matches the first determination result, the first determination result is validated.

According to the configuration as described above, the face orientation of the captured occupant is discriminated using a discriminator that has learned the captured image including the head as teacher data, and the discrimination result is also verified. Therefore, it is possible to improve the accuracy of determining the direction of the face of a passenger whose face does not face the camera.

A face direction determination device according to the fifth aspect can be configured as follows.
The input interface accepts a position data set including a plurality of point elements each associated with a position in three-dimensional space;
The processor
extracting a position data subset corresponding to the first detection region from the position data set;
Based on the positional relationship between the center position of the head estimated based on the position data subset and the center-of-gravity positions of the plurality of positions associated with the plurality of point elements included in the position data subset, The orientation of the imaged occupant's face is discriminated.

A computer program according to the sixth aspect can be configured as follows.
to the processor;
extracting a position data subset corresponding to the first detection region from a position data set including a plurality of point elements each associated with a three-dimensional position;
Based on the positional relationship between the center position of the head estimated based on the position data subset and the barycenter positions of the plurality of three-dimensional positions associated with the plurality of point elements included in the position data subset to discriminate the orientation of the captured face of the occupant.

According to the configuration as described above, even when the occupant's face is not facing the camera, it is possible to determine the specific orientation of the occupant's face.

A face direction determination device according to the fifth aspect can be configured as follows.
The feature amount is HOG (Histograms of Oriented Gradients).

A computer program according to the sixth aspect can be configured as follows.
The feature amount is HOG (Histograms of Oriented Gradients).

According to the configuration as described above, the edge of the head can be detected with relatively high accuracy. Therefore, it is possible to improve the accuracy of determining the orientation of the captured face of the occupant.

A face direction determination device according to the fifth aspect can be configured as follows.
The classifier is a support vector machine.

A computer program according to the sixth aspect can be configured as follows.
The classifier is a support vector machine.

According to the configuration as described above, a discriminator having high generalization performance can be obtained, so that it is possible to improve the accuracy of discriminating the orientation of the captured face of the occupant.

A face direction determination device according to the fifth aspect can be configured as follows.
The input interface receives the position data set from a TOF (Time of Flight) camera as part of the image data.

A computer program according to the sixth aspect can be configured as follows.
The position data set is obtained from a TOF (Time of Flight) camera as part of the image data.

According to the configuration as described above, it is possible to efficiently acquire the position information related to the occupant's head together with the image information.

One aspect for achieving the above object is a storage medium storing a computer program according to any one of the second, fourth, and sixth aspects.

According to the present invention, it is possible to determine the orientation of the face of an occupant whose face does not face the camera.

1 illustrates the configuration of a face direction determination system according to an embodiment; 2 illustrates part of a vehicle in which the face direction determination system of FIG. 1 is mounted. 2 illustrates the flow of processing executed by the face direction determination apparatus of FIG. 1; 2 is a diagram for explaining processing executed by the face direction determination device of FIG. 1; FIG. 2 is a diagram for explaining processing executed by the face direction determination device of FIG. 1; FIG. 2 is a diagram for explaining processing executed by the face direction determination device of FIG. 1; FIG. 2 is a diagram for explaining processing executed by the face direction determination device of FIG. 1; FIG. FIG. 4 shows a first specific example of face direction determination processing in FIG. 3 ; FIG. FIG. 4 shows a second specific example of the face direction determination process in FIG. 3. FIG. FIG. 10 is a diagram for explaining the processing of FIG. 9; FIG. FIG. 10 is a diagram for explaining the processing of FIG. 9; FIG. FIG. 4 shows a third specific example of the face direction determination process in FIG. 3. FIG. 13 is a diagram for explaining the processing of FIG. 12; FIG. 13 is a diagram for explaining the processing of FIG. 12; FIG.

Exemplary embodiments are described in detail below with reference to the accompanying drawings. In each drawing used for the following description, the scale is appropriately changed so that each member has a recognizable size.

FIG. 1 schematically shows the configuration of a face direction determination system 1 according to one embodiment. A face direction determination system 1 includes a TOF (Time of Flight) camera 2 and a face direction determination device 3 . FIG. 2 shows part of a vehicle 4 on which the face direction determination system 1 is mounted. An arrow L indicates a direction along the longitudinal direction of the vehicle 4 . An arrow H indicates a direction along the height direction of the vehicle 4 .

The TOF camera 2 is arranged at an appropriate position in the passenger compartment 41 of the vehicle 4 shown in FIG. 2 and acquires an image including the captured passenger compartment 41 .

The TOF camera 2 has a light emitting element and a light receiving element. The light emitting element emits, for example, infrared light as detection light. The emitted detection light is reflected by the object and enters the light receiving element as return light. By measuring the time from when the detection light is emitted from the light emitting element to when the return light is incident on the light receiving element, the distance to the object from which the return light is generated is calculated. By calculating the distance for each of the plurality of pixels that make up the image acquired by the TOF camera 2, each pixel has two-dimensional position coordinates (U, V) in the image, as well as It includes distance information d(U, V) that indicates the distance (depth) to a part of the corresponding object.

The TOF camera 2 outputs an image data ID corresponding to the acquired image. The image data ID contains multiple pixel data sets. Each of the plurality of pixel data sets is associated with a corresponding one of the plurality of pixels making up the acquired image. Each of the plurality of pixel data sets includes position coordinates (U, V) and distance information d(U, V). That is, each of the multiple pixel data sets is associated with a position in the three-dimensional space. Each of the multiple pixel data sets is an example of a point element. The image data ID is also an example of a position data set.

The face direction determination device 3 is mounted at an appropriate position on the vehicle 4 . The face direction determination device 3 is a device for determining the face direction of the driver 5 in the vehicle interior 41 captured based on the image data ID provided from the TOF camera 2 . Driver 5 is an example of a passenger.

The face direction determination device 3 has an input interface 31 . The input interface 31 receives image data ID output from the TOF camera 2 .

The face direction determination device 3 has a processor 32 . The processor 32 executes processing for determining the direction of the face of the driver 5 based on the image data ID input to the input interface 31 .

The flow of processing performed by processor 32 will now be described with reference to FIG. The processor 32 first performs preliminary processing on the image data ID (STEP 1).

The position coordinates (U, V) and the distance information d (U, V) included in each pixel data set are calculated using the following equation in the camera coordinate system when the image center coordinates are defined as (c _X , c _Y ). can be converted to a point (X, Y, Z) on the three-dimensional space in . f represents the focal length of the lens that the TOF camera 2 has.

The processor 32 converts the position coordinates (U, V) and the distance information d (U, V) to a point (X, Y, Z) on the three-dimensional space in the camera coordinate system based on the above formula. The conversion from the position coordinates (U, V) and the distance information d (U, V) to the point (X, Y, Z) on the three-dimensional space in the camera coordinate system is performed by a processor built into the TOF camera 2. may be done. In this case, each of the plurality of pixel data sets included in the image data ID output from the TOF camera 2 includes position coordinates (X, Y, Z). Also in this case, each of the plurality of pixel data sets is associated with a position in the three-dimensional space. Also in this case, each of the plurality of pixel data sets is an example of a point element. In this case, the image data ID is also an example of the position data set.

The position coordinates (X, Y, Z) in the camera coordinate system can be converted into position coordinates (W, L, H) in the vehicle coordinate system with a specific position in the vehicle 4 as the origin. A W axis is a coordinate axis extending in the left-right direction of the vehicle 4 . For example, when a specific position in the driver's seat is set as the origin, the W-axis coordinate value takes a positive value on the right side of the origin and a negative value on the left side of the origin as seen from the driver 5. . The L axis is a coordinate axis extending in the front-rear direction of the vehicle 4 . For example, the L-axis coordinate value takes a positive value in front of the origin and a negative value behind the origin. The H-axis is a coordinate axis extending in the vertical direction of the vehicle 4 . For example, the H-axis coordinate value takes a positive value above the origin and a negative value below the origin.

Processor 32 transforms each pixel data set from position coordinates (X, Y, Z) in the camera coordinate system to position coordinates (W, L, H) in the vehicle coordinate system. The origin is chosen as a position corresponding to the hipbone of the driver 5, for example. Coordinate transformation to the vehicle coordinate system can be performed using well-known coordinate rotation transformation, translation transformation, scale transformation, and the like.

Processor 32 then performs processing for limiting the spatial region to be processed. Specifically, from the image data ID acquired from the TOF camera 2, a plurality of pixel data sets PD corresponding to the limited spatial region are extracted.

(A) of FIG. 4 shows an example of the image data ID acquired from the TOF camera 2 . As shown in FIG. 2 , there is a high probability that the head 51 of the driver 5 seated in the driver's seat is positioned above the passenger compartment 41 . Therefore, the spatial region to be processed is limited to the upper portion of the vehicle interior 41 . For example, the limited spatial region can be defined as a spatial region in which the coordinate values of the W axis are -510 mm to 360 mm, the coordinate values of the L axis are -280 mm to 700 mm, and the coordinate values of the H axis are 430 mm to 750 mm. . (B) of FIG. 4 shows an example of a plurality of pixel data sets PD corresponding to a limited spatial region.

By performing the process of limiting the spatial region, the load of the discrimination process of the head 51, which will be described later, can be reduced. However, this process may be omitted.

Subsequently, the processor 32 divides the image data ID acquired from the TOF camera 2 or the multiple pixel data sets PD extracted as described above into multiple data subsets DS (STEP 2 in FIG. 3).

FIG. 5A shows a plurality of pixel data sets PD projected onto the LH coordinate plane formed by the L axis and H axis. The processor 32 divides a plurality of pixel data sets PD projected onto the LH coordinate plane into a plurality of data subsets DS, thereby dividing one of a plurality of spatial regions (vehicle spatial regions) forming a part of the vehicle interior 41 into a plurality of data subsets DS. Associate each data subset DS with one.

The vehicle interior spatial region corresponding to each data subset DS has a predetermined width dL along the L axis. The width dL can be predetermined as a dimension corresponding to the human head. "Size corresponding to the human head" is the size of the head published in the human body size database of the National Institute of Advanced Industrial Science and Technology and the human characteristics database of the National Institute of Technology and Evaluation. can be determined based on information about

Processor 32 then identifies a high-likelihood data subset LDS (STEP 3 in FIG. 3). The high-likelihood data subset LDS is a data subset DS corresponding to the vehicle interior spatial region where the head 51 of the driver 5 is likely to exist.

Specifically, processor 32 counts the number of significant pixel data sets contained in each data subset DS. By “significant pixel data set” is meant a pixel data set having distance information D(U,V) that is likely to have been obtained due to reflection of the detection light by the driver 5 . The numerical range of such distance information D can be appropriately determined based on the positional relationship between the TOF camera 2 and the driver 5 .

For multiple pixel data sets PD projected onto the LH coordinate plane, the vehicle interior spatial region corresponding to the data subset DS containing the most significant pixel data sets includes the head 51 and torso 52 of the driver 5. It is highly likely that

Accordingly, as shown in FIG. 5B, processor 32 identifies the data subset DS containing the most significant pixel data sets as the high-likelihood data subset LDS. In other words, based on the number of pixel data sets corresponding to driver 5 included in each data subset DS, processor 32 selects at least one of a plurality of vehicle interior spatial regions where head 51 is likely to be present. Identify the corresponding high-likelihood data subset LDS.

Processor 32 then performs head discrimination based on the identified high-likelihood data subset LDS (STEP 4 in FIG. 3).

First, the processor 32 identifies the top of the head 51a of the driver 5, as shown in FIG. 5B. Specifically, of the significant pixel data sets included in the high-likelihood data subset LDS, the one at the highest position is identified as the parietal region 51a. In other words, of the significant pixel data sets included in the high-likelihood data subset LDS, the one having the largest H-axis coordinate value is identified as the parietal region 51a.

Subsequently, as shown in FIGS. 6A and 6B, the processor 32 defines a head space region HSR with a high probability that the imaged head 51 of the driver 5 is included. The head space region HSR is determined based on the specified coordinates of the top of the head 51a. Specifically, based on the information on the size of the head published in the human body size database of the National Institute of Advanced Industrial Science and Technology and the human characteristics database of the National Institute of Technology and Evaluation, etc. A dimension hdL along the L axis, a dimension hdH along the H axis, and a dimension hdW along the W axis of the spatial region HSR are defined.

For example, the dimension hdL is determined to have a value of 176 mm centered on the coordinates of the top of the head 51a. The dimension hdH is determined to have a value of 146 mm downward from the coordinates of the top of the head 51a. The dimension hdW is determined to have a value of 176 mm centered on the coordinates of the top of the head 51a.

Next, the processor 32 extracts the head data subset HSD corresponding to the head spatial region HSR defined above from the plurality of pixel data sets PD, as shown in FIG. 7A. The head data subset HSD is an example of a position data subset.

Subsequently, the processor 32 performs processing for determining the orientation of the imaged face of the driver 5 (STEP 5 in FIG. 3). FIG. 8 shows a first example of the processing.

Specifically, the processor 32 counts the number of significant pixel data sets included in the head data subset HSD (STEP 511). By “significant pixel data set” is meant a pixel data set having distance information D(U,V) that is likely to have been obtained due to reflection of the detection light by the driver 5 . The significant pixel data sets included in the head data subset HSD are likely to have been obtained based on return light from the driver's 5 face.

In this embodiment, the TOF camera 2 acquires the image of the driver 5 from the upper left. If the face of the driver 5 faces the front, left, or upward, the area of the face facing the TOF camera 2 increases. Therefore, more light is returned to the TOF camera 2 and the number of significant pixel data sets contained in the head data subset HSD is also increased. On the other hand, when the face of the driver 5 faces rightward or downward, the area of the face facing the TOF camera 2 decreases. Therefore, the return light to the TOF camera 2 is reduced and the number of significant pixel data sets contained in the head data subset HSD is also reduced.

That is, based on the number of significant pixel data sets included in the head data subset HSD, it can be determined whether the driver's 5 face faces the TOF camera 2 or not.

Specifically, processor 32 determines whether the counted number of significant pixel data sets is less than a predetermined value (STEP 512). The predetermined value can be appropriately determined according to the specifications of the TOF camera 2 and the positional relationship with the driver 5 . If the counted number of significant pixel data sets is less than the predetermined value (YES in STEP 512), processor 32 determines that driver 5's face is not facing TOF camera 2 (STEP 513). If the counted number of significant pixel data sets is equal to or greater than a predetermined value (NO in STEP 512), processor 32 determines that driver 5's face faces TOF camera 2 (STEP 514).

According to the configuration as described above, the orientation of the face of the driver 5 can be determined based on the amount of information regarding the position of the head 51 of the driver 5 in the three-dimensional space. Since it is not necessary to rely only on image information, it is possible to determine the orientation of the face of the driver 5 whose face is not facing the camera.

FIG. 9 shows a second example of processing that can be performed by the processor 32 to determine the orientation of the captured driver's 5 face.

The processor 32 estimates the center position C of the head 51 of the driver 5 (STEP 521). Specifically, as shown in FIG. 7B, the processor 32 calculates the center-of-gravity position G(Wg, Lg, Hg) is first identified.

Since the head data subset HSD is acquired based on the return light from the surface of the head 51, the barycentric position G specified as described above is closer to the surface than the center of the head 51 (actual barycenter). tend to lean towards. The head 51 of the driver 5 can be approximated by a sphere S as shown in FIG. 10(A). Assuming that a plurality of pixel data sets (point elements) included in the head data subset HSD are distributed on the surface of the sphere S, the distances from each point element to the center of the sphere S are equal.

Therefore, the processor 32 searches for a point with the smallest variation in distance from each point element, and corrects the position of the center of gravity G so as to be the position of such a point. The processor 32 estimates the corrected center-of-gravity position G′ as the center position C of the head 51 .

ｎは、頭部データサブセットＨＳＤに含まれる複数の点要素の数を表している。
ｐ_kは、頭部データサブセットＨＳＤに含まれる各点要素の位置を表している。
Ｇは、上記のように特定された頭部５１の重心位置を表している。
ｄ（ｐ_k，Ｇ）は、ある点要素の位置と重心位置の間のユークリッド距離を表している。 The point with the smallest variation in distance from each point element can be searched for, for example, as the point with the smallest value of the evaluation function represented by the following equation.

n represents the number of point elements included in the head data subset HSD.
p _k represents the position of each point element contained in the head data subset HSD.
G represents the center-of-gravity position of the head 51 specified as described above.
d(p _k , G) represents the Euclidean distance between the position of a certain point element and the position of the center of gravity.

Processor 32 then identifies the radius of sphere S that approximates head 51 (STEP 522 in FIG. 9). The radius can be specified, for example, based on the Euclidean distance between the original center-of-gravity position G and the corrected center-of-gravity position G'.

(B) of FIG. 10 illustrates a case where the face of the driver 5 faces rightward in the vehicle compartment 41 . (C) of FIG. 10 illustrates a case where the face of the driver 5 faces downward in the passenger compartment 41 . In these examples where the face of the driver 5 does not face the TOF camera 2, the distribution of the point elements contained in the head data subset HSD is biased in a particular direction. The radius of a sphere S assumed to have such point elements distributed on its surface tends not to have a reasonable value.

The processor 32 then determines whether the specified radius value of the sphere S is within a predetermined range (STEP 523). The predetermined range can be, for example, 50 mm to 90 mm. If it is determined that the radius of the sphere S is not within the predetermined range (NO in STEP 523), the processor 32 determines that the face of the driver 5 is not facing the TOF camera 2 (STEP 524). If it is determined that the radius of sphere S is within the predetermined range (YES in STEP 523), processor 32 determines that driver 5's face faces TOF camera 2 (STEP 525).

According to the above configuration, the estimated position of the center of the head 51 from the information on the position of the head 51 of the driver 5 in the three-dimensional space and the radius of the sphere S that approximates the head 51 centering on the estimated position is obtained, and the orientation of the face of the driver 5 can be determined based on the value of the radius. Since it is not necessary to rely only on image information, it is possible to determine the orientation of the face of the driver 5 whose face is not facing the camera.

As shown in FIG. 9, if it is determined that the radius of the sphere S is within the predetermined range (YES in STEP 523), the processor 32 moves from the center position C of the head 51 (that is, the center of the sphere S) An average value of differences between the distance to each point element included in the head data subset HSD and the radius of the sphere S can be calculated. The average value is an example of a representative value. The average value can be replaced by the median value or the mode value of the difference between the distance from the center position C of the head 51 to each point element included in the head data subset HSD and the radius of the sphere S.

When the face of the driver 5 faces the TOF camera 2 as shown in FIG. 10A, the sphere S can approximate the actual head 51 well, so the head data subset HSD The distance to each of the plurality of point elements contained in tends to approach the radius of the sphere S. On the other hand, when the face of the driver 5 does not face the TOF camera 2 as shown in (B) and (C) of FIG. Due to the directional bias, the distance from the center position C to each of the plurality of point elements contained in the head data subset HSD tends to be away from the radius of the sphere S.

The processor 32 then determines whether the calculated average value of the differences is less than a predetermined value (STEP 526 in FIG. 9). The predetermined value can be 10 mm, for example. If it is determined that the average value of the calculated differences is less than the predetermined value (YES in STEP 526), processor 32 determines that driver 5's face faces TOF camera 2 (STEP 525). If it is determined that the average value of the calculated differences is equal to or greater than the predetermined value (NO in STEP 526), processor 32 determines that driver 5's face faces TOF camera 2 (STEP 524).

According to such a configuration, it is possible to more precisely determine the direction of the face of driver 5 using the radius of sphere S that approximates head 51 .

The orientation of the face of the driver 5 may be determined based only on the determination process of STEP526 instead of the determination process of STEP523.

If it is determined that the face of driver 5 is not facing the TOF camera (STEP 524), processor 32 may further perform processing to identify the orientation of driver 5's face (STEP 527). Specifically, it is specified whether the face of the driver 5 is facing rightward or downward in the passenger compartment 41 .

As shown in FIG. 11A, when the face of the driver 5 faces rightward in the vehicle interior 41, the center-of-gravity position G of the plurality of point elements included in the head data subset HSD is the head It tends to be located relatively forward (+L direction) when viewed from the center position C of 51 . On the other hand, as shown in FIG. 11B, when the face of the driver 5 faces downward in the vehicle interior 41, the center-of-gravity position G of the plurality of point elements included in the head data subset HSD is the head It tends to be positioned relatively to the left (-W direction) when viewed from the center position C of the portion 51 .

Based on the positional relationship between the position G of the center of gravity and the position C of the center of gravity, the processor 32 determines whether the face of the driver 5 is facing rightward or downward. Specifically, as shown in FIG. 11C, a specific direction within the WL coordinate plane viewed from the center position C is set as the reference direction RD. The processor 32 determines that the face of the driver 5 faces rightward in the vehicle interior 41 if the center of gravity position G is positioned to the right of the reference direction RD when viewed from the center position C. The processor 32 determines that the face of the driver 5 faces downward in the vehicle interior 41 when the center of gravity position G is positioned to the left of the reference direction RD when viewed from the center position C.

According to such a configuration, even when the face of the driver 5 does not face the TOF camera 2, the specific face orientation can be determined.

In the first processing example and the second processing example that have been described so far, the input interface 31 of the face direction determination device 3 converts the information about the position of the head 51 of the driver 5 in the three-dimensional space into the image data ID is received from the TOF camera 2 as part of the . According to such a configuration, it is possible to efficiently acquire the position information related to the head 51 of the driver 5 together with the image information. However, the imaging device that outputs image information and the device that outputs position information may be different. The latter device can be exemplified by a LiDAR (Light Detection and Ranging) sensor.

FIG. 12 shows a third example of processing for determining the direction of the face of driver 5 that can be performed by processor 32 .

The processor 32 first acquires the image data ID through the input interface 31 (STEP 531).

Subsequently, the processor 32 sets a detection area DA that may include the head 51 of the driver 5 imaged in the image data ID (STEP 532). Identification of the detection area DA can be performed using an appropriate image recognition technique. (A) of FIG. 13 shows an example of the detection area DA specified in the image data ID. Detection area DA is an example of a first detection area.

Subsequently, the processor 32 uses a discriminator to determine whether or not the imaged face of the driver 5 included in the detection area DA faces the TOF camera 2 (STEP 533). The discriminator is configured by learning an image including at least the captured head as teacher data in order to discriminate the direction of the face of the driver 5 captured based on the feature amount related to the brightness of the image. . A discriminator is a designation for a processing algorithm executed by processor 32 .

In this embodiment, HOG (Histogram of Oriented Gradients) is detected as the feature quantity. HOG is a feature quantity that indicates the distribution of local luminance in the gradient direction of an image.

(B) of FIG. 13 illustrates a specific configuration of the detection area DA. The size of the detection area DA is determined based on the size of the head 51 of the driver 5 that can be included in the image acquired by the TOF camera 2 . For example, if the maximum size that the head 51 can take in the image is 48×56 pixels, the size of the detection area DA set in the image data ID can also be 48×56 pixels.

A detection cell DC and a detection block DB are defined in the detection area DA. A detection cell DC is the minimum unit of area set for detecting HOG. In this embodiment, the detection cells DC have a vertically long shape in order to facilitate detection of the edges of the face of the driver 5 extending in the vertical direction. For example, the size of the detection cell DC can be 4×8 pixels. A detection block DB is an area having a size of 3×3 cells.

HOG detection is performed for each of nine detection cells DC included in detection block DB. That is, HOG for 9 cells is detected for the detection block DB positioned at the upper left corner of the detection area DA shown in FIG. 13B. When HOG is detected for all the detection cells DC included in the detection block DB, the position of the detection block DB within the detection area DA is changed. Specifically, the detection block DB is moved rightward by four pixels (that is, by one cell). HOG for nine cells is similarly detected for all the detection cells DC included in the detection block DB at the position after the movement.

HOG detection is repeated while the detection block DB moves to the right in this manner. When the detection block DB reaches the right end of the detection area DA, it is moved downward by eight pixels (that is, by one cell), and HOG detection is repeated while moving leftward within the detection area DA. When the detection block DB reaches the left end of the detection area DA, it is further moved downward by 8 pixels, and the HOG detection is repeated while moving rightward again within the detection area DA. The above process is repeated until the detection block DB reaches the lower right corner of the detection area DA.

When HOG detection is completed while moving the detection block DB, identification is performed by the identifier. The discriminator performs machine learning using an image including the face facing the TOF camera 2 and an image including the face not facing the TOF camera 2 as training data, and determines whether the face of the driver 5 is not facing the TOF camera 2. It is configured to determine likelihoods corresponding to probabilities. In this embodiment, the discriminator is a soft-margin SVM (support vector machine) using the RBF kernel function. Likelihoods are calculated as SVM scores.

Specifically, the processor 32 inputs the HOG distribution obtained for the detection area DA to the discriminator. The discriminator calculates an SVM score for the detection area DA. If the calculated SVM score is less than the predetermined value (NO in STEP 533 of FIG. 12), the processor 32 determines that the face of the driver 5 in the image within the detection area DA is likely to face the TOF camera 2. (STEP 534). This determination result obtained for the detection area DA is an example of the first determination result.

If the calculated SVM score is equal to or greater than a predetermined value (YES in STEP 533 of FIG. 12), processor 32 determines that the face of driver 5 in the image within detection area DA is not facing TOF camera 2. Perform processing to verify.

Specifically, the processor 32 sets the comparison area CA at a position partially overlapping the detection area DA (STEP 535). The comparison area CA has the same size as the detection area DA. The comparison area CA also has the same configuration as the detection area DA shown in FIG. 13(B). The comparison area CA is an example of a second detection area.

For example, as shown in FIG. 14A, the comparison area CA is set so as to correspond to a position offset upward by eight pixels from the detection area DA. The SVM score is calculated based on the above-described HOG detection and matching with the discriminator for the image included in the comparison area CA set in this manner. The processor 32 determines the possibility that the face of the driver 5 in the image within the comparison area CA is not facing the TOF camera based on the calculated SVM score. This determination result obtained for the comparison area CA is an example of the second determination result.

Subsequently, as shown in FIG. 14B, the processor 32 sets a comparison area CA at a position offset by eight pixels to the right from the comparison area CA shown in FIG. 14A. The processor 32 determines the possibility that the face of the driver 5 in the image within the comparison area CA is not facing the TOF camera in the same manner as described above.

(C) of FIG. 14 shows a comparison area CA set at a position offset downward by eight pixels from the comparison area CA shown in (B) of FIG. 14 . (D) of FIG. 14 shows a comparison area CA set at a position offset downward by eight pixels from the comparison area CA shown in (C) of FIG. 14 . (E) of FIG. 14 shows a comparison area CA set at a position offset by eight pixels to the left from the comparison area CA shown in (D) of FIG. 14 . (F) of FIG. 14 shows a comparison area CA set at a position offset by eight pixels to the left from the comparison area CA shown in (E) of FIG. 14 . (G) of FIG. 14 shows a comparison area CA set at a position offset by eight pixels upward from the comparison area CA shown in (F) of FIG. (H) of FIG. 14 shows a comparison area CA set at a position offset by eight pixels upward from the comparison area CA shown in (G) of FIG. 14 .

That is, the processor 32 determines whether the face of the driver 5 in the image within the comparison area CA is not facing the TOF camera for each of eight comparison areas CA set so as to partially overlap the detection area DA. Judge the possibilities.

Subsequently, the processor 32 determines whether or not the number of comparison areas CA in which it is determined that "the face of the driver 5 does not face the TOF camera 2" is equal to or greater than a predetermined value (STEP 536). The predetermined value can be 1, for example. In other words, the processor 32 determines whether the number of second determination results matching the first determination result is equal to or greater than a predetermined value.

If the number of comparison areas CA judged that "the face of the driver 5 does not face the TOF camera 2" is equal to or greater than a predetermined value (YES in STEP 536), the processor 32 validates the first judgment result. . That is, it is determined that the face of the driver 5 is not facing the TOF camera 2 (STEP 537). If the number of comparison areas CA judged that "the face of the driver 5 does not face the TOF camera 2" is less than the predetermined value (NO in STEP 536), the processor 32 invalidates the first judgment result. . That is, it is determined that the face of the driver 5 faces the TOF camera 2 (STEP 534).

In the above example, after obtaining the second discrimination results for the eight comparison areas CA shown in (A) to (H) of FIG. being judged. However, the number of comparison areas CA to be set can be changed as appropriate. In addition, each time a second discrimination result is obtained in the set comparison area CA, it is compared with the above-described predetermined value, and when the number of second discrimination results that match the first discrimination result reaches the predetermined value, the comparison is made. The setting of the area CA may be canceled. In this case, the transition to STEP 537 is made immediately.

According to the configuration as described above, the orientation of the face of the driver 5 captured is determined using a classifier that has learned the image including the captured head as training data, and the determination result is also verified. Therefore, it is possible to improve the accuracy of determination of the direction of the face of the occupant whose face is not facing the camera.

If it is determined that the face of the driver 5 is not facing the TOF camera (STEP 537), the processor 32 further performs the process of identifying the orientation of the face of the driver 5 described with reference to FIGS. can be done (STEP 538). That is, the processor 32 extracts the head data subset HSD corresponding to the head 51 of the driver 5 included in the detection area DA from the image data ID, and estimates the center position C of the head 51 . Subsequently, the processor 32 determines whether the face of the driver 5 is located in the vehicle interior based on the positional relationship between the estimated center position C and the center-of-gravity position G of a plurality of pixel data sets (point elements) included in the head data subset HSD. It is specified whether it is facing rightward or downward in 41 .

According to such a configuration, even when the face of the driver 5 does not face the TOF camera 2, the specific face orientation can be determined.

In the third processing example described so far, the input interface 31 of the face direction determination device 3 receives information regarding the position of the head 51 of the driver 5 in the three-dimensional space as part of the image data ID from the TOF camera 2. are accepted from According to such a configuration, in order to perform processing such as STEP 538, it is possible to efficiently acquire the position information related to the head 51 of the driver 5 together with the image information. However, if the process requiring the position information of the head 51 of the driver 5 as in STEP 538 is not performed, instead of the TOF camera 2, an imaging device that outputs only the image data ID can be used.

As shown in FIG. 1 , the face direction determination device 3 has an output interface 33 . The output interface 33 can output data FD indicating the determined face direction of the driver 5 . The output data FD is used in subsequent recognition processing. In the recognition process, for example, the orientation, inclination, movement, etc. of the head 51 of the driver 5 can be recognized by monitoring changes over time in the orientation of the face indicated by the data. As a result, it is possible to detect whether the driver 5 looks aside, dozes off, or has an abnormal behavior due to a seizure while driving.

The functions of processor 32 described above may be implemented by a general-purpose microprocessor. A CPU, MPU, and GPU can be exemplified as a general-purpose microprocessor. Examples of general-purpose memory include ROM and RAM. In this case, the ROM may store a computer program for executing the above-described process. ROM is an example of a storage medium that stores computer programs. The processor 32 designates at least part of the program stored on the ROM, develops it on the RAM, and cooperates with the RAM to execute the above-described processing. Processor 32 may be implemented by a dedicated integrated circuit such as a microcontroller, ASIC, or FPGA capable of executing a computer program that implements the processes described above. Processor 32 may be implemented by a combination of a general purpose microprocessor and a dedicated integrated circuit.

As shown in FIG. 1 , the face direction determination device 3 can be configured to communicate with an external server 7 via a network 6 . In this case, a computer program for executing the processes described above can be downloaded from the external server 7 via the network 6 . The external server 7 is an example of a storage medium storing computer programs.

The above embodiments are merely examples for facilitating understanding of the present invention. The configurations according to the above embodiments can be modified and improved as appropriate without departing from the scope of the present invention.

In the above-described embodiment, the outer shape of the head is captured using HOG, which can express the rough shape of the object based on the brightness gradient of the edge, as the feature amount related to the brightness of the image. However, Haar-Like corresponding to the local brightness difference of the image, or LBP (Local Binary Pattern) corresponding to the local luminance distribution of the image may be used as the feature quantity.

In the above embodiment, an SVM with relatively high generalization performance is used as a discriminator. However, random forests and perceptrons may also be used as discriminators.

In the above-described embodiment, the “direction of the face not facing the TOF camera 2 ” is rightward and downward in the passenger compartment 41 . However, "the direction of the face not facing the TOF camera 2" can change according to the positional relationship between the TOF camera 2 and the driver's seat. For example, in the case of a vehicle in which the driver's seat is arranged on the left side of the passenger compartment 41, the TOF camera 2 acquires an image of the driver 5 from the upper right. In this case, the “direction of the face not facing the TOF camera 2 ” is leftward and downward in the passenger compartment 41 .

In the above embodiment, the orientation of the face of the driver 5 captured in the vehicle interior 41 is used for determination. However, by appropriately arranging the TOF camera 2, the face direction of the captured other occupant may be used for determination.

2: TOF camera, 3: face direction determination device, 31: input interface, 32: processor, 41: vehicle interior, 5: driver, 51: head, 7: external server, ID: image data, PD: pixel data set, HSD: head data subset, C: head center position, G: centroid position of multiple point elements, S: sphere, DA: detection area, CA: comparison area

Claims (7)

an input interface that accepts a position data set that includes a plurality of point elements each associated with a position in three-dimensional space;
a processor for determining the face orientation of an occupant in the vehicle interior based on the position data set;
and
The processor
extracting from the position data set a position data subset corresponding to a region in three-dimensional space where the occupant's head is likely to exist;
estimating a center position of the head based on the position data subset;
identifying the radius of a sphere approximating the head centered at the center location;
determining the orientation of the occupant's face based on the radius;
The input interface receives the position data set from a TOF (Time of Flight) camera,
each of the plurality of point elements corresponds to one of the plurality of pixel data output from the TOF camera;
Face direction identification device. The processor
calculating a representative value of the distance from the center position to each of the plurality of point elements included in the position data subset;
determining the orientation of the occupant's face based on a comparison of the representative value and the radius;
The face direction determination device according to claim 1 . The processor determines the orientation of the occupant's face based on the positional relationship between the center position and the barycentric positions of the plurality of positions in the three-dimensional space associated with the plurality of point elements included in the position data subset. determine the
3. The face direction determination device according to claim 1 or 2 . A computer program for causing a processor to determine the face orientation of an occupant in a vehicle interior based on a position data set including a plurality of point elements each associated with a three-dimensional position, comprising:
When the computer program is executed, the processor:
extracting from the position data set a position data subset corresponding to a spatial region where the occupant's head is likely to exist;
estimating a center position of the head based on the position data subset;
identify the radius of a sphere approximating the head centered at the center position;
determining the orientation of the occupant's face based on the radius;
The position data set is obtained from a TOF (Time of Flight) camera,
each of the plurality of point elements corresponds to one of the plurality of pixel data output from the TOF camera;
computer program. to the processor;
calculating a representative value of distances from the center position to each of the plurality of point elements included in the position data subset;
determining the orientation of the occupant's face based on the comparison between the representative value and the radius;
5. Computer program according to claim 4 . The processor determines the direction of the occupant's face based on the positional relationship between the centroid position of the plurality of three-dimensional positions associated with the plurality of point elements included in the position data subset and the center position. let
6. Computer program according to claim 4 or 5 . A storage medium storing the computer program according to any one of claims 4 to 6 .

Images