JP2023012364A - Detection apparatus - Google Patents

Abstract

To provide a detection apparatus which can improve robustness for accuracy of detecting a position of an object on an image.SOLUTION: A detection apparatus includes: a pre-processing unit 31 which applies a plurality of different pre-processes to an image representing a detection target, to generate a plurality of pre-processed images; a detection unit 32 which inputs the image and each of the pre-processed images to a detector for detecting the detection target, to detect a position of the detection target in the image and each of the pre-processed images; and a position specifying unit 33 which specifies a statistical representative value of the positions of the detection target in the image and each of the pre-processed images, as a position of the detection target on the image.SELECTED DRAWING: Figure 3

Description

The present invention relates to a detection device for detecting an object represented in an image.

Techniques for detecting a given object represented in an image have been researched. In particular, research has been conducted on techniques for detecting a given object from an image by inputting the image to a detector that has been pre-trained to detect the given object. In such technology, images obtained under various environments and a large number of images generated by various cameras with different specifications are prepared as teacher images used for learning the detector. desirable. If only an image obtained under a specific environment or an image generated by a camera with specific specifications is used as the teacher image, the detector may not exhibit sufficient detection accuracy. Therefore, a technique has been proposed in which a plurality of images are generated from a single image representing an identification target, and the plurality of images are used to identify the state of the identification target (see, for example, Patent Document 1). .

The state identification device disclosed in Patent Literature 1 performs a process of changing the appearance of an identification target in one input image of the identification target and maintaining that state, and performs this process. Generate an image group containing the image generated in . The state identifier then uses the generated images in building a discriminative model and/or in using the discriminative model to identify a state. In addition, the state identification device changes at least one type of appearance in one image in a direction opposite to the direction in the one image.

JP 2018-116589 A

In the above technique, an image group generated from one image is used to identify the state of the identification object. However, the detection accuracy of the position of the object to be detected on the image may be degraded due to differences in the imaging environment at the time of image generation or the camera that generated the image.

SUMMARY OF THE INVENTION Accordingly, it is an object of the present invention to provide a detection apparatus capable of improving robustness with respect to detection accuracy of the position of an object on an image.

According to one embodiment, a detection device is provided. This detection device includes a preprocessing unit that generates a plurality of preprocessed images by applying a plurality of different preprocesses to an image representing a detection target, and an image and the plurality of preprocessed images. are input to a detector for detecting a detection target, a detection unit that detects the position of the detection target in each of the image and the plurality of preprocessed images, and the image and the plurality of preprocessed images. a position specifying unit that specifies a statistical representative value of the position of the detection target in each as the position of the detection target on the image.

Advantageous Effects of Invention The detection device according to the present invention has the effect of being able to improve the robustness with respect to the detection accuracy of the position of an object on an image.

1 is a schematic configuration diagram of a vehicle control system in which a detection device is mounted; FIG. It is a hardware block diagram of the electronic control unit which is one embodiment of a detection apparatus. FIG. 4 is a functional block diagram of the processor of the electronic control unit relating to detection processing; FIG. 4 is an explanatory diagram of an overview of detection processing according to the present embodiment; 4 is an operation flowchart of detection processing;

A detection device, and a detection method and detection computer program executed on the detection device will be described below with reference to the drawings. This detection device generates a plurality of preprocessed images by applying a plurality of mutually different preprocessings to an original image obtained by photographing a detection target. This detection device inputs an original image and a plurality of preprocessed images, respectively, to a detector for detecting a detection target, thereby detecting a detection target in each of the original image and the plurality of preprocessed images. Detect location. Then, the detection device detects the position by specifying the statistical representative value of the position of the detection target in each of the original image and the plurality of preprocessed images as the position of the detection target on the original image. Improve accuracy robustness.

An example in which the detection device is applied to a driver monitoring device for monitoring a driver based on a series of time-series images obtained by continuously photographing the face of a driver of a vehicle will be described below. This driver monitor device extracts an image (hereinafter referred to as a face image for convenience of explanation) generated by a driver monitor camera provided to photograph the driver's head, and extracts individual organs (such as eyes) of the driver's face. , nose, mouth, etc.). The driver monitoring device determines the driver's condition based on the detected feature points of each organ. That is, the feature points of each organ of the driver's face are an example of the detection target. Here, this driver monitor device executes a plurality of preprocessings on a face image to generate a plurality of preprocessed images, and inputs the face image and each preprocessed image to a detector, respectively, thereby detecting the face. One or more feature points are detected for each organ of the driver's face from the image and each preprocessed image. For each feature point, the driver monitor device takes the statistical representative value of the position of the feature point detected from the face image and each preprocessed image as the position of the feature point in the face image.

It should be noted that the detection device according to this embodiment is not limited to a driver monitor device, and is required to detect an object from an image obtained by a camera that takes a picture of the detection object, such as a web camera or other surveillance camera. Suitable for various uses. Moreover, the detection target is not limited to individual organs of a person's face, but may be parts other than the head of a person (for example, hands, feet, etc.), the whole body, or various objects other than a person. (For example, it may be a vehicle, an animal other than a person, etc.).

FIG. 1 is a schematic configuration diagram of a vehicle control system in which a detection device is mounted. FIG. 2 is a hardware configuration diagram of an electronic control unit, which is one embodiment of the detection device. In this embodiment, a vehicle control system 1 mounted on a vehicle 10 and controlling the vehicle 10 includes a driver monitor camera 2, a user interface 3, and an electronic control unit (ECU) 4 which is an example of a detection device. have. The driver monitor camera 2, the user interface 3, and the ECU 4 are communicably connected via an in-vehicle network conforming to a standard such as a controller area network. The vehicle control system 1 may further include a GPS receiver (not shown) for positioning the vehicle 10 itself. In addition, the vehicle control system 1 includes a camera (not shown) for photographing the surroundings of the vehicle 10, or a distance sensor such as LiDAR or radar that measures the distance from the vehicle 10 to objects existing around the vehicle 10. (not shown). Furthermore, the vehicle control system 1 may have a wireless communication terminal (not shown) for wireless communication with other devices. Furthermore, the vehicle control system 1 may have a navigation device (not shown) for searching the travel route of the vehicle 10 .

The driver monitor camera 2 is an example of a camera or an in-vehicle imaging unit, and includes a two-dimensional detector configured by an array of photoelectric conversion elements sensitive to visible light or infrared light, such as a CCD or C-MOS; It has an imaging optical system that forms an image of an area to be photographed on the dimensional detector. The driver monitor camera 2 may further have a light source for illuminating the driver, such as an infrared LED. Then, the driver monitor camera 2 is mounted on the instrument panel or the like so that the head of the driver seated in the driver's seat of the vehicle 10 is included in the photographing target area, that is, the head of the driver can be photographed. It is mounted in close proximity to the driver. Then, the driver monitor camera 2 photographs the driver's head at predetermined photographing intervals (for example, 1/30 second to 1/10 second) to generate a facial image representing the driver's face. The face image obtained by the driver monitor camera 2 may be a color image or a gray image. Each time the driver monitor camera 2 generates a face image, it outputs the generated face image to the ECU 4 via the in-vehicle network.

The user interface 3 is an example of a notification unit, and has a display device such as a liquid crystal display or an organic EL display. The user interface 3 is installed in the interior of the vehicle 10, for example, on an instrument panel, facing the driver. The user interface 3 notifies the driver of the information by displaying various information received from the ECU 4 via the in-vehicle network. The user interface 3 may also have a speaker installed inside the vehicle. In this case, the user interface 3 notifies the driver of various information received from the ECU 4 via the in-vehicle network by outputting the information as an audio signal.

The ECU 4 detects the orientation of the driver's face based on the face image, and determines the driver's state based on the orientation of the face. Then, the ECU 4 warns the driver via the user interface 3 when the driver is in a state unsuitable for driving such as looking away.

As shown in FIG. 2, the ECU 4 has a communication interface 21, a memory 22, and a processor . The communication interface 21, memory 22 and processor 23 may each be configured as separate circuits, or may be integrally configured as one integrated circuit.

The communication interface 21 has an interface circuit for connecting the ECU 4 to the in-vehicle network. Then, the communication interface 21 passes the received face image to the processor 23 each time it receives a face image from the driver monitor camera 2 . Further, when receiving information to be displayed on the user interface 3 from the processor 23 , the communication interface 21 outputs the information to the user interface 3 .

The memory 22 is an example of a storage unit, and has, for example, a volatile semiconductor memory and a nonvolatile semiconductor memory. The memory 22 stores various algorithms and various data used in driver monitor processing including detection processing executed by the processor 23 of the ECU 4 . For example, the memory 22 stores various parameters used for generating a preprocessed image and various parameters used for detecting feature points of each organ of the face. Furthermore, the memory 22 temporarily stores various data generated during the driver monitor process, such as the face image received from the driver monitor camera 2 and the preprocessed image.

The processor 23 has one or more CPUs (Central Processing Units) and their peripheral circuits. Processor 23 may further comprise other arithmetic circuitry such as a logic arithmetic unit, a math unit or a graphics processing unit. The processor 23 then executes driver monitor processing including detection processing.

FIG. 3 is a functional block diagram of processor 23 relating to driver monitor processing. The processor 23 has a preprocessing section 31 , a detection section 32 , a position specifying section 33 and a state determination section 34 . These units of the processor 23 are, for example, functional modules implemented by computer programs running on the processor 23 . Alternatively, each of these units of processor 23 may be a dedicated arithmetic circuit provided in processor 23 . Among these units of the processor 23, the preprocessing unit 31, the detection unit 32, and the position specifying unit 33 are related to detection processing.

The preprocessing unit 31 generates a plurality of preprocessed images by applying a plurality of different preprocesses to the face image received by the ECU 4 from the driver monitor camera 2 .

In the present embodiment, the preprocessing unit 31 performs at least one of conversion processing of the size or aspect ratio of the face image and conversion processing of the contrast as preprocessing, for example. Furthermore, the preprocessing unit 31 may perform color correction processing, edge enhancement processing, or smoothing processing on the face image as preprocessing. Also, the preprocessing unit 31 may perform a combination of some of the above-described size conversion, contrast conversion, color correction, and other processing as preprocessing.

For example, the preprocessing unit 31 resamples the face image at a predetermined sampling rate, and converts the size of the face image in the horizontal direction and the vertical direction as one of the preprocessed images. Generate an image. In this case, the preprocessing unit 31 trims a partial area assumed to represent a specific part of the face (for example, eyes) from the face image, resamples the partial area, and resizes the partial area. may be added as one of the preprocessed images. Note that the partial area can be, for example, an area in which a specific part of the face is represented in the face image obtained immediately before.

In addition, the preprocessing unit 31 resamples the face image at different sampling rates in the vertical direction and the horizontal direction, and converts the aspect ratio of the face image as another preprocessed image. Generate an image. Note that the preprocessing unit 31 may apply methods such as simple thinning, nearest neighbor method, bilinear interpolation, and bicubic interpolation as the above resampling method. Then, the preprocessing unit 31 may apply two or more different resampling methods to one face image to generate two or more size-converted images or aspect-ratio-converted images. Furthermore, the preprocessing unit 31 may apply different sampling rates to generate two or more sampled images or aspect ratio converted images having different sizes or aspect ratios from one face image.

Further, the preprocessing unit 31 converts the value of each pixel of the face image according to a contrast conversion curve that indicates the relationship between the value of the input pixel and the value of the output pixel, thereby converting the value of each pixel of the face image into another one of the preprocessed images. , a contrast-converted image is generated by converting the contrast of the face image. Note that the pixel value can be, for example, luminance or each color component value. Further, the preprocessing unit 31 calculates a color transformation coefficient so that the color of a predetermined area in the face image becomes a preset reference color, and uses the calculated color transformation coefficient to determine the value of each pixel of the face image. , a color-corrected image obtained by color-correcting the face image is generated as another preprocessed image. In this case, the predetermined area is preferably an area in which a predetermined object in the vehicle compartment (for example, the ceiling of the vehicle compartment) is always represented regardless of the driver's posture.

Furthermore, the preprocessing unit 31 applies a predetermined edge enhancement filter to the face image to generate an edge-enhanced image in which the edges of the face image are enhanced as another preprocessed image. Furthermore, the preprocessing unit 31 applies a predetermined smoothing filter to the face image to generate a smoothed image obtained by smoothing the face image as another preprocessed image.

The preprocessing unit 31 outputs each generated preprocessed image to the detection unit 32 .

The detection unit 32 detects one or more feature points for each organ of the driver's face from the face image and each preprocessed image.

In order to detect one or more feature points for each organ of the face, the detection unit 32 is trained in advance to detect the feature points of the organ (for example, the inner corner of the eye, the outer corner of the eye, the tip of the nose, the corner of the mouth, etc.). Input a face image to the detector. Thereby, the detection unit 32 detects one or more feature points for each organ of the face from the face image. Similarly, the detection unit 32 detects one or more feature points of each organ of the face in each preprocessed image by inputting each preprocessed image into the detector. The detection unit 32 uses a deep neural network (DNN) having a convolutional neural network (CNN) architecture such as a Single Shot MultiBox Detector (SSD) or Faster R-CNN as such a detector. can be used. Alternatively, the detector 32 may use detectors pre-trained to detect feature points of individual facial organs based on other machine learning techniques, such as support vector machines or AdaBoost, as such detectors. may Alternatively, the detector 32 may use a detector that uses information of the entire face, such as an Active Shape Model (ASM) or an Active Appearance Model (AAM).

The detection unit 32 outputs to the position specifying unit 33 information representing one or more feature points of each organ of the face detected for the face image and each preprocessed image. The information representing one or more feature points for each organ includes, for example, the position coordinates of each feature point, facial organs represented by the feature points (e.g., eyes, nose, mouth, etc.) and positions in the organs (e.g., and an identification number indicating the inner corner of the eye, the outer corner of the eye, the tip of the nose, the corner of the mouth, etc.).

The position specifying unit 33, for each of one or more feature points of individual organs of the driver's face detected in the face image and each preprocessed image, calculates a statistical representative value of the position of the feature point on the face image. It is specified as the position of the feature point.

For example, the position specifying unit 33 calculates the center of gravity or the average value of the position of the feature point of interest in each of the face image and each preprocessed image as the statistical representative value of the position of the feature point. Further, when the preprocessed image generated by the preprocessing unit 31 includes a size-converted image or an aspect-ratio-converted image, the position specifying unit 33 generates the size-converted image or the aspect-ratio-converted image. Transformation processing and inverse transformation processing are performed on feature points. As a result, the position specifying unit 33 can express the position of the feature point on the size-converted image or the aspect-ratio-converted image by coordinates in the original face image. Also, if one of the preprocessed images includes an image obtained by resampling a partial area, the position specifying unit 33 resamples the preprocessed image at the reciprocal of the sampling rate at the time of generation. Thus, the position of the feature point on the resampled preprocessed image is obtained. Furthermore, the position specifying unit 33 corrects the position of the feature point according to the position of the partial area in the original face image, so that the position of the feature point can be represented by the coordinates in the original face image.

According to the modified example, the position specifying unit 33 excludes, from among the positions of the feature points in each of the face image and each preprocessed image, the feature points that satisfy a predetermined outlier criterion for the feature points of interest. You may Then, the position specifying unit 33 may calculate a statistical representative value of the positions of the feature points of interest based on the positions of the remaining feature points. In this case, the position specifying unit 33 may set a predetermined outlier criterion according to, for example, the k-nearest neighbor method, the Random Sample Consensus (RANSAC) method, or the Least Median of Square (LMedS) method. In this way, the position specifying unit 33 excludes, from among the feature points of interest detected from each of the face image and each preprocessed image, feature points whose positions satisfy the outlier criterion from the calculation of the statistical representative value. By doing so, the position of the feature point of interest can be calculated more accurately. Further, when a plurality of feature points are detected for one organ, the position specifying unit 33 changes the correspondence between the feature points in the face image and each preprocessed image for that organ, and changes the correspondence between the feature points. Calculate the sum of squared distances for the set of feature points. Then, the position specifying unit 33 may set the pair of feature points when the sum of the squares of the distances is the minimum as the feature points corresponding to each other.

FIG. 4 is an explanatory diagram of an overview of detection processing according to the present embodiment. The input face image 400 is subjected to n different preprocessing (n is an integer of 2 or more) to generate n preprocessed images 401-1 to 401-n. Then, the face image 400 and the preprocessed images 401-1 to 401-n are input to the detector 402, so that each facial organ is detected from the face image 400 and the preprocessed images 401-1 to 401-n. A feature point 403 is detected at . Then, for each feature point 403, feature points that satisfy the outlier criterion are excluded from the positions of the feature points in the face image 400 and the preprocessed images 401-1 to 401-n. Then, the position of the feature point 403 in the face image 400 is specified as a statistical representative value of the position of that feature point in each non-excluded image.

The position specifying unit 33 notifies the state determination unit 34 of each position of one or more feature points of each organ of the driver's face.

The state determination unit 34 determines the state of the driver based on the position of the feature point of each organ of the driver's face represented in the face image.

In this embodiment, the state determination unit 34 compares the direction of the driver's face shown in the face area with the reference direction of the driver's face to determine whether the driver's state is suitable for driving the vehicle 10. determine whether Note that the reference direction of the face is stored in the memory 22 in advance.

The state determination unit 34 fits the feature points of individual organs of the face to a three-dimensional face model representing the three-dimensional shape of the face. Then, the state determination unit 34 detects the face orientation of the three-dimensional face model when each feature point is best fitted to the three-dimensional face model as the face orientation of the driver. Alternatively, the state determination unit 34 may detect the orientation of the driver's face according to another method of determining the orientation of the face represented in the image based on the feature points of each organ of the face. The orientation of the driver's face is represented by, for example, a combination of the pitch angle, yaw angle, and roll angle with reference to the direction facing the driver monitor camera 2 .

The state determination unit 34 calculates the absolute value of the difference between the orientation of the driver's face represented in the face area and the reference direction of the driver's face, and compares the absolute value of the difference with a predetermined permissible face orientation range. Then, if the absolute value of the difference is out of the allowable face orientation range, the state determination unit 34 determines that the driver is looking away, that is, the driver is not in a state suitable for driving the vehicle 10 .

It should be noted that the driver may face a direction other than the front of the vehicle 10 in order to check the situation around the vehicle 10 . However, even in such a case, if the driver is concentrating on driving the vehicle 10, the driver will not continue to face the vehicle 10 in any direction other than the front. Therefore, according to the modified example, the state determination unit 34 determines that the period in which the absolute value of the difference between the orientation of the driver's face and the reference direction of the driver's face is outside the allowable face orientation range is a predetermined time (for example, several seconds). If this continues, it may be determined that the driver's condition is not suitable for driving the vehicle 10 .

When the state determination unit 34 determines that the driver's state is not suitable for driving the vehicle 10 , the state determination unit 34 generates warning information including a warning message to warn the driver to face the front of the vehicle 10 . Then, the state determination unit 34 outputs the generated warning information to the user interface 3 via the communication interface 21, thereby causing the user interface 3 to display the warning message. Alternatively, the state determination unit 34 causes the speaker of the user interface 3 to output a sound warning the driver to face the front of the vehicle 10 .

FIG. 5 is an operation flowchart of driver monitor processing executed by the processor 23 . The processor 23 may execute the driver monitor process including the detection process according to the following operation flowchart. In the operation flowchart shown below, the processing of steps S101 to S103 corresponds to the detection processing.

The preprocessing unit 31 of the processor 23 generates a plurality of preprocessed images by applying a plurality of different preprocesses to the face image received by the ECU 4 from the driver monitor camera 2 (step S101). Further, the detection unit 32 of the processor 23 detects one or more feature points for each organ of the driver's face from the face image and each preprocessed image (step S102).

The position specifying unit 33 of the processor 23 calculates, for each one or more feature points of individual organs of the driver's face detected in the face image and each preprocessed image, a statistical representative value of the position of the feature point in the face image. It is specified as the position of the feature point on the top (step S103). Note that, as described above, the position specifying unit 33 calculates the statistical representative value of the feature points whose positions satisfy the predetermined outlier criterion among the feature points detected from each of the face image and each preprocessed image. can be excluded from

The state determination unit 34 of the processor 23 detects the orientation of the driver's face based on the positions of the feature points of the individual organs, and determines whether or not the driver's state is suitable for driving the vehicle 10 (step S104). ). Then, the state determination unit 34 executes warning processing or the like according to the determination result. After that, the processor 23 terminates the driver monitor process.

As described above, this detection device generates a plurality of preprocessed images by applying a plurality of different preprocessings to the original image obtained by photographing the detection target. do. This detection device inputs an original image and a plurality of preprocessed images, respectively, to a detector for detecting a detection target, thereby detecting a detection target in each of the original image and the plurality of preprocessed images. Detect location. The detection device then identifies a statistically representative value of the position of the detection target in each of the original image and the plurality of preprocessed images as the position of the detection target on the original image. As a result, this detection device can improve the robustness of the position detection accuracy for the detection target on the image.

According to a variant, for a particular type of preprocessed image generated from a series of facial images obtained over a predetermined period of time, if the position of the feature point always satisfies the outlier criterion, the position identifying unit 33 thereafter , the preprocessing unit 31 may be prevented from generating that particular type of preprocessed image. This reduces the amount of calculation for the entire detection process.

A computer program that realizes the functions of the processor 23 of the ECU 4 according to the above embodiment or modification is provided in a form recorded in a computer-readable portable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium. may be

As described above, those skilled in the art can make various modifications within the scope of the present invention according to the embodiment.

1 vehicle control system 10 vehicle 2 driver monitor camera 3 user interface 4 electronic control unit (ECU)
21 communication interface 22 memory 23 processor 31 preprocessing unit 32 detection unit 33 position specifying unit 34 state determination unit

Claims (1)

a preprocessing unit that generates a plurality of preprocessed images by applying a plurality of different preprocesses to an image representing a detection target;
Detecting the position of the detection target in each of the image and the plurality of preprocessed images by inputting each of the image and the plurality of preprocessed images into a detector for detecting the detection target. a detection unit that
a position specifying unit that specifies a statistical representative value of the position of the detection target in each of the image and the plurality of preprocessed images as the position of the detection target on the image;
A detection device having

Images