WO2020189182A1

WO2020189182A1 - Condition monitoring device for driver

Info

Publication number: WO2020189182A1
Application number: PCT/JP2020/007141
Authority: WO
Inventors: 智也山下
Original assignee: 株式会社デンソー
Priority date: 2019-03-18
Filing date: 2020-02-21
Publication date: 2020-09-24
Also published as: JP2020155835A

Abstract

The present invention comprises: an image detection unit (4) that outputs a cropped image cut out from a formed image on the basis of a set cropping area and cropping information indicating the cropping area; and a condition monitoring unit (3) that sets the cropping area in the image detection unit such that the center position of a driver's face becomes the center position of the cropped image on the basis of the cropped image and the cropping information input from the image detection unit, and monitors a condition of the driver on the basis of the cropped image.

Description

Driver condition monitoring device

Cross-reference of related applications

This application is based on Japanese Application No. 2019-050303 filed on March 18, 2019, and the contents of the description are incorporated herein by reference.

This disclosure relates to a driver's condition monitoring device.

In recent years, it has been proposed to take a picture of the driver with a camera, monitor the driver's line-of-sight direction, etc., and call the driver's attention when it is determined that the driving is hindered.

Japanese Patent No. 6295442

Since the position of the driver's face moves according to the posture of the driver, it is necessary to set the shooting range by the camera to be relatively large so as to include the moving range of the driver's face. Therefore, the number of pixels in one frame of the imaged image output from the camera is increased, which increases the processing load of the condition monitoring device.

By the way, as shown in Patent Document 1, a video camera or the like provided with a subject tracking function is provided. This subject tracking function cuts out an image to be photographed from an image and follows it. If this subject tracking function is used in a camera, it is sufficient to process an image having a small number of pixels obtained by cutting out the driver's face from the imaged image, so that the processing of the condition monitoring device is compared with the case of processing the entire imaged image. The load can be reduced.

However, with the subject tracking function, there is a problem that it is not possible to know from the clipped image itself from which part of the imaged image the image input from the camera was acquired. In other words, in the cropped image, the subject is always taken so as to be the center of the cropped image, so the position of the driver's face cannot be known for each frame, and even if the driver's posture is out of order, it cannot be detected. Occurs. When the driver's posture is disturbed in this way, it becomes difficult to make a correct judgment when, for example, determining the driver's sideways view based on the driver's line-of-sight direction.

Therefore, we focused on some image detection ICs with a built-in image sensor that had a crop function, and used the crop function to set the crop area so that the driver's face was the center of the image. It is conceivable to crop a crop image from the image. As a result, the position of the driver's face can be specified based on the crop area set in the image detection IC, so that the driver's line-of-sight direction can be correctly recognized.

However, even if the crop area is set in the image detection IC, it is inevitable that there will be a delay between the setting of the crop area and the output of the crop image corresponding to the crop area from the image detection IC. Therefore, if a crop area is set in the image detection IC and the position of the driver's face is specified assuming that the input crop image corresponds to the crop area, if the driver's face moves in a short time, , It is assumed that the actual face position may deviate from the specified face position, and the driver's line-of-sight direction cannot be recognized correctly.

An object of the present disclosure is to provide a driver's condition monitoring device capable of correctly identifying a crop area corresponding to a crop image input from an image detection unit.

In one aspect of the present disclosure, the image detection unit outputs a crop image cut out from the imaged image based on the set crop area and crop information indicating the crop area. The condition monitoring unit sets a crop area in the image detection unit so that the center position of the driver's face becomes the center position of the crop image based on the crop image and the crop information input from the image detection unit, and the crop image. Monitor the driver's condition based on.

The above objectives and other objectives, features and advantages of the present disclosure will be clarified by the following detailed description with reference to the accompanying drawings. The drawing is
FIG. 1 is a block diagram schematically showing an overall configuration according to an embodiment. FIG. 2 is a diagram schematically showing a driver, a DSM, and an image processing ECU. FIG. 3 is a diagram showing register values. FIG. 4 is a diagram showing a crop area. FIG. 5 is a diagram showing an embedded register value in the first row of the crop area. FIG. 6 is a diagram showing a crop area. FIG. 7 is a diagram showing the movement of the crop area. FIG. 8 is a diagram showing an image resized from a crop image. FIG. 9 is a diagram showing a crop area in a state where the center position of the driver's face is greatly moved. FIG. 10 is a flowchart showing the operation of the image processing ECU.

Hereinafter, an embodiment applied to the DSM (driver status monitor) will be described with reference to the drawings.
As shown in FIG. 2, a DSM1 (corresponding to an image detection unit) is installed in a portion of the vehicle in front of the driver. The DSM 1 is connected to the image processing ECU 3 (corresponding to the condition monitoring unit) via, for example, an in-vehicle network 2 using CAN (controller area network, registered trademark) as a communication protocol, and outputs the captured image to the image processing ECU 3. ..

The image processing ECU 3 detects the driver's line-of-sight direction based on the image input from the DSM 1, and calls attention to the driver when it is determined that there is a problem in driving. An image processing device is composed of the DSM 1 and the image processing ECU 3.

The DSM1 is mainly composed of the image detection IC4 shown in FIG. The image detection IC 4 is composed of a two-dimensional image sensor 5 and a signal processing circuit 6. An object to be imaged is imaged on the image sensor 5 by an optical system (not shown), and a frame showing the imaged image is output at a frame rate of, for example, 30 fps.
As shown in FIG. 4, the pixel of the image sensor 5 is, for example, 1600 pixels × 1300 pixels, but the pixel is not limited to this.

In DSM1, since the object to be photographed is the driver's face, the driver's face is set to form an image on the image sensor 5. The signal processing circuit 6 outputs an image generated by performing predetermined signal processing on the signal input from the image sensor 5.

The signal processing circuit 6 has a plurality of registers 7 for setting its own function. By setting a value in a predetermined register 7 from the image processing ECU 3, a predetermined function of the signal processing circuit 6 is enabled / disabled, and details of each function are set. The predetermined functions include shutter speed and gain adjustment.

The image detection IC4 used in this embodiment is manufactured by Omnivision, and has a crop function in addition to shutter speed and gain adjustment as functions of the image detection IC4. The crop function is a function of cutting out a part of an image image taken by the image sensor 5 and outputting it as a crop image. Since the number of pixels of the crop image is smaller than that of the imaged image, the processing load of the image processing ECU 3 can be reduced.

As the register 7 related to the crop function, as shown in FIG. 3, in order to enable / disable the crop function, enable / disable the register value embedding function, set the start coordinates indicating the crop area, the vertical size, and the horizontal size. It is provided in.

When the crop function is enabled, the image detection IC 4 cuts out the pixels in the crop area from the imaged image captured by the image sensor 5 to create and output a crop image. When the register value embedding function is enabled, the start coordinates set in the register 7 are embedded in the crop image as crop information and output.

By the way, since the image processing ECU 3 sets the crop area for the image detection IC 4, the crop image input from the image detection IC 4 should correspond to the crop area.

However, since the internal clock of the image detection IC 4 operates independently of the image processing ECU 3, it operates asynchronously with the image processing ECU 3. Therefore, even if the crop area is set in the image detection IC 4 from the image processing ECU 3, there is a delay until the crop image corresponding to the crop area is output from the image detection IC 4.

If the crop image and the crop area do not correspond in this way, even if the position of the driver's face is specified based on the crop area, it cannot be specified correctly. Therefore, the driver's line-of-sight direction cannot be correctly recognized based on the position of the driver's face.

Therefore, we decided to use the register value embedding function as a function of the image detection IC4. That is, as described above, since the image detection IC4 manufactured by Omnivision has a function of embedding the register value in the pixel data, the embedding function is used.

The register value embedding function will be described below. When the predetermined register 7 of the image detection IC 4 is set to output the register value indicating the crop area shown in FIG. 4, the start coordinates of the crop area are embedded in the pixel data corresponding to the first row indicated by the broken line in the crop image. It is supposed to be included.

Specifically, as shown in FIG. 5, when the pixel data in the first row of the crop area is represented by, for example, Byte Orders 1 to 48, the shaded

Byte Orders

1, 2, 5, 6, 9, 10 ... Are "0x00". Is fixed, and register values are embedded in

other Byte Orderers

3, 4, 7, 8, 11, 12 ....

When the start coordinates of the crop area are set to (0x0048, 0x0082), "0x38", "0x10", and "0x00" are embedded in

Byte Orderers

3, 7, and 11. Similarly, "0x38", "0x11", and "0xA8" are embedded in

Byte Orders

15, 19, and 23, and "0x38", "0x12", and "0x00" are embedded in

Byte Orders

27, 31, and 35, and Byte. "0x38", "0x13", and "0xB2" are embedded in

Orders

39, 43, and 47.

In the above settings, "0x00" is set at register address 0x3810, "0xA8" is set at register address 3811, "0x00" is set at register address 3812, and "0xB2" is set at register address 3813. It shows that it is.
Therefore, the image processing ECU 3 can acquire the start coordinates of the crop area corresponding to the input crop image by extracting the resist value embedded in the first line of the crop image.

Next, the operation of the image processing ECU 3 will be described. When the image processing ECU 3 is activated in response to the ON of the ignition switch, the image processing ECU 3 is initially set with respect to the DSM 1 as shown in FIG. 10 (S1). In the initial setting, the register 7 for setting the crop function and the register 7 for setting the embedding of the register value in the crop image shown in FIG. 3 are set to be valid respectively. Further, the crop area is set in the register 7 for setting the crop area.

In the initial setting, the initial value is set for the crop area as shown in FIG. The initial value of the crop area is set so that the center position of the crop area is located at the center position of the pixel area.

The image detection IC4 cuts out the crop area set from the pixel area of the image sensor 5, and embeds the start coordinates of the crop area set in the register 7 in the first line of the crop image corresponding to the crop area to obtain a crop image. Output. In this case, since the driver's face is imaged on the image sensor 5, the driver's face is output as a crop image.

The image processing ECU 3 performs face detection from the crop image input from DSM1 (S2). To detect a face, for example, feature points such as eyes, nose, and mouth are extracted. At this time, if the driver's posture is out of order, the center position of the face has moved from the center position of the crop image (S3: NO), so that the center position of the face is the crop image as shown in FIG. Change the crop area setting so that it is in the center position (S4). In this case, since the size of the crop area is fixed in the present embodiment, the start coordinates set at addresses 3810 to 3813 of the register 7 built in the image detection IC 4 are changed. That is, when the center position of the face is moving as shown in FIG. 7, a value obtained by adding the movement amount to the start coordinate by the movement amount is set as the new start coordinate.

When the image processing ECU 3 changes the start coordinate of the register 7 of the image detection IC4 in this way, the image detection IC4 embeds the changed start coordinate in the crop image corresponding to the crop area of the changed start coordinate and outputs the changed start coordinate. become.
Specifically, as shown in FIG. 8, a crop image of 1280 × 960 size is cropped from a pixel size of 1600 × 1300 size.

As shown in FIG. 9, when the center position of the driver's face is largely moved from the center position of the imaged image and the center position of the driver's face cannot be positioned at the center position of the crop image. , Set the crop area so that the corner of the crop image is located at the corner of the imaged image.

When the crop image is set as described above, it is determined whether the brightness of the face area is sufficient (S5), and if it is not sufficient (S5: NO), the shutter speed, gain, etc. are determined by the image detection IC4. Adjust by writing to register 7 (S6). When the brightness of the face area becomes sufficient (S5: YES), the driver's line-of-sight direction is recognized based on the crop image (S7). At this time, since the actual eye position of the driver can be specified from the crop image and the start coordinates of the crop area, the actual line-of-sight direction of the driver can be correctly recognized.

By the way, although the number of pixels of the crop image is smaller than the number of pixels of the imaged image, the pitch of the pixels is the same, and it is possible to determine the details of the driver's face. A crop image was used to recognize the direction of the line of sight.
However, for example, in the case of face recognition based on a crop image to identify the driver, the number of pixels is large, so that the processing load of the image processing ECU 3 is large.

Therefore, the image processing ECU 3 resizes by thinning out the pixel data of the crop image (S8). This resized size is the minimum required size and is VGA (640 × 480) size as shown in FIG.

Then, the image processing ECU 3 identifies the driver by face recognition based on the resized crop image. In this face recognition, it is sufficient if the driver's face can be identified, so that the face can be recognized even if the size is VGA. By identifying the driver in this way, the steering wheel, seat, rearview mirror, etc. stored in advance for each driver can be automatically adjusted.
By performing the above operation for each frame, the image processing ECU 3 can correctly detect the center position of the driver's face and recognize the face in all the frames without dropping the frame.

As described above, paying attention to the crop function of the image detection IC4, the crop area is dynamically changed so as to follow the driver's face position, and only the face area is cut out to process the image data of the minimum size. Since the data is transferred to the ECU 3, the processing load of the image processing ECU 3 can be significantly reduced.

In addition, as the driver state recognition technology using a high-pixel camera expands in the future, it is expected that the load of image processing will increase as the number of pixels increases, but it was cut out from the imaged image. Since the crop image is processed, a high-speed CPU and a large-capacity memory are not required, which is advantageous in terms of product size, cost, and heat generation.

According to such an embodiment, the following effects can be obtained.
The image processing ECU 3 provides a crop area to the DSM 1 so that the center position of the driver's face becomes the center position of the crop image based on the crop image input from the DSM 1 and the start coordinates of the crop area embedded in the crop image. Since the setting and the line-of-sight direction of the driver are recognized based on the crop image, the line-of-sight direction of the driver can be correctly recognized even when the driver's posture is lost.
Since the driver's face is recognized based on the resized crop image by thinning out the pixel data, the processing load of the image processing IC3 can be further reduced.

(Other embodiments)
The image detection sensor may be configured to independently output crop information without embedding it in the crop image.
As crop information, in addition to the starting coordinates of the crop area, the vertical size and the horizontal size may be output.
The driver's face recognition may be performed without resizing the crop image.
The state of the driver is not limited to the direction of the driver's line of sight, and for example, the degree of drowsiness of the driver may be determined.

Although the present disclosure has been described in accordance with an embodiment, it is understood that the present disclosure is not limited to the embodiment or structure. The present disclosure also includes various modifications and modifications within an equal range. In addition, various combinations and forms, as well as other combinations and forms that include only one element, more, or less, are also within the scope of the present disclosure.

Claims

An image detection unit (4) that outputs a crop image cut out from an image image based on a set crop area and crop information indicating the crop area, and
The crop area is set in the image detection unit so that the center position of the driver's face becomes the center position of the crop image based on the crop image and the crop information input from the image detection unit, and the crop area is set. A condition monitoring unit (3) that monitors the driver's condition based on a crop image, and
Driver condition monitoring device equipped with.
The driver's condition monitoring device according to claim 1, wherein the crop information includes at least the start coordinates of the crop area.
The driver's condition monitoring device according to claim 1 or 2, wherein the image detection unit embeds the crop information in the crop image.
The driver's condition monitoring device according to any one of claims 1 to 3, wherein the image detection unit recognizes the driver's face based on the crop image.
The driver's condition monitoring device according to claim 4, wherein the condition monitoring unit recognizes the driver's face based on the crop image resized by thinning out the pixel data.