JP2023083837A - Imaging apparatus for crew member monitoring system and the crew member monitoring system - Google Patents

Abstract

To provide an imaging apparatus for a crew member monitoring system, capable of imaging data for creating a clear detection image even in the case where a crew member moves in a photographing and a camera swings, and provide a member monitoring system having the imaging apparatus.SOLUTION: The present invention provides an imaging apparatus 1 for a crew member monitoring system, comprising: a first light source 11 that irradiates a light of a first wavelength to a crew member 33 in a vehicle; a second light source 12 that irradiates the light of a second wavelength different from the first wavelength from a position different from that of the first light source 11 to the crew member 33; and a spectroscopic camera 13 that images the crew member 33 to which the light of the first wavelength and the light of the second wavelength, and acquires a data cube 20 which can fetch out a first image data 21 having information on the wavelength region containing the first wavelength and a second image data 22 having information on the wavelength region containing a second wavelength.SELECTED DRAWING: Figure 3

Description

The present invention relates to an imaging device for an occupant monitoring system and an occupant monitoring system used in a vehicle cabin.

Conventionally, an image capturing device that detects the direction of the driver's face and line of sight from an image captured by the driver driving a vehicle, and removes the reflection of light on the glasses worn by the driver from the image. There is known an image capturing device capable of capturing images (see Patent Literature 1).

When light is reflected in the lenses and frames of eyeglasses, resulting in high-intensity areas, image information of the driver's eyes cannot be obtained, and the accuracy of detecting the direction of the driver's face and line of sight is reduced. There is a risk. The image capturing device described in Patent Document 1 includes two light illuminators arranged at different positions for illuminating the driver, and the driver is photographed when light is illuminated from one of the light illuminators. It is possible to acquire an image of the driver and an image of the driver captured when light is illuminated from the other light irradiator. Then, these images are combined to form an image from which reflection of light on the eyeglasses worn by the driver is removed, thereby improving the detection accuracy of the driver's face direction and line of sight.

JP 2010-124043 A

However, according to the image capturing device described in Patent Document 1, the image of the driver captured when light is illuminated from one light irradiator and the image of the driver captured when light is illuminated from the other light irradiator Since it is necessary to capture the image of the driver at a different timing than the image of the driver, if the driver moves or shakes the camera while these two images are captured, the image of the driver will be captured between the two images. There will be a difference in the image of Therefore, in the image obtained by synthesizing these two images from which the reflection of light on the glasses is removed, the image of the driver becomes unclear, and the direction of the face, line of sight, etc. for judging the condition of the driver becomes unclear. information may not be detected accurately.

An object of the present invention is to provide an imaging device for an occupant monitoring system that acquires image data used to create a detection image for detecting information such as the direction of the face of an occupant in a vehicle, the line of sight, and the degree of eye opening, An imaging device for an occupant monitoring system that can acquire image data for creating a clear image for detection even when the occupant moves or the camera shakes during imaging, and an occupant monitoring system equipped with the imaging device. to provide.

In order to achieve the above object, one aspect of the present invention provides the following [1] and [2] photographing devices for an occupant monitoring system, and [3] an occupant monitoring system.

[1] A first light source that irradiates an occupant in a vehicle with light of a first wavelength, and a light of a second wavelength that is different from the first wavelength to the occupant from a position different from the first light source. and the occupant irradiated with the light of the first wavelength and the light of the second wavelength, and the first light having information of a wavelength range including the first wavelength An imaging device for an occupant monitoring system, comprising: a spectroscopic camera that acquires a data cube capable of extracting image data and second image data having information of a wavelength region including the second wavelength as different data.
[2] The photographing device for an occupant monitoring system according to [1] above, wherein the vehicle is an automobile.
[3] Using the occupant monitoring system imaging device according to [1] or [2] above, and the first image data and the second image data, the light of the first wavelength and the second and an image processing unit that creates a detection image used for detecting information for determining the condition of the occupant, in which reflection of light with a wavelength of is suppressed.

According to the present invention, an imaging device for an occupant monitoring system that acquires image data used for creating a detection image for detecting information such as the direction of the face of an occupant in a vehicle, the line of sight, and the degree of eye opening, An imaging device for an occupant monitoring system that can acquire image data for creating a clear image for detection even when the occupant moves or the camera shakes during imaging, and an occupant monitoring system equipped with the imaging device. can provide.

FIG. 1 is a schematic diagram of the interior of an automobile for showing the installation locations of a first light source, a second light source, and a spectroscopic camera that constitute an imaging device according to the present embodiment. FIG. 2 is a schematic diagram showing how an imaging device captures an image around the face of an occupant sitting in the driver's seat. FIG. 3 is a block diagram schematically showing an example of a configuration of an occupant monitoring system including an imaging device mounted on a vehicle. FIG. 4 is a flow chart schematically showing the flow of occupant monitoring by the occupant monitoring system. FIG. 5 is a schematic diagram of the structure of the data cube acquired by the spectroscopic camera. FIGS. 6A and 6B are schematic diagrams of the first image data and the second image data, respectively. FIG. 7 is a schematic diagram showing how passengers in the bus are monitored.

A photographing device 1 for a passenger monitoring system according to the embodiment of the present invention is a photographing device for a passenger monitoring system that acquires image data used for creating a detection image for detecting the state of a passenger in a vehicle cabin. .

The photographing device 1 includes a first light source 11 for irradiating light of a first wavelength to an occupant in a vehicle, and a second wavelength different from the first wavelength to the occupant from a position different from the first light source 11. and a first image having information of a wavelength range including the first wavelength by photographing an occupant irradiated with the light of the first wavelength and the light of the second wavelength and a spectroscopic camera 13 that acquires a data cube that can extract second image data having information of a wavelength range including the data and the second wavelength as different data.

FIG. 1 is a schematic diagram of the vehicle interior of an automobile for showing the installation locations of the first light source 11, the second light source 12, and the spectroscopic camera 13 that constitute the photographing device 1. As shown in FIG. P ₀ shown in FIG. 1 indicates an example of the installation location of the spectroscopic camera 13 , and P ₁ and P ₂ indicate examples of the installation locations of the first light source 11 and the second light source 12 .

As shown in FIG. 1, the spectroscopic camera 13 is installed, for example, on the pad portion (P ₀ ) of the steering wheel 31 . The first light source 11 and the second light source 12 are installed, for example, on the rim portion (P ₁ ) of the steering wheel 31 or the A pillar (front pillar) 32 (P ₂ ). For example, the spectroscopic camera 13 is placed within 1 m from the face of the passenger to be photographed, and the first light source 11 and the second light source 12 are placed within 2 m from the face of the passenger.

FIG. 2 is a schematic diagram showing how the photographing device 1 photographs the face and the periphery of the passenger 33 sitting in the driver's seat 30. As shown in FIG. In the example shown in FIG. 2, the first light source 11 and the second light source 12 are installed on the rim portion of the steering wheel 31, and the spectroscopic camera 13 is installed on the pad portion of the steering wheel 31. FIG.

The light of the first wavelength emitted from the first light source 11 and the light of the second wavelength emitted from the second light source 12 illuminate the face of an occupant 33 sitting in the driver's seat 30, thereby illuminating the area around the face. The spectroscopic camera 13 photographs the periphery of the face of the occupant 33 in this state, and acquires the data cube.

The spectroscopic camera 13 is a camera, such as a multispectral camera or a hyperspectral camera, capable of obtaining a spectral spectrum (luminance vs. wavelength) for each pixel. The image data acquired by the spectroscopic camera 13 by photographing is three-dimensional (or four-dimensional or more) data including wavelength information in addition to two-dimensional planar data, and is called a data cube.

The 1st light source 11 and the 2nd light source 12 are lighting apparatuses provided with light emitting elements, such as LED, for example. The first wavelength, which is the emission wavelength of the first light source 11 , and the second wavelength, which is the emission wavelength of the second light source 12 , are determined by the spectral performance of the spectroscopic camera 13 . That is, from the data cube acquired by the spectroscopic camera 13, the first image data having the information of the wavelength region including the first wavelength and the second image data having the information of the wavelength region including the second wavelength are different data. It suffices if the first wavelength and the second wavelength are separated to the extent that they can be extracted as .

Typically, the first light source 11 and the second light source 12 each emit light in a single wavelength region in the near-infrared region. That is, each of the first wavelength and the second wavelength is included in the near-infrared region. In order to reliably acquire the first image data and the second image data as different data, for example, it is preferable that the first wavelength and the second wavelength (the peak wavelengths thereof) are separated by 50 nm or more.

FIG. 3 is a block diagram schematically showing an example of the configuration of an occupant monitoring system 100 including the photographing device 1 mounted on a vehicle. The occupant monitoring system 100 is a system that monitors the area around the face of the occupant 33 in the vehicle. For example, by issuing a warning according to the state of the driver determined by monitoring, it is possible to suppress inattentive driving and drowsy driving. can be done.

The occupant monitoring system 100 includes an imaging device 1 having a first light source 11, a second light source 12, and a spectroscopic camera 13; Using the first image data and the second image included in the obtained data cube, the state of the occupant 33 in which reflection of the light of the first wavelength and the light of the second wavelength is suppressed is determined. An image processing unit 15 that creates a detection image used for detecting information (referred to as judgment information) such as face direction, line of sight, eye opening degree, etc. for the purpose, and stores the detection image created by the image processing unit 15 a storage unit 16; an image analysis unit 17 that analyzes the detection image stored in the storage unit 16 and detects determination information; a warning unit 19 that issues a warning to the occupant 33 according to the state of the occupant 33 determined by the state determination unit 18; the first light source 11; the second light source 12; the spectroscopic camera 13; It includes a control unit 10 that controls operations of a processing unit 15, an image analysis unit 17, a state determination unit 18, a warning unit 19, and the like.

The control unit 10, the storage unit 14, the image processing unit 15, the storage unit 16, the image analysis unit 17, and the state determination unit 18 that constitute the occupant monitoring system 100 are, for example, an ECU (electronic control unit) mounted on the vehicle. It is included in the implemented MCU (microcontroller unit). Also, the warning unit 19 is, for example, an in-vehicle device that issues a warning by sound or light. Note that the occupant monitoring system 100 may not include the warning unit 19 in its configuration. In this case, the warning unit 19 is connected to the occupant monitoring system 100 as an external device.

FIG. 4 is a flowchart schematically showing the flow of monitoring of the occupant 33 by the occupant monitoring system 100. As shown in FIG. First, light of a first wavelength from the first light source 11 and light of a second wavelength from the second light source 12 are simultaneously irradiated around the face of the occupant 33, and the first light and the second light are emitted. The area around the face of the occupant 33 in the irradiated state is photographed by the spectroscopic camera 13 (step S1). The data cube acquired by the spectroscopic camera 13 by photographing is stored in the storage unit 14 .

FIG. 5 is a schematic diagram of the structure of the data cube acquired by the spectroscopic camera 13. As shown in FIG. FIG. 5 shows how the datacube 20 contains a plurality of images each containing information in different wavelength ranges. First image data 21 having information on a wavelength range including the emission wavelength of the first light source 11 and information on the wavelength range including the emission wavelength of the second light source 12 in a plurality of images constituting these data cubes 20 A second image data 22 having information is included.

For example, if the wavelength range of the first image data 21 is λ ₁ to λ ₂ , the brightness of wavelengths outside the range of λ ₁ to λ ₂ is zero in each pixel included in the first image data 21. . Since the second wavelength, which is the emission wavelength of the second light source 12, is outside the range of λ ₁ to λ ₂ , the light emitted from the second light source 12 is not reflected in the first image data 21. . Similarly, since the first wavelength, which is the emission wavelength of the first light source 11, is outside the wavelength range of the second image data 22, the light emitted from the first light source 11 is reflected in the second image data. 22 is not reflected.

6A and 6B are schematic diagrams of the first image data 21 and the second image data 22, respectively. As shown in FIGS. 6A and 6B, light 351 having the first wavelength emitted from the first light source 11 and reflected by the glasses of the occupant 33 is reflected in the first image data 21 . In addition, light 352 having the second wavelength emitted from the second light source 12 and reflected by the glasses of the occupant 33 is reflected in the second image data 22 . Here, since the first light source 11 and the second light source 12 are installed at different positions, the position where the light 351 of the first image data 21 is reflected and the light 352 of the second image data 22 are reflected. The insertion position (coordinates on the image) is different.

Next, the image processing unit 15 extracts the first image data and the second image data from the data cube stored in the storage unit 14, and uses the first image data and the second image data for detection. An image is created (step S2). The detection image created by the image processing section 15 is stored in the storage section 16 .

Note that the first image data and the second image data extracted from the data cube may be stored in the storage unit 14 after the spectroscopic camera 13 acquires the data cube. In this case, the first image data and the second image data stored in the storage unit 14 are used in step S2.

Creation of the detection image in step S2 is performed by image synthesis (additive synthesis, subtractive synthesis, etc.) using the first image data 21 and the second image data 22 . Specifically, for example, the difference is extracted by subtractive synthesis of the first image data 21 and the second image data 22, and the light 351 and the light 352 are extracted (the luminance of the light 351 and the light 352 is selectively Information of the first image data 21 and the information of the second image data 22 are added (light 351 and light 352) is created, and the difference image data is subtracted and combined with the added image data to obtain a detection image in which the light 351 and the light 352 are removed or almost removed. The method of synthesizing the first image data 21 and the second image data 22 for creating the detection image is not particularly limited, and various known methods can be used.

The first image data 21 and the second image data 22 used for creating the detection image in step S2 are obtained simultaneously by the spectroscopic camera 13. Even if the spectroscopic camera 13 is shaken, the images of the first image data 21 and the second image data 22 do not shift due to the time lag. Therefore, even if the occupant 33 is moving or the spectroscopic camera 13 is shaking at the time of photographing, a clear detection image can be created.

Next, the image analysis unit 17 analyzes the detection image stored in the storage unit 16 to detect determination information such as the face orientation, line of sight, and degree of eye opening of the occupant 33 (step S3). At this time, since the light 351 and the light 352 have been removed or almost removed from the detection image in step S2, the information of the portion where the light 351 and the light 352 were reflected (such as the eye condition of the passenger 33) can be obtained with high accuracy. can be detected. Further, even if the occupant 33 is moving or the spectroscopic camera 13 is shaking at the time of photographing, the determination information can be accurately detected from the clear detection image.

Next, the state determination unit 18 determines the state of the occupant 33 based on the determination information detected by the image analysis unit 17 (step S4). For example, it is determined that the occupant 33 is looking aside when the direction of the face or line of sight of the occupant 33 is not facing the front, and it is determined that the occupant 33 is drowsy when the eye opening degree is low.

Next, the warning unit 19 issues a warning to the occupant 33 according to the state of the occupant 33 determined by the state determination unit 18 (step S5). For example, if the state determination unit 18 determines that the driver is inattentive, the driver is warned by voice to avoid driving without looking aside, and if it is determined that the driver is drowsy, the driver is notified by sound or light. Issue a wake-up alert.

In the above description, the occupant monitoring system 100 equipped with the imaging device 1 is used to monitor the driver in the vehicle interior of the vehicle. It may also be a passenger of a vehicle other than an automobile, such as a train or an airplane. When monitoring a passenger other than the driver, it is possible to determine the condition of the passenger, such as an abnormality in physical condition, for example.

FIG. 7 is a schematic diagram showing how passengers in the bus are monitored (illustration of the first light source 11 and the second light source 12 is omitted). In the example shown in FIG. 7, a plurality of passengers 33a to 33e sitting on seats 42 are monitored by the spectroscopic camera 13 of the photographing device 1 installed on the ceiling of the bus.

In such a case, for example, it is possible to determine the condition of the occupant, such as abnormal physical condition or abnormal behavior. In addition, since the spectroscopic camera 13 is used, it is possible to reduce the influence of external light and lighting inside the vehicle and improve the state detection accuracy in the morning and at night. Furthermore, by using near-infrared light sources as the first light source 11 and the second light source 12, it is possible to reduce the reflection of the passengers 33a to 33e on the window glass 41 and suppress erroneous detection.

(Effect of Embodiment)
According to the above-described embodiment of the present invention, by simultaneously acquiring the first image data 21 and the second image data 22 by the spectroscopic camera 13, it is possible to detect when the occupant 33 is moving at the time of photographing or when the spectroscopic camera 13 is A clear image for detection can be created even in the case of shaking. As a result, it is possible to accurately detect the determination information such as the direction of the face of the occupant 33, the line of sight, and the degree of eye opening, and to determine the state of the occupant 33 with high accuracy. For this reason, for example, it is possible to suppress inattentiveness and dozing off of the driver, and to make stable state judgments when transferring driving authority at autonomous driving levels 3 to 4 (for example, whether the driver is in a state where driving is possible when authority is transferred to the driver). judgment).

Although the embodiments of the present invention have been described above, the present invention is not limited to the above embodiments, and various modifications can be made without departing from the scope of the invention. Also, the constituent elements of the above-described embodiments can be arbitrarily combined without departing from the gist of the invention.

Moreover, the above embodiments do not limit the invention according to the scope of claims. Also, it should be noted that not all combinations of features described in the embodiments are essential to the means for solving the problems of the invention.

Reference Signs List 1 imaging device 10 control unit 11 first light source 12 second light source 13 spectroscopic camera 14, 16 storage unit 15 image processing unit 17 image analysis unit 18 state determination unit 19 warning unit 100 occupant monitoring system

Claims (3)

a first light source for illuminating an occupant in the vehicle with a first wavelength of light;
a second light source that irradiates the occupant with light of a second wavelength different from the first wavelength from a position different from that of the first light source;
photographing the occupant irradiated with the light of the first wavelength and the light of the second wavelength; a spectroscopic camera that acquires a data cube that can extract the second image data having information of the wavelength range including as different data;
with
Imager for occupant monitoring system. wherein the vehicle is an automobile;
The imaging device for an occupant monitoring system according to claim 1. An imaging device for an occupant monitoring system according to claim 1 or 2;
Information for determining the state of the occupant in which reflection of the light of the first wavelength and the light of the second wavelength is suppressed using the first image data and the second image data An image processing unit that creates a detection image used for detection of
with
Occupant monitoring system.

Images