WO2021106073A1 - Occupant monitoring device, occupant monitoring system, and occupant monitoring method - Google Patents

Occupant monitoring device, occupant monitoring system, and occupant monitoring method Download PDF

Info

Publication number
WO2021106073A1
WO2021106073A1 PCT/JP2019/046160 JP2019046160W WO2021106073A1 WO 2021106073 A1 WO2021106073 A1 WO 2021106073A1 JP 2019046160 W JP2019046160 W JP 2019046160W WO 2021106073 A1 WO2021106073 A1 WO 2021106073A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
equipment
unit
occupant monitoring
occupant
Prior art date
Application number
PCT/JP2019/046160
Other languages
French (fr)
Japanese (ja)
Inventor
啓人 正司
Original Assignee
三菱電機株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 三菱電機株式会社 filed Critical 三菱電機株式会社
Priority to JP2021560799A priority Critical patent/JP7204941B2/en
Priority to PCT/JP2019/046160 priority patent/WO2021106073A1/en
Publication of WO2021106073A1 publication Critical patent/WO2021106073A1/en

Links

Images

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R1/00—Optical viewing arrangements; Real-time viewing arrangements for drivers or passengers using optical image capturing systems, e.g. cameras or video systems specially adapted for use in or on vehicles
    • B60R1/12—Mirror assemblies combined with other articles, e.g. clocks
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
    • B60R11/00—Arrangements for holding or mounting articles, not otherwise provided for
    • B60R11/04—Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Definitions

  • the present invention relates to an occupant monitoring device, an occupant monitoring system, and an occupant monitoring method.
  • Patent Document 1 describes an occupant monitoring device that images an occupant in the passenger seat by a camera housed in a rear-view mirror case and monitors the occupant using the captured image.
  • This occupant monitoring device captures an image of the front pillar imaged when the rearview mirror is adjusted to an appropriate posture so that the occupant in the driver's seat can easily see the rear view, by a camera built in the rearview mirror. Detect from. Then, when the front pillar is detected at a predetermined position in the image, the occupant monitoring device determines that the occupant in the passenger seat can be imaged.
  • the occupant monitoring device described in Patent Document 1 may erroneously detect the occupant shown in the image as the occupant to be monitored even if the occupant shown in the image is not the occupant to be monitored. was there.
  • the present invention has been made to solve the above-mentioned problems, and detects an occupant to be monitored from an image captured by a camera built in an in-vehicle device such as a rear-view mirror whose posture can be adjusted.
  • a camera built in an in-vehicle device such as a rear-view mirror whose posture can be adjusted.
  • the purpose is to solve the above-mentioned problems, and detects an occupant to be monitored from an image captured by a camera built in an in-vehicle device such as a rear-view mirror whose posture can be adjusted.
  • the occupant monitoring device has an image acquisition unit that acquires an image of the vehicle interior captured by a camera built in an in-vehicle device whose attitude can be adjusted, and adjusts the attitude of the in-vehicle device so that it can be used for the original purpose.
  • the equipment in the vehicle interior that is imaged by the camera is set as the first equipment, and the equipment detection area is set at a predetermined position of the image acquired by the image acquisition unit, and the first equipment is detected from the equipment detection area.
  • the monitoring area setting unit that sets the occupant monitoring area for the image and the image acquisition unit based on the image position of the first equipment. It is provided with an occupant monitoring unit that monitors occupants using the image of the occupant monitoring area set by the monitoring area setting unit among the acquired images.
  • the first equipment imaged by the camera is detected from the image, and the image is obtained based on the image position of the first equipment.
  • the occupant monitoring area is set, the occupant to be monitored can be detected from the image captured by the camera built in the in-vehicle device whose posture can be adjusted.
  • FIG. 6A shows an example of an image in which the equipment detection unit cannot detect the rear window
  • FIG. 6B shows an example of an image in which a part of the occupant monitoring area set by the monitoring area setting unit protrudes. It is a flowchart which shows the operation example of the occupant monitoring by the occupant monitoring system which concerns on Embodiment 1.
  • FIG. 8A and 8B are diagrams showing a hardware configuration example of the occupant monitoring device according to the first embodiment.
  • FIG. 1 is a perspective view showing a configuration example of the rearview mirror 1 according to the first embodiment.
  • FIG. 2 is a perspective view showing an example of a vehicle 10 in which the rearview mirror 1 of FIG. 1 is installed.
  • the left-right direction of the vehicle 10 is the X-axis
  • the front-rear direction is the Y-axis
  • the up-down direction is the Z-axis.
  • the rearview mirror 1 is an in-vehicle device installed on the front window (not shown) of the vehicle 10 or the ceiling in the vicinity thereof, and is used by the driver 11 to visually recognize the rear view.
  • the driver 11 confirms the rear of the vehicle 10 by looking at the rear window 12 through the rear-view mirror 1.
  • the room mirror 1 includes a mirror case 2, an arm 3, an infrared filter 4, a half mirror 5, infrared light emitting units 6a and 6b, an infrared camera 7, and a partition member 8.
  • An arm 3 is provided on the back surface of the mirror case 2.
  • the arm 3 fixes the mirror case 2 to the front window (not shown) or the ceiling in the vicinity thereof. Further, the arm 3 changes the posture of the rearview mirror 1 in the X-axis direction and the Z-axis direction.
  • the driver 11 may grasp the rear-view mirror 1 by hand and adjust the posture of the rear-view mirror 1. Alternatively, when the actuator 3 is built in the arm 3, the driver 11 may operate the actuator to adjust the posture of the rearview mirror 1.
  • a half mirror 5 is mounted on the front of the mirror case 2.
  • An infrared filter 4 is arranged on the back surface of the half mirror 5 and inside the mirror case 2.
  • Infrared light emitting units 6a and 6b and an infrared camera 7 are arranged on the back surface of the infrared filter 4 and inside the mirror case 2.
  • the infrared light emitting unit 6a and the infrared light emitting unit 6b are arranged on both sides of one infrared camera 7, but only one of the infrared light emitting unit 6a and the infrared light emitting unit 6b is arranged. It may be a configured configuration.
  • the infrared emitting units 6a and 6b irradiate the inside of the vehicle 10 with infrared rays.
  • the infrared camera 7 receives the infrared rays reflected in the vehicle 10 and transmitted through the half mirror 5 and the infrared filter 4 among the infrared rays emitted by the infrared emitting units 6a and 6b, and outputs them as an image.
  • the infrared light emitting unit 6a, the infrared light emitting unit 6b, and the infrared camera 7 are partitioned by a partition member 8. Therefore, the infrared rays emitted by the infrared light emitting units 6a and 6b are reflected by the half mirror 5 and the infrared filter 4 and do not enter the infrared camera 7.
  • Infrared light emitting parts 6a and 6b, an infrared camera 7, and a partition member 8 are arranged on the back surface side of the infrared filter 4. Therefore, the infrared light emitting units 6a and 6b, the infrared camera 7, and the partition member 8 cannot be seen from the outside of the rearview mirror 1. Therefore, the infrared camera 7 for occupant monitoring can be arranged in the vehicle 10 without making the driver 11 and the passenger, who are the subjects, aware of the existence of the infrared camera 7 and the infrared light emitting units 6a and 6b.
  • FIG. 3 is a diagram showing an example of an image captured by the infrared camera 7 built in the rearview mirror 1 according to the first embodiment.
  • the image shown in FIG. 3 is an image captured by the infrared camera 7 when the rearview mirror 1 is adjusted to an appropriate posture so that the driver 11 can easily see the rear.
  • This image shows the driver 11, the passenger seat, the rear window 12, the left and right rear side windows 14, the left and right rear pillars 16, the roof lining 17, and the like in the driver's seat.
  • the imaging range of the infrared camera 7 is the rear window 12 and the driver's seat on which the driver 11 who is the monitoring target of the occupant monitoring device 20 is boarded when the posture of the rearview mirror 1 is adjusted for the original purpose. It is assumed that it is set in the range that includes.
  • FIG. 4 is a block diagram showing a configuration example of the occupant monitoring system 30 according to the first embodiment.
  • the occupant monitoring system 30 includes a rearview mirror 1 and an occupant monitoring device 20.
  • the occupant monitoring device 20 is connected to an infrared camera 7 built in the rearview mirror 1 and a warning device 26 provided in the vehicle 10.
  • the warning device 26 has a speaker, a display, or the like, and warns the driver 11 by voice, display, or the like.
  • the occupant monitoring device 20 includes an image acquisition unit 21, an equipment detection unit 22, a monitoring area setting unit 23, an occupant monitoring unit 24, and a warning unit 25.
  • the image acquisition unit 21 acquires an image of the inside of the vehicle 10 captured by the infrared camera 7 built in the posture-adjustable rearview mirror 1 from the infrared camera 7.
  • the image acquisition unit 21 outputs the acquired image to the equipment detection unit 22 and the occupant monitoring unit 24.
  • the equipment detection unit 22 receives an image from the image acquisition unit 21 and sets the equipment detection area 13 at a predetermined position of this image.
  • the first equipment in the vehicle interior which is imaged by the infrared camera 7, is the rear window 12.
  • the equipment detection area 13 is an area for detecting the rear window 12, which is the first equipment, and is preset at a fixed position in the image.
  • the equipment detection unit 22 sets the equipment detection area 13 in the image captured by the infrared camera 7, and detects the rear window 12 from the equipment detection area 13.
  • the equipment detection unit 22 When the rear window 12 can be detected from the equipment detection area 13, the driver 11 can see the rear view through the rearview mirror 1. If the rear window 12 cannot be detected from the equipment detection area 13, it is highly possible that the driver 11 cannot see the rear view through the rearview mirror 1.
  • the equipment detection unit 22 When the equipment detection unit 22 can detect the rear window 12 from the equipment detection area 13, the equipment detection unit 22 outputs the image and the image position of the rear window 12 to the monitoring area setting unit 23. On the other hand, when the image and the rear window 12 cannot be detected from the equipment detection area 13, the equipment detection unit 22 notifies the warning unit 25 that the detection has not been performed.
  • the monitoring area setting unit 23 receives the image and the image position of the rear window 12 from the equipment detection unit 22.
  • the monitoring area setting unit 23 sets the occupant monitoring area 15 for the image based on the image position of the rear window 12 notified from the equipment detection unit 22.
  • the monitoring area setting unit 23 holds information regarding the position of the occupant monitoring area 15 with reference to the position of the rear window 12 in advance.
  • the positional relationship between the rear window 12 and the occupant monitoring area 15 when the occupant to be monitored is the driver 11 is illustrated in FIG.
  • the monitoring area setting unit 23 notifies the occupant monitoring unit 24 of the image position of the set occupant monitoring area 15. When a part or all (hereinafter, referred to as "at least a part") of the set occupant monitoring area 15 protrudes from the image, the monitoring area setting unit 23 notifies the warning unit 25 to that effect.
  • the occupant monitoring unit 24 receives an image from the image acquisition unit 21 and sets the occupant monitoring area 15 notified from the monitoring area setting unit 23 in this image.
  • the occupant monitoring unit 24 monitors the driver 11 by using the image of the occupant monitoring area 15 among the images received from the image acquisition unit 21. For example, the occupant monitoring unit 24 determines a state such as drowsiness, inattentiveness, or fatigue of the driver 11, and if the driver 11 determines that the state is abnormal, the occupant monitoring unit 24 notifies the warning unit 25 to that effect.
  • the warning unit 25 When the warning unit 25 receives a notification from the equipment detection unit 22, the monitoring area setting unit 23, or the occupant monitoring unit 24, the warning unit 25 instructs the warning device 26 to issue a warning.
  • the warning device 26 issues a warning when it receives an instruction from the warning unit 25.
  • FIG. 5 is a flowchart showing an operation example of the rearview mirror initial posture adjustment by the occupant monitoring system 30 according to the first embodiment.
  • the occupant monitoring system 30 performs the operation shown in the flowchart of FIG. 5 every time before the vehicle 10 departs.
  • step ST1 the image acquisition unit 21 acquires an image captured by the infrared camera 7 from the infrared camera 7 in a state where the infrared light emitting units 6a and 6b are irradiated with infrared rays.
  • step ST2 the equipment detection unit 22 sets the equipment detection area 13 at a predetermined position of the image acquired by the image acquisition unit 21, and detects the rear window 12 from the equipment detection area 13.
  • the monitoring area setting unit 23 uses the image position of the rear window 12 detected by the equipment detection unit 22 as a reference. , The occupant monitoring area 15 is set for the image acquired in step ST1 (step ST3).
  • step ST5 the equipment detection unit 22 instructs the warning unit 25 to issue a warning prompting the driver 11 to adjust the posture of the rearview mirror 1.
  • the warning unit 25 issues a warning from the warning device 26 in accordance with this instruction.
  • FIG. 6A shows an example of an image in which the equipment detection unit 22 cannot detect the rear window 12 from the equipment detection area 13. Since the rearview mirror 1 is facing downward, the rear window 12 is not shown in this image.
  • the equipment detection unit 22 assumes that the rear window 12 can be detected from the equipment detection area 13 when the rear window 12 is contained in the equipment detection area 13, and at least a part of the rear window 12 is from the equipment detection area 13. If it protrudes, it is assumed that the rear window 12 cannot be detected from the equipment detection area 13.
  • step ST4 the monitoring area setting unit 23 determines whether or not the set occupant monitoring area 15 fits in the image. When the occupant monitoring area 15 fits within the image (step ST4 “YES”), the monitoring area setting unit 23 notifies the occupant monitoring unit 24 of the image position of the occupant monitoring area 15.
  • step ST4 “NO”) When at least a part of the set occupant monitoring area 15 protrudes from the image (step ST4 “NO”), the monitoring area setting unit 23 is likely to be in a state where the occupant monitoring unit 24 cannot monitor the driver 11. Therefore, in step ST6, the monitoring area setting unit 23 instructs the warning unit 25 to issue a warning prompting the driver 11 to adjust the posture of the rearview mirror 1. The warning unit 25 issues a warning from the warning device 26 in accordance with this instruction.
  • FIG. 6B shows an example of an image in which a part of the occupant monitoring area 15 set by the monitoring area setting unit 23 protrudes.
  • Half of the occupant monitoring area 15 set with reference to the image position of the rear window 12 is in a state of protruding from the image.
  • the driver 11 cannot see the rear view through the rearview mirror 1.
  • the occupant monitoring device 20 After the warning device 26 issues a warning in step ST5 or step ST6, the occupant monitoring device 20 performs the operation shown in the flowchart of FIG. 5, assuming that the driver 11 has adjusted the posture of the rearview mirror 1. If the warning device 26 issues a warning in step ST5 or step ST6, there is a high possibility that the face of the driver 11 will not fit within the occupant monitoring area 15. Therefore, the driver 11 by the occupant monitoring unit 24 It becomes difficult to monitor.
  • the driver 11 Since the occupant monitoring device 20 urges the driver 11 to adjust the posture of the rear-view mirror 1 before the vehicle 10 departs, the driver 11 forgets to adjust the posture of the rear-view mirror 1 and departs to the room while driving. It is possible to suppress adjusting the posture of the mirror 1. Therefore, the driver 11 can be concentrated on driving. Further, for example, when the vehicle 10 is shared by a plurality of people, the driver after the change can adjust the rearview mirror 1 to a posture suitable for himself / herself before the vehicle departs.
  • FIG. 7 is a flowchart showing an operation example of occupant monitoring by the occupant monitoring system 30 according to the first embodiment.
  • the occupant monitoring system 30 starts the operation shown in the flowchart of FIG. 7 after performing the operation of adjusting the initial position of the rearview mirror shown in FIG.
  • the occupant monitoring unit 24 Prior to the start of the operation shown in the flowchart of FIG. 7, the occupant monitoring unit 24 has already received the image position of the occupant monitoring area 15 from the monitoring area setting unit 23.
  • step ST11 the image acquisition unit 21 acquires an image captured by the infrared camera 7 from the infrared camera 7 in a state where the infrared light emitting units 6a and 6b are irradiated with infrared rays.
  • step ST12 the equipment detection unit 22 sets the equipment detection area 13 at a predetermined position of the image acquired by the image acquisition unit 21, and detects the rear window 12 from the equipment detection area 13.
  • step ST12 “NO” If the equipment detection unit 22 cannot detect the rear window 12 from the equipment detection area 13 (step ST12 “NO”), the driver 11 may have readjusted the posture of the rearview mirror 1 while the vehicle 10 is running. .. Therefore, in step ST16, the equipment detection unit 22 instructs the warning unit 25 to issue a warning prompting the driver 11 to adjust the posture of the rearview mirror 1.
  • the warning unit 25 issues a warning from the warning device 26 in accordance with this instruction.
  • the equipment detection unit 22 can detect the rear window 12 from the equipment detection area 13 (step ST12 “YES”), the detected image position of the rear window 12 is the image position of the rear window 12 detected in the previous step ST12. It is determined whether or not the change has occurred from (step ST13). In the first step ST13, the equipment detection unit 22 determines whether or not the image position of the rear window 12 detected in the first step ST13 has changed from the image position of the rear window 12 detected in step ST2 of FIG. You just have to judge.
  • step ST13 “YES”) If the image position of the rear window 12 changes (step ST13 “YES”), it is possible that the driver 11 has readjusted the posture of the rearview mirror 1 while the vehicle 10 is running. Therefore, in step ST17, the equipment detection unit 22 notifies the monitoring area setting unit 23 to reset the occupant monitoring area 15. In accordance with this notification, the monitoring area setting unit 23 resets the occupant monitoring area 15 with respect to the image acquired in step ST11 with reference to the image position of the rear window 12 detected in step ST12. Then, the monitoring area setting unit 23 notifies the occupant monitoring unit 24 of the image position of the reset occupant monitoring area 15.
  • step ST13 “NO”) the occupant monitoring unit 24 monitors the driver 11 using the image of the occupant monitoring area 15 among the images acquired in step ST11.
  • Step ST14 Even if the posture of the rear-view mirror 1 in which the infrared camera 7 is built is changed, the position of the occupant monitoring area 15 set in the image captured by the infrared camera 7 is changed according to the posture of the rear-view mirror 1. Therefore, even if the infrared camera 7 for occupant monitoring is arranged inside the rear-view mirror 1 whose posture changes, the driver 11 can be detected from the image.
  • step ST15 “YES” When an abnormality is found in the driver 11 (step ST15 “YES”), the occupant monitoring unit 24 instructs the warning unit 25 to issue a warning to the driver 11.
  • the warning unit 25 issues a warning from the warning device 26 in accordance with this instruction (step ST18).
  • step ST15 “NO” if no abnormality of the driver is found in the occupant monitoring unit 24 (step ST15 “NO”), the occupant monitoring device 20 performs the operation shown in the flowchart of FIG.
  • FIGS. 8A and 8B are diagrams showing a hardware configuration example of the occupant monitoring device 20 according to the first embodiment.
  • the functions of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25 in the occupant monitoring device 20 are realized by the processing circuit. That is, the occupant monitoring device 20 includes a processing circuit for realizing the above functions.
  • the processing circuit may be a processing circuit 100 as dedicated hardware, or a processor 101 that executes a program stored in the memory 102.
  • the processing circuit 100 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, or an ASIC (Application Specific Integrated Circuit). ), FPGA (Field Processor Gate Array), or a combination thereof.
  • the functions of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25 may be realized by a plurality of processing circuits 100, or the functions of each unit may be combined into one process. It may be realized by the circuit 100.
  • the functions of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25 are software, firmware, and so on. Or it is realized by a combination of software and firmware.
  • the software or firmware is described as a program and stored in the memory 102.
  • the processor 101 realizes the functions of each part by reading and executing the program stored in the memory 102. That is, the occupant monitoring device 20 includes a memory 102 for storing a program in which the steps shown in the flowcharts of FIGS. 5 and 7 are eventually executed when executed by the processor 101. Further, it can be said that this program causes the computer to execute the procedures or methods of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25.
  • the processor 101 is a CPU (Central Processing Unit), a processing device, an arithmetic unit, a microprocessor, or the like.
  • the memory 102 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), or a flash memory, and may be a non-volatile or volatile semiconductor memory such as a hard disk or a flexible disk. It may be an optical disc such as a CD (Compact Disc) or a DVD (Digital Versaille Disc).
  • the functions of the image acquisition unit 21, equipment detection unit 22, monitoring area setting unit 23, occupant monitoring unit 24, and warning unit 25 are partially realized by dedicated hardware and partly realized by software or firmware. You may try to do it.
  • the processing circuit in the occupant monitoring device 20 can realize the above-mentioned functions by hardware, software, firmware, or a combination thereof.
  • the occupant monitoring device 20 includes an image acquisition unit 21, an equipment detection unit 22, a monitoring area setting unit 23, and an occupant monitoring unit 24.
  • the image acquisition unit 21 acquires an image of the inside of the vehicle 10 captured by the infrared camera 7 built in the posture-adjustable rearview mirror 1.
  • the equipment detection unit 22 uses the equipment in the vehicle 10 imaged by the infrared camera 7 as the rear window 12 when the posture of the room mirror 1 is adjusted for use for the original purpose, and is acquired by the image acquisition unit 21.
  • the equipment detection area 13 is set at a predetermined position of the image, and the rear window 12 is detected from the equipment detection area 13.
  • the monitoring area setting unit 23 sets the occupant monitoring area 15 for the image with reference to the image position of the rear window 12.
  • the occupant monitoring unit 24 monitors the driver 11 using the image of the occupant monitoring area 15 set by the monitoring area setting unit 23 among the images acquired by the image acquisition unit 21.
  • the occupant monitoring device 20 can detect the driver 11 to be monitored from the image captured by the infrared camera 7 built in the posture-adjustable rearview mirror 1.
  • the occupant monitoring device 20 includes a warning unit 25.
  • the warning device 25 is the warning device 26 if the equipment detection unit 22 cannot detect the rear window 12 or at least one of the cases where at least a part of the occupant monitoring area 15 set by the monitoring area setting unit 23 protrudes from the image. Instruct to issue a warning.
  • the occupant monitoring device 20 is in a state in which the driver 11 provided in the vehicle 10 cannot see the rear view through the rearview mirror 1, or the occupant monitoring unit 24 cannot monitor the driver 11. It is possible to urge the driver 11 to readjust the posture of the rearview mirror 1. Therefore, the occupant monitoring device 20 makes both the posture in which the infrared camera 7 built in the rear-view mirror 1 can image the driver 11 to be monitored and the posture appropriate for using the rear-view mirror 1 for the original purpose. be able to.
  • the equipment detection unit 22 cannot detect the rear window 12 in steps ST2 and ST12. In that case, the equipment detection unit 22 detects a predetermined second equipment in the vehicle 10 from the image, and estimates the image position of the rear window 12 which is the first equipment with reference to the image position of the second equipment. You may. For example, when the left and right rear side windows 14 are the second equipment, the equipment detection unit 22 detects at least one of the left rear side window 14 and the right rear side window 14 from the image. The equipment detection unit 22 may determine whether the detected rear side window 14 is on the left side or the right side of the vehicle 10 based on the shape of the rear side window 14 or the angle of the rear pillar 16.
  • the equipment detection unit 22 estimates the image position of the rear window 12 with reference to the detected image position of the rear side window 14.
  • the positional relationship between the rear side window 14 on the left side and the rear window 12 and the positional relationship between the rear side window 14 on the right side and the rear window 12 are preset with respect to the equipment detection unit 22.
  • the monitoring area setting unit 23 sets the occupant monitoring area 15 for the image based on the image position of the rear window 12 estimated by the equipment detection unit 22.
  • the equipment detection unit 22 detects the rear window 12 from the equipment detection area 13. Therefore, it is desirable that the rear pillar 16 and the roof lining 17 around the rear window 12 are made of a material that does not absorb infrared rays. As a result, the infrared rays emitted by the infrared light emitting units 6a and 6b pass through the rear window 12 and are reflected around the rear window 12, and the difference in pixel values between the rear window 12 and its surroundings becomes large. Therefore, the rear window 12 can be easily detected by the equipment detection unit 22.
  • the occupant monitoring unit 24 monitors the driver 11, but occupants other than the driver 11 may be monitored.
  • the monitoring area setting unit 23 holds in advance information about the image position corresponding to the seat on which the monitored occupant such as the passenger seat is boarded as the position of the occupant monitoring area 15 based on the position of the rear window 12. ..
  • the imaging range of the infrared camera 7 is set to a range including the rear window 12 and the seat on which the monitored occupant is boarded when the posture of the rearview mirror 1 is adjusted for the original purpose.
  • the occupant monitoring unit 24 may monitor two or more occupants at the same time.
  • the monitoring area setting unit 23 sets two or more occupant monitoring areas 15 on the image with reference to the position of the rear window 12. Further, the imaging range of the infrared camera 7 is set to a range including the rear window 12 and all the seats on which the monitored occupant is boarded when the posture of the rearview mirror 1 is adjusted for the original purpose. Will be done.
  • the infrared camera 7 for occupant monitoring is built in the rear-view mirror 1, but it may be built in an in-vehicle device other than the rear-view mirror 1.
  • In-vehicle devices incorporating the infrared camera 7 include a display installed on the ceiling of the front seat and whose posture can be adjusted, an overhead console installed on the ceiling of the front seat and opened and closed, and the like.
  • the equipment detection unit 22 sets the equipment in the vehicle interior to be imaged by the infrared camera 7 built in the display or the like when the posture of the display or the like is adjusted for the original purpose. 1 facility.
  • the imaging range of the infrared camera 7 is set to a range including the first equipment and the seat on which the monitored occupant is boarded when the posture of the display or the like is adjusted for the original purpose. To.
  • the infrared camera 7 is used as the camera for occupant monitoring, but a visible light camera may be used.
  • the functions of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25 are integrated into the occupant monitoring device 20 mounted on the vehicle 10.
  • these functions may be distributed to two or more of the server device on the network, the mobile terminal such as a smartphone, or the occupant monitoring device 20.
  • the occupant monitoring device 20 may be constructed as a server device on the network.
  • the occupant monitoring system monitors occupants using images captured by a camera built in an in-vehicle device whose attitude can be adjusted, a vehicle, a railroad, a ship, a moving body including an aircraft, or the like can be used. Suitable for use in occupant monitoring systems that monitor occupants.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Mechanical Engineering (AREA)
  • Signal Processing (AREA)
  • Fittings On The Vehicle Exterior For Carrying Loads, And Devices For Holding Or Mounting Articles (AREA)

Abstract

In the present invention, an image acquisition unit (21) acquires an image of the inside of a vehicle (10), the image being captured by an infrared camera (7) installed in a posture-adjustable rear-view mirror (1). When the posture of the rear-view mirror (1) is adjusted for use according to the original purpose thereof, an equipment detection unit (22) sets equipment to be imaged by the infrared camera (7) to a rear window (12), sets an equipment detection region (13) to a predetermined position in the image acquired by the image acquisition unit (21), and detects the rear window (12) from the equipment detection region (13). When the rear window (12) is detected by the equipment detection unit (22), a monitoring region setting unit (23) sets an occupant monitoring region (15) in the image on the basis of an image position of the rear window (12). An occupant monitoring unit (24) monitors a driver (11) using the image of the occupant monitoring region (15).

Description

乗員監視装置、乗員監視システム、及び乗員監視方法Crew monitoring device, occupant monitoring system, and occupant monitoring method
　この発明は、乗員監視装置、乗員監視システム、及び乗員監視方法に関するものである。 The present invention relates to an occupant monitoring device, an occupant monitoring system, and an occupant monitoring method.
　特許文献１には、ルームミラーのケース内に収容されたカメラによって助手席に搭乗した乗員を撮像し、撮像した画像を用いて当該乗員を監視する乗員監視装置が記載されている。この乗員監視装置は、運転席に搭乗した乗員が後方を視認しやすいようにルームミラーを適切な姿勢に調整した場合に撮像されるフロントピラーを、当該ルームミラーに内蔵されたカメラが撮像した画像から検出する。そして、乗員監視装置は、フロントピラーが画像中の所定の位置に検出された場合、助手席に搭乗した乗員を撮像できていると判定する。 Patent Document 1 describes an occupant monitoring device that images an occupant in the passenger seat by a camera housed in a rear-view mirror case and monitors the occupant using the captured image. This occupant monitoring device captures an image of the front pillar imaged when the rearview mirror is adjusted to an appropriate posture so that the occupant in the driver's seat can easily see the rear view, by a camera built in the rearview mirror. Detect from. Then, when the front pillar is detected at a predetermined position in the image, the occupant monitoring device determines that the occupant in the passenger seat can be imaged.
特開２０００－２６４１２８号公報Japanese Unexamined Patent Publication No. 2000-264128
　しかしながら、運転席に搭乗した乗員が後方を視認できるルームミラーの姿勢には一定の許容範囲があるため、助手席に搭乗した乗員が常に画像中の特定の位置に写るとは限らない。そのため、特許文献１に記載された乗員監視装置は、画像に映る乗員が監視対象の乗員ではない場合であっても、画像に映る当該乗員を監視対象の乗員として誤って検出してしまう可能性があった。 However, since there is a certain tolerance in the posture of the rear-view mirror that allows the occupant in the driver's seat to see the rear, the occupant in the passenger seat is not always shown in a specific position in the image. Therefore, the occupant monitoring device described in Patent Document 1 may erroneously detect the occupant shown in the image as the occupant to be monitored even if the occupant shown in the image is not the occupant to be monitored. was there.
　この発明は、上記のような課題を解決するためになされたもので、ルームミラー等の姿勢調整可能な車載機器に内蔵されたカメラによって撮像された画像から、監視対象の乗員を検出することを目的とする。 The present invention has been made to solve the above-mentioned problems, and detects an occupant to be monitored from an image captured by a camera built in an in-vehicle device such as a rear-view mirror whose posture can be adjusted. The purpose.
　この発明に係る乗員監視装置は、姿勢調整可能な車載機器に内蔵されたカメラによって撮像された車室内の画像を取得する画像取得部と、車載機器の姿勢が本来の目的で使用するために調整された場合にカメラによって撮像される車室内の設備を第１設備とし、画像取得部により取得された画像の予め定められた位置に設備検出領域を設定して第１設備を設備検出領域から検出する設備検出部と、設備検出部により第１設備が検出された場合、第１設備の画像位置を基準にして、画像に対して乗員監視領域を設定する監視領域設定部と、画像取得部により取得された画像のうちの監視領域設定部により設定された乗員監視領域の画像を用いて乗員を監視する乗員監視部とを備えるものである。 The occupant monitoring device according to the present invention has an image acquisition unit that acquires an image of the vehicle interior captured by a camera built in an in-vehicle device whose attitude can be adjusted, and adjusts the attitude of the in-vehicle device so that it can be used for the original purpose. The equipment in the vehicle interior that is imaged by the camera is set as the first equipment, and the equipment detection area is set at a predetermined position of the image acquired by the image acquisition unit, and the first equipment is detected from the equipment detection area. When the first equipment is detected by the equipment detection unit and the equipment detection unit, the monitoring area setting unit that sets the occupant monitoring area for the image and the image acquisition unit based on the image position of the first equipment. It is provided with an occupant monitoring unit that monitors occupants using the image of the occupant monitoring area set by the monitoring area setting unit among the acquired images.
　この発明によれば、車載機器の姿勢が本来の目的で使用するために調整された場合にカメラによって撮像される第１設備を画像から検出し、第１設備の画像位置を基準にして画像に対して乗員監視領域を設定するようにしたので、姿勢調整可能な車載機器に内蔵されたカメラによって撮像された画像から監視対象の乗員を検出できる。 According to the present invention, when the posture of the in-vehicle device is adjusted for use for the original purpose, the first equipment imaged by the camera is detected from the image, and the image is obtained based on the image position of the first equipment. On the other hand, since the occupant monitoring area is set, the occupant to be monitored can be detected from the image captured by the camera built in the in-vehicle device whose posture can be adjusted.
実施の形態１に係るルームミラーの構成例を示す斜視図である。It is a perspective view which shows the structural example of the rearview mirror which concerns on Embodiment 1. FIG. 図１のルームミラーが設置された車両の例を示す斜視図である。It is a perspective view which shows the example of the vehicle which installed the rearview mirror of FIG. 実施の形態１に係るルームミラーに内蔵された赤外線カメラが撮像した画像の例を示す図である。It is a figure which shows the example of the image which the infrared camera built in the room mirror which concerns on Embodiment 1 took. 実施の形態１に係る乗員監視システムの構成例を示すブロック図である。It is a block diagram which shows the configuration example of the occupant monitoring system which concerns on Embodiment 1. FIG. 実施の形態１に係る乗員監視システムによるルームミラー初期姿勢調整の動作例を示すフローチャートである。It is a flowchart which shows the operation example of the room mirror initial posture adjustment by the occupant monitoring system which concerns on Embodiment 1. FIG. 図６Ａは、設備検出部がリアウインドウを検出できない画像の例を示し、図６Ｂは、監視領域設定部が設定した乗員監視領域の一部がはみ出た画像の例を示す。FIG. 6A shows an example of an image in which the equipment detection unit cannot detect the rear window, and FIG. 6B shows an example of an image in which a part of the occupant monitoring area set by the monitoring area setting unit protrudes. 実施の形態１に係る乗員監視システムによる乗員監視の動作例を示すフローチャートである。It is a flowchart which shows the operation example of the occupant monitoring by the occupant monitoring system which concerns on Embodiment 1. FIG. 図８Ａ及び図８Ｂは、実施の形態１に係る乗員監視装置のハードウェア構成例を示す図である。8A and 8B are diagrams showing a hardware configuration example of the occupant monitoring device according to the first embodiment.
　以下、この発明をより詳細に説明するために、この発明を実施するための形態について、添付の図面に従って説明する。
実施の形態１．
　図１は、実施の形態１に係るルームミラー１の構成例を示す斜視図である。図２は、図１のルームミラー１が設置された車両１０の例を示す斜視図である。ここでは、車両１０の左右方向がＸ軸、前後方向がＹ軸、上下方向がＺ軸である。ルームミラー１は、車両１０のフロントウインドウ（図示せず）又はその付近の天井に設置された車載機器であり、運転者１１が後方を視認するために使用される。図２に矢印で示されるように、運転者１１は、ルームミラー１を介してリアウインドウ１２を見ることで、車両１０の後方を確認する。 Hereinafter, in order to explain the present invention in more detail, a mode for carrying out the present invention will be described with reference to the accompanying drawings.
Embodiment 1.
FIG. 1 is a perspective view showing a configuration example of the rearview mirror 1 according to the first embodiment. FIG. 2 is a perspective view showing an example of a vehicle 10 in which the rearview mirror 1 of FIG. 1 is installed. Here, the left-right direction of the vehicle 10 is the X-axis, the front-rear direction is the Y-axis, and the up-down direction is the Z-axis. The rearview mirror 1 is an in-vehicle device installed on the front window (not shown) of the vehicle 10 or the ceiling in the vicinity thereof, and is used by the driver 11 to visually recognize the rear view. As shown by the arrows in FIG. 2, the driver 11 confirms the rear of the vehicle 10 by looking at the rear window 12 through the rear-view mirror 1.
　ルームミラー１は、ミラーケース２、アーム３、赤外線フィルタ４、ハーフミラー５、赤外線発光部６ａ，６ｂ、赤外線カメラ７、及び仕切り部材８を含む。ミラーケース２の背面にはアーム３が設けられている。アーム３は、ミラーケース２をフロントウインドウ（図示せず）又はその付近の天井に固定する。また、このアーム３は、ルームミラー１の姿勢をＸ軸方向及びＺ軸方向に変化させる。運転者１１がルームミラー１を手で掴んでルームミラー１の姿勢を調整してもよい。または、アーム３にアクチュエータが内蔵されている場合、運転者１１がアクチュエータを操作することによりルームミラー１の姿勢を調整してもよい。 The room mirror 1 includes a mirror case 2, an arm 3, an infrared filter 4, a half mirror 5, infrared light emitting units 6a and 6b, an infrared camera 7, and a partition member 8. An arm 3 is provided on the back surface of the mirror case 2. The arm 3 fixes the mirror case 2 to the front window (not shown) or the ceiling in the vicinity thereof. Further, the arm 3 changes the posture of the rearview mirror 1 in the X-axis direction and the Z-axis direction. The driver 11 may grasp the rear-view mirror 1 by hand and adjust the posture of the rear-view mirror 1. Alternatively, when the actuator 3 is built in the arm 3, the driver 11 may operate the actuator to adjust the posture of the rearview mirror 1.
　ミラーケース２の正面には、ハーフミラー５が装着されている。ハーフミラー５の裏面かつミラーケース２の内部には、赤外線フィルタ４が配置されている。赤外線フィルタ４の裏面かつミラーケース２の内部には、赤外線発光部６ａ，６ｂ及び赤外線カメラ７が配置されている。図１の例は、１つの赤外線カメラ７の両脇に赤外線発光部６ａと赤外線発光部６ｂとが配置された構成であるが、赤外線発光部６ａ又は赤外線発光部６ｂのいずれか一方のみが配置された構成であってもよい。 A half mirror 5 is mounted on the front of the mirror case 2. An infrared filter 4 is arranged on the back surface of the half mirror 5 and inside the mirror case 2. Infrared light emitting units 6a and 6b and an infrared camera 7 are arranged on the back surface of the infrared filter 4 and inside the mirror case 2. In the example of FIG. 1, the infrared light emitting unit 6a and the infrared light emitting unit 6b are arranged on both sides of one infrared camera 7, but only one of the infrared light emitting unit 6a and the infrared light emitting unit 6b is arranged. It may be a configured configuration.
　赤外線発光部６ａ，６ｂは、車両１０内に向けて赤外線を照射する。赤外線カメラ７は、赤外線発光部６ａ，６ｂが照射した赤外線のうち、車両１０内で反射しハーフミラー５及び赤外線フィルタ４を透過した赤外線を受光し、画像として出力する。なお、赤外線発光部６ａ、赤外線発光部６ｂ、及び赤外線カメラ７が仕切り部材８で仕切られている。そのため、赤外線発光部６ａ，６ｂは、照射した赤外線がハーフミラー５及び赤外線フィルタ４に反射して赤外線カメラ７に入射しない。 The infrared emitting units 6a and 6b irradiate the inside of the vehicle 10 with infrared rays. The infrared camera 7 receives the infrared rays reflected in the vehicle 10 and transmitted through the half mirror 5 and the infrared filter 4 among the infrared rays emitted by the infrared emitting units 6a and 6b, and outputs them as an image. The infrared light emitting unit 6a, the infrared light emitting unit 6b, and the infrared camera 7 are partitioned by a partition member 8. Therefore, the infrared rays emitted by the infrared light emitting units 6a and 6b are reflected by the half mirror 5 and the infrared filter 4 and do not enter the infrared camera 7.
　赤外線フィルタ４の裏面側に、赤外線発光部６ａ，６ｂ、赤外線カメラ７、及び仕切り部材８が配置されている。そのため、赤外線発光部６ａ，６ｂ、赤外線カメラ７、及び仕切り部材８は、ルームミラー１の外部から見えない。そのため、被写体である運転者１１及び同乗者に、赤外線カメラ７及び赤外線発光部６ａ，６ｂの存在を意識させることなく、乗員監視用の赤外線カメラ７を車両１０に配置することができる。 Infrared light emitting parts 6a and 6b, an infrared camera 7, and a partition member 8 are arranged on the back surface side of the infrared filter 4. Therefore, the infrared light emitting units 6a and 6b, the infrared camera 7, and the partition member 8 cannot be seen from the outside of the rearview mirror 1. Therefore, the infrared camera 7 for occupant monitoring can be arranged in the vehicle 10 without making the driver 11 and the passenger, who are the subjects, aware of the existence of the infrared camera 7 and the infrared light emitting units 6a and 6b.
　図３は、実施の形態１に係るルームミラー１に内蔵された赤外線カメラ７が撮像した画像の例を示す図である。図３に示される画像は、運転者１１が後方を視認しやすいようにルームミラー１を適切な姿勢に調整した場合に、赤外線カメラ７によって撮像された画像である。この画像には、運転席に搭乗している運転者１１、助手席、リアウインドウ１２、左右のリアサイドウインドウ１４、左右のリアピラー１６、及びルーフライニング１７等が写っている。赤外線カメラ７の撮像範囲は、ルームミラー１の姿勢が本来の目的で使用するために調整された場合に、リアウインドウ１２と乗員監視装置２０の監視対象である運転者１１が搭乗する運転席とが含まれる範囲に設定されているものとする。 FIG. 3 is a diagram showing an example of an image captured by the infrared camera 7 built in the rearview mirror 1 according to the first embodiment. The image shown in FIG. 3 is an image captured by the infrared camera 7 when the rearview mirror 1 is adjusted to an appropriate posture so that the driver 11 can easily see the rear. This image shows the driver 11, the passenger seat, the rear window 12, the left and right rear side windows 14, the left and right rear pillars 16, the roof lining 17, and the like in the driver's seat. The imaging range of the infrared camera 7 is the rear window 12 and the driver's seat on which the driver 11 who is the monitoring target of the occupant monitoring device 20 is boarded when the posture of the rearview mirror 1 is adjusted for the original purpose. It is assumed that it is set in the range that includes.
　図４は、実施の形態１に係る乗員監視システム３０の構成例を示すブロック図である。乗員監視システム３０は、ルームミラー１と、乗員監視装置２０とを含む。乗員監視装置２０には、ルームミラー１に内蔵された赤外線カメラ７と、車両１０に設けられた警告機器２６とが接続されている。この警告機器２６は、スピーカ又はディスプレイ等を有しており、音声又は表示等によって運転者１１に対して警告する。乗員監視装置２０は、画像取得部２１、設備検出部２２、監視領域設定部２３、乗員監視部２４、及び警告部２５を備える。 FIG. 4 is a block diagram showing a configuration example of the occupant monitoring system 30 according to the first embodiment. The occupant monitoring system 30 includes a rearview mirror 1 and an occupant monitoring device 20. The occupant monitoring device 20 is connected to an infrared camera 7 built in the rearview mirror 1 and a warning device 26 provided in the vehicle 10. The warning device 26 has a speaker, a display, or the like, and warns the driver 11 by voice, display, or the like. The occupant monitoring device 20 includes an image acquisition unit 21, an equipment detection unit 22, a monitoring area setting unit 23, an occupant monitoring unit 24, and a warning unit 25.
　画像取得部２１は、姿勢調整可能なルームミラー１に内蔵された赤外線カメラ７によって撮像された車両１０内の画像を、赤外線カメラ７から取得する。画像取得部２１は、取得した画像を、設備検出部２２及び乗員監視部２４へ出力する。 The image acquisition unit 21 acquires an image of the inside of the vehicle 10 captured by the infrared camera 7 built in the posture-adjustable rearview mirror 1 from the infrared camera 7. The image acquisition unit 21 outputs the acquired image to the equipment detection unit 22 and the occupant monitoring unit 24.
　設備検出部２２は、画像取得部２１から画像を受け取り、この画像の予め定められた位置に設備検出領域１３を設定する。ここで、運転者１１が後方を視認しやすいようにルームミラー１を適切な姿勢に調整した場合に、つまり車載機器であるルームミラー１の姿勢が本来の目的で使用するために調整された場合に、赤外線カメラ７によって撮像される車室内の第１設備をリアウインドウ１２とする。設備検出領域１３は、第１設備であるリアウインドウ１２を検出するための領域であり、画像内の固定位置に予め設定されている。設備検出部２２は、赤外線カメラ７が撮像した画像に設備検出領域１３を設定し、リアウインドウ１２を設備検出領域１３から検出する。設備検出領域１３からリアウインドウ１２を検出できた場合、運転者１１がルームミラー１を介して後方を視認できていることになる。設備検出領域１３からリアウインドウ１２を検出できなかった場合、運転者１１がルームミラー１を介して後方を視認できない状態である可能性が高い。設備検出部２２は、設備検出領域１３からリアウインドウ１２を検出できた場合、画像とリアウインドウ１２の画像位置とを監視領域設定部２３へ出力する。一方、設備検出領域１３から画像とリアウインドウ１２を検出できなかった場合、設備検出部２２は、検出できなかった旨を警告部２５に通知する。 The equipment detection unit 22 receives an image from the image acquisition unit 21 and sets the equipment detection area 13 at a predetermined position of this image. Here, when the rearview mirror 1 is adjusted to an appropriate posture so that the driver 11 can easily see the rear view, that is, when the posture of the rearview mirror 1 which is an in-vehicle device is adjusted to be used for the original purpose. In addition, the first equipment in the vehicle interior, which is imaged by the infrared camera 7, is the rear window 12. The equipment detection area 13 is an area for detecting the rear window 12, which is the first equipment, and is preset at a fixed position in the image. The equipment detection unit 22 sets the equipment detection area 13 in the image captured by the infrared camera 7, and detects the rear window 12 from the equipment detection area 13. When the rear window 12 can be detected from the equipment detection area 13, the driver 11 can see the rear view through the rearview mirror 1. If the rear window 12 cannot be detected from the equipment detection area 13, it is highly possible that the driver 11 cannot see the rear view through the rearview mirror 1. When the equipment detection unit 22 can detect the rear window 12 from the equipment detection area 13, the equipment detection unit 22 outputs the image and the image position of the rear window 12 to the monitoring area setting unit 23. On the other hand, when the image and the rear window 12 cannot be detected from the equipment detection area 13, the equipment detection unit 22 notifies the warning unit 25 that the detection has not been performed.
　監視領域設定部２３は、画像とリアウインドウ１２の画像位置とを、設備検出部２２から受け取る。監視領域設定部２３は、設備検出部２２から通知されたリアウインドウ１２の画像位置を基準にして、画像に対して乗員監視領域１５を設定する。なお、監視領域設定部２３は、リアウインドウ１２の位置を基準とした乗員監視領域１５の位置に関する情報を予め保持している。監視対象の乗員が運転者１１である場合の、リアウインドウ１２と乗員監視領域１５の位置関係は、図３に例示されている。監視領域設定部２３は、設定した乗員監視領域１５の画像位置を、乗員監視部２４へ通知する。監視領域設定部２３は、設定した乗員監視領域１５の一部又は全部（以下、「少なくとも一部」と称する）が画像からはみ出る場合、その旨を警告部２５に通知する。 The monitoring area setting unit 23 receives the image and the image position of the rear window 12 from the equipment detection unit 22. The monitoring area setting unit 23 sets the occupant monitoring area 15 for the image based on the image position of the rear window 12 notified from the equipment detection unit 22. The monitoring area setting unit 23 holds information regarding the position of the occupant monitoring area 15 with reference to the position of the rear window 12 in advance. The positional relationship between the rear window 12 and the occupant monitoring area 15 when the occupant to be monitored is the driver 11 is illustrated in FIG. The monitoring area setting unit 23 notifies the occupant monitoring unit 24 of the image position of the set occupant monitoring area 15. When a part or all (hereinafter, referred to as "at least a part") of the set occupant monitoring area 15 protrudes from the image, the monitoring area setting unit 23 notifies the warning unit 25 to that effect.
　乗員監視部２４は、画像取得部２１から画像を受け取り、監視領域設定部２３から通知された乗員監視領域１５をこの画像に設定する。乗員監視部２４は、画像取得部２１から受け取った画像のうちの乗員監視領域１５の画像を用いて、運転者１１を監視する。例えば、乗員監視部２４は、運転者１１の眠気、脇見、又は疲労等の状態を判定し、運転者１１が異常な状態であると判定した場合、その旨を警告部２５に通知する。 The occupant monitoring unit 24 receives an image from the image acquisition unit 21 and sets the occupant monitoring area 15 notified from the monitoring area setting unit 23 in this image. The occupant monitoring unit 24 monitors the driver 11 by using the image of the occupant monitoring area 15 among the images received from the image acquisition unit 21. For example, the occupant monitoring unit 24 determines a state such as drowsiness, inattentiveness, or fatigue of the driver 11, and if the driver 11 determines that the state is abnormal, the occupant monitoring unit 24 notifies the warning unit 25 to that effect.
　警告部２５は、設備検出部２２、監視領域設定部２３、又は乗員監視部２４からの通知を受けた場合、警告機器２６に対して警告を発するよう指示する。警告機器２６は、警告部２５から指示を受けると警告を発する。 When the warning unit 25 receives a notification from the equipment detection unit 22, the monitoring area setting unit 23, or the occupant monitoring unit 24, the warning unit 25 instructs the warning device 26 to issue a warning. The warning device 26 issues a warning when it receives an instruction from the warning unit 25.
　次に、乗員監視システム３０の動作を説明する。
　図５は、実施の形態１に係る乗員監視システム３０によるルームミラー初期姿勢調整の動作例を示すフローチャートである。例えば、乗員監視システム３０は、車両１０の発車前に毎回、図５のフローチャートに示される動作を行う。 Next, the operation of the occupant monitoring system 30 will be described.
FIG. 5 is a flowchart showing an operation example of the rearview mirror initial posture adjustment by the occupant monitoring system 30 according to the first embodiment. For example, the occupant monitoring system 30 performs the operation shown in the flowchart of FIG. 5 every time before the vehicle 10 departs.
　ステップＳＴ１において、画像取得部２１は、赤外線発光部６ａ，６ｂが赤外線を照射した状態で赤外線カメラ７が撮像した画像を、赤外線カメラ７から取得する。 In step ST1, the image acquisition unit 21 acquires an image captured by the infrared camera 7 from the infrared camera 7 in a state where the infrared light emitting units 6a and 6b are irradiated with infrared rays.
　ステップＳＴ２において、設備検出部２２は、画像取得部２１により取得された画像の予め定められた位置に設備検出領域１３を設定し、設備検出領域１３からリアウインドウ１２を検出する。 In step ST2, the equipment detection unit 22 sets the equipment detection area 13 at a predetermined position of the image acquired by the image acquisition unit 21, and detects the rear window 12 from the equipment detection area 13.
　設備検出部２２が設備検出領域１３からリアウインドウ１２を検出できた場合（ステップＳＴ２“ＹＥＳ”）、監視領域設定部２３は、設備検出部２２が検出したリアウインドウ１２の画像位置を基準にして、ステップＳＴ１で取得された画像に対して乗員監視領域１５を設定する（ステップＳＴ３）。 When the equipment detection unit 22 can detect the rear window 12 from the equipment detection area 13 (step ST2 “YES”), the monitoring area setting unit 23 uses the image position of the rear window 12 detected by the equipment detection unit 22 as a reference. , The occupant monitoring area 15 is set for the image acquired in step ST1 (step ST3).
　設備検出部２２が設備検出領域１３からリアウインドウ１２を検出できなかった場合（ステップＳＴ２“ＮＯ”）、運転者１１がルームミラー１を介して後方を視認できない状態である可能性が高い。そのため、ステップＳＴ５において、設備検出部２２は、ルームミラー１の姿勢調整を運転者１１に促す警告を発するよう、警告部２５に対して指示する。警告部２５は、この指示に従い、警告機器２６から警告を発する。 When the equipment detection unit 22 cannot detect the rear window 12 from the equipment detection area 13 (step ST2 “NO”), there is a high possibility that the driver 11 cannot see the rear view through the rearview mirror 1. Therefore, in step ST5, the equipment detection unit 22 instructs the warning unit 25 to issue a warning prompting the driver 11 to adjust the posture of the rearview mirror 1. The warning unit 25 issues a warning from the warning device 26 in accordance with this instruction.
　図６Ａは、設備検出部２２が設備検出領域１３からリアウインドウ１２を検出できない画像の例を示す。ルームミラー１が下を向いているため、この画像にはリアウインドウ１２が写っていない。例えば、設備検出部２２は、リアウインドウ１２が設備検出領域１３内に収まっている場合、設備検出領域１３からリアウインドウ１２を検出できたとし、リアウインドウ１２の少なくとも一部が設備検出領域１３からはみ出した場合、設備検出領域１３からリアウインドウ１２を検出できなかったとする。 FIG. 6A shows an example of an image in which the equipment detection unit 22 cannot detect the rear window 12 from the equipment detection area 13. Since the rearview mirror 1 is facing downward, the rear window 12 is not shown in this image. For example, the equipment detection unit 22 assumes that the rear window 12 can be detected from the equipment detection area 13 when the rear window 12 is contained in the equipment detection area 13, and at least a part of the rear window 12 is from the equipment detection area 13. If it protrudes, it is assumed that the rear window 12 cannot be detected from the equipment detection area 13.
　ステップＳＴ４において、監視領域設定部２３は、設定した乗員監視領域１５が画像内に収まるか否かを判定する。乗員監視領域１５が画像内に収まる場合（ステップＳＴ４“ＹＥＳ”）、監視領域設定部２３は、乗員監視領域１５の画像位置を、乗員監視部２４へ通知する。 In step ST4, the monitoring area setting unit 23 determines whether or not the set occupant monitoring area 15 fits in the image. When the occupant monitoring area 15 fits within the image (step ST4 “YES”), the monitoring area setting unit 23 notifies the occupant monitoring unit 24 of the image position of the occupant monitoring area 15.
　監視領域設定部２３は、設定した乗員監視領域１５の少なくとも一部が画像からはみ出る場合（ステップＳＴ４“ＮＯ”）、乗員監視部２４が運転者１１を監視できない状態である可能性が高い。そのため、ステップＳＴ６において、監視領域設定部２３は、ルームミラー１の姿勢調整を運転者１１に促す警告を発するよう、警告部２５に対して指示する。警告部２５は、この指示に従い、警告機器２６から警告を発する。 When at least a part of the set occupant monitoring area 15 protrudes from the image (step ST4 “NO”), the monitoring area setting unit 23 is likely to be in a state where the occupant monitoring unit 24 cannot monitor the driver 11. Therefore, in step ST6, the monitoring area setting unit 23 instructs the warning unit 25 to issue a warning prompting the driver 11 to adjust the posture of the rearview mirror 1. The warning unit 25 issues a warning from the warning device 26 in accordance with this instruction.
　図６Ｂは、監視領域設定部２３が設定した乗員監視領域１５の一部がはみ出た画像の例を示す。リアウインドウ１２の画像位置を基準にして設定された乗員監視領域１５の半分が、画像からはみ出した状態となっている。この例において、運転者１１は、ルームミラー１を介して後方を視認できない可能性が高い。 FIG. 6B shows an example of an image in which a part of the occupant monitoring area 15 set by the monitoring area setting unit 23 protrudes. Half of the occupant monitoring area 15 set with reference to the image position of the rear window 12 is in a state of protruding from the image. In this example, it is highly possible that the driver 11 cannot see the rear view through the rearview mirror 1.
　ステップＳＴ５又はステップＳＴ６において警告機器２６が警告を発した後、乗員監視装置２０は、運転者１１がルームミラー１の姿勢を調整したと仮定して、図５のフローチャートに示される動作を行う。なお、ステップＳＴ５又はステップＳＴ６において警告機器２６によって警告が発せられるような場合は、運転者１１の顔が乗員監視領域１５内に収まらない可能性が高いため、乗員監視部２４による運転者１１の監視が困難となる。 After the warning device 26 issues a warning in step ST5 or step ST6, the occupant monitoring device 20 performs the operation shown in the flowchart of FIG. 5, assuming that the driver 11 has adjusted the posture of the rearview mirror 1. If the warning device 26 issues a warning in step ST5 or step ST6, there is a high possibility that the face of the driver 11 will not fit within the occupant monitoring area 15. Therefore, the driver 11 by the occupant monitoring unit 24 It becomes difficult to monitor.
　乗員監視装置２０が、車両１０の発車前に運転者１１に対してルームミラー１の姿勢を調整するよう促すので、運転者１１がルームミラー１の姿勢調整を忘れて発車して走行中にルームミラー１の姿勢を調整することを抑制できる。そのため、運転者１１を運転に集中させることができる。また、例えば、複数人で車両１０をシェアする場合、発車前に、交代後の運転者が自身にとって適切な姿勢にルームミラー１を調整できる。 Since the occupant monitoring device 20 urges the driver 11 to adjust the posture of the rear-view mirror 1 before the vehicle 10 departs, the driver 11 forgets to adjust the posture of the rear-view mirror 1 and departs to the room while driving. It is possible to suppress adjusting the posture of the mirror 1. Therefore, the driver 11 can be concentrated on driving. Further, for example, when the vehicle 10 is shared by a plurality of people, the driver after the change can adjust the rearview mirror 1 to a posture suitable for himself / herself before the vehicle departs.
　図７は、実施の形態１に係る乗員監視システム３０による乗員監視の動作例を示すフローチャートである。乗員監視システム３０は、図５に示されるルームミラー初期位置調整の動作を行った後、図７のフローチャートに示される動作を開始する。図７のフローチャートに示される動作の開始前に、乗員監視部２４は、乗員監視領域１５の画像位置を監視領域設定部２３から受け取り済みである。 FIG. 7 is a flowchart showing an operation example of occupant monitoring by the occupant monitoring system 30 according to the first embodiment. The occupant monitoring system 30 starts the operation shown in the flowchart of FIG. 7 after performing the operation of adjusting the initial position of the rearview mirror shown in FIG. Prior to the start of the operation shown in the flowchart of FIG. 7, the occupant monitoring unit 24 has already received the image position of the occupant monitoring area 15 from the monitoring area setting unit 23.
　ステップＳＴ１１において、画像取得部２１は、赤外線発光部６ａ，６ｂが赤外線を照射した状態で赤外線カメラ７が撮像した画像を、赤外線カメラ７から取得する。 In step ST11, the image acquisition unit 21 acquires an image captured by the infrared camera 7 from the infrared camera 7 in a state where the infrared light emitting units 6a and 6b are irradiated with infrared rays.
　ステップＳＴ１２において、設備検出部２２は、画像取得部２１により取得された画像の予め定められた位置に設備検出領域１３を設定し、設備検出領域１３からリアウインドウ１２を検出する。 In step ST12, the equipment detection unit 22 sets the equipment detection area 13 at a predetermined position of the image acquired by the image acquisition unit 21, and detects the rear window 12 from the equipment detection area 13.
　設備検出部２２が設備検出領域１３からリアウインドウ１２を検出できなかった場合（ステップＳＴ１２“ＮＯ”）、車両１０の走行中に運転者１１がルームミラー１の姿勢を再調整した可能性がある。そのため、ステップＳＴ１６において、設備検出部２２は、ルームミラー１の姿勢調整を運転者１１に促す警告を発するよう、警告部２５に対して指示する。警告部２５は、この指示に従い、警告機器２６から警告を発する。 If the equipment detection unit 22 cannot detect the rear window 12 from the equipment detection area 13 (step ST12 “NO”), the driver 11 may have readjusted the posture of the rearview mirror 1 while the vehicle 10 is running. .. Therefore, in step ST16, the equipment detection unit 22 instructs the warning unit 25 to issue a warning prompting the driver 11 to adjust the posture of the rearview mirror 1. The warning unit 25 issues a warning from the warning device 26 in accordance with this instruction.
　設備検出部２２は、設備検出領域１３からリアウインドウ１２を検出できた場合（ステップＳＴ１２“ＹＥＳ”）、検出したリアウインドウ１２の画像位置が、前回のステップＳＴ１２で検出したリアウインドウ１２の画像位置から変化したか否かを判定する（ステップＳＴ１３）。なお、初回のステップＳＴ１３においては、設備検出部２２は、初回のステップＳＴ１３で検出したリアウインドウ１２の画像位置が図５のステップＳＴ２で検出したリアウインドウ１２の画像位置から変化したか否かを判定すればよい。 When the equipment detection unit 22 can detect the rear window 12 from the equipment detection area 13 (step ST12 “YES”), the detected image position of the rear window 12 is the image position of the rear window 12 detected in the previous step ST12. It is determined whether or not the change has occurred from (step ST13). In the first step ST13, the equipment detection unit 22 determines whether or not the image position of the rear window 12 detected in the first step ST13 has changed from the image position of the rear window 12 detected in step ST2 of FIG. You just have to judge.
　リアウインドウ１２の画像位置が変化した場合（ステップＳＴ１３“ＹＥＳ”）、車両１０の走行中に運転者１１がルームミラー１の姿勢を再調整した可能性がある。そのため、ステップＳＴ１７において、設備検出部２２は、乗員監視領域１５を再設定するよう、監視領域設定部２３に通知する。監視領域設定部２３は、この通知に従い、ステップＳＴ１２で検出されたリアウインドウ１２の画像位置を基準にして、ステップＳＴ１１で取得された画像に対して乗員監視領域１５を再設定する。そして、監視領域設定部２３は、再設定した乗員監視領域１５の画像位置を、乗員監視部２４に通知する。 If the image position of the rear window 12 changes (step ST13 “YES”), it is possible that the driver 11 has readjusted the posture of the rearview mirror 1 while the vehicle 10 is running. Therefore, in step ST17, the equipment detection unit 22 notifies the monitoring area setting unit 23 to reset the occupant monitoring area 15. In accordance with this notification, the monitoring area setting unit 23 resets the occupant monitoring area 15 with respect to the image acquired in step ST11 with reference to the image position of the rear window 12 detected in step ST12. Then, the monitoring area setting unit 23 notifies the occupant monitoring unit 24 of the image position of the reset occupant monitoring area 15.
　リアウインドウ１２の画像位置が変化していない場合（ステップＳＴ１３“ＮＯ”）、乗員監視部２４は、ステップＳＴ１１で取得された画像のうちの乗員監視領域１５の画像を用いて運転者１１を監視する（ステップＳＴ１４）。赤外線カメラ７が内蔵されたルームミラー１の姿勢が変化した場合でも、この赤外線カメラ７が撮像した画像に設定する乗員監視領域１５の位置がルームミラー１の姿勢に応じて変更される。そのため、姿勢が変化するルームミラー１の内部に乗員監視用の赤外線カメラ７を配置しても、画像から運転者１１を検出できる。 When the image position of the rear window 12 has not changed (step ST13 “NO”), the occupant monitoring unit 24 monitors the driver 11 using the image of the occupant monitoring area 15 among the images acquired in step ST11. (Step ST14). Even if the posture of the rear-view mirror 1 in which the infrared camera 7 is built is changed, the position of the occupant monitoring area 15 set in the image captured by the infrared camera 7 is changed according to the posture of the rear-view mirror 1. Therefore, even if the infrared camera 7 for occupant monitoring is arranged inside the rear-view mirror 1 whose posture changes, the driver 11 can be detected from the image.
　運転者１１の異常が認められた場合（ステップＳＴ１５“ＹＥＳ”）、乗員監視部２４は、運転者１１に警告を発するよう、警告部２５に対して指示する。警告部２５は、この指示に従い、警告機器２６から警告を発する（ステップＳＴ１８）。一方、乗員監視部２４において運転者の異常が認められなければ（ステップＳＴ１５“ＮＯ”）、乗員監視装置２０は、図７のフローチャートに示される動作を行う。 When an abnormality is found in the driver 11 (step ST15 “YES”), the occupant monitoring unit 24 instructs the warning unit 25 to issue a warning to the driver 11. The warning unit 25 issues a warning from the warning device 26 in accordance with this instruction (step ST18). On the other hand, if no abnormality of the driver is found in the occupant monitoring unit 24 (step ST15 “NO”), the occupant monitoring device 20 performs the operation shown in the flowchart of FIG.
　最後に、実施の形態１に係る乗員監視装置２０のハードウェア構成を説明する。
　図８Ａ及び図８Ｂは、実施の形態１に係る乗員監視装置２０のハードウェア構成例を示す図である。乗員監視装置２０における画像取得部２１、設備検出部２２、監視領域設定部２３、乗員監視部２４、及び警告部２５の機能は、処理回路により実現される。即ち、乗員監視装置２０は、上記機能を実現するための処理回路を備える。処理回路は、専用のハードウェアとしての処理回路１００であってもよいし、メモリ１０２に格納されるプログラムを実行するプロセッサ１０１であってもよい。 Finally, the hardware configuration of the occupant monitoring device 20 according to the first embodiment will be described.
8A and 8B are diagrams showing a hardware configuration example of the occupant monitoring device 20 according to the first embodiment. The functions of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25 in the occupant monitoring device 20 are realized by the processing circuit. That is, the occupant monitoring device 20 includes a processing circuit for realizing the above functions. The processing circuit may be a processing circuit 100 as dedicated hardware, or a processor 101 that executes a program stored in the memory 102.
　図８Ａに示されるように、処理回路が専用のハードウェアである場合、処理回路１００は、例えば、単一回路、複合回路、プログラム化したプロセッサ、並列プログラム化したプロセッサ、ＡＳＩＣ（Ａｐｐｌｉｃａｔｉｏｎ　Ｓｐｅｃｉｆｉｃ　Ｉｎｔｅｇｒａｔｅｄ　Ｃｉｒｃｕｉｔ）、ＦＰＧＡ（Ｆｉｅｌｄ　Ｐｒｏｇｒａｍｍａｂｌｅ　Ｇａｔｅ　Ａｒｒａｙ）、又はこれらを組み合わせたものが該当する。画像取得部２１、設備検出部２２、監視領域設定部２３、乗員監視部２４、及び警告部２５の機能を複数の処理回路１００で実現してもよいし、各部の機能をまとめて１つの処理回路１００で実現してもよい。 As shown in FIG. 8A, when the processing circuit is dedicated hardware, the processing circuit 100 may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, or an ASIC (Application Specific Integrated Circuit). ), FPGA (Field Processor Gate Array), or a combination thereof. The functions of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25 may be realized by a plurality of processing circuits 100, or the functions of each unit may be combined into one process. It may be realized by the circuit 100.
　図８Ｂに示されるように、処理回路がプロセッサ１０１である場合、画像取得部２１、設備検出部２２、監視領域設定部２３、乗員監視部２４、及び警告部２５の機能は、ソフトウェア、ファームウェア、又はソフトウェアとファームウェアとの組み合わせにより実現される。ソフトウェア又はファームウェアはプログラムとして記述され、メモリ１０２に格納される。プロセッサ１０１は、メモリ１０２に格納されたプログラムを読みだして実行することにより、各部の機能を実現する。即ち、乗員監視装置２０は、プロセッサ１０１により実行されるときに、図５及び図７のフローチャートで示されるステップが結果的に実行されることになるプログラムを格納するためのメモリ１０２を備える。また、このプログラムは、画像取得部２１、設備検出部２２、監視領域設定部２３、乗員監視部２４、及び警告部２５の手順又は方法をコンピュータに実行させるものであるとも言える。 As shown in FIG. 8B, when the processing circuit is the processor 101, the functions of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25 are software, firmware, and so on. Or it is realized by a combination of software and firmware. The software or firmware is described as a program and stored in the memory 102. The processor 101 realizes the functions of each part by reading and executing the program stored in the memory 102. That is, the occupant monitoring device 20 includes a memory 102 for storing a program in which the steps shown in the flowcharts of FIGS. 5 and 7 are eventually executed when executed by the processor 101. Further, it can be said that this program causes the computer to execute the procedures or methods of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25.
　ここで、プロセッサ１０１とは、ＣＰＵ（Ｃｅｎｔｒａｌ　Ｐｒｏｃｅｓｓｉｎｇ　Ｕｎｉｔ）、処理装置、演算装置、又はマイクロプロセッサ等のことである。
　メモリ１０２は、ＲＡＭ（Ｒａｎｄｏｍ　Ａｃｃｅｓｓ　Ｍｅｍｏｒｙ）、ＲＯＭ（Ｒｅａｄ　Ｏｎｌｙ　Ｍｅｍｏｒｙ）、ＥＰＲＯＭ（Ｅｒａｓａｂｌｅ　Ｐｒｏｇｒａｍｍａｂｌｅ　ＲＯＭ）、又はフラッシュメモリ等の不揮発性もしくは揮発性の半導体メモリであってもよいし、ハードディスク又はフレキシブルディスク等の磁気ディスクであってもよいし、ＣＤ（Ｃｏｍｐａｃｔ　Ｄｉｓｃ）又はＤＶＤ（Ｄｉｇｉｔａｌ　Ｖｅｒｓａｔｉｌｅ　Ｄｉｓｃ）等の光ディスクであってもよい。 Here, the processor 101 is a CPU (Central Processing Unit), a processing device, an arithmetic unit, a microprocessor, or the like.
The memory 102 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), or a flash memory, and may be a non-volatile or volatile semiconductor memory such as a hard disk or a flexible disk. It may be an optical disc such as a CD (Compact Disc) or a DVD (Digital Versaille Disc).
　なお、画像取得部２１、設備検出部２２、監視領域設定部２３、乗員監視部２４、及び警告部２５の機能について、一部を専用のハードウェアで実現し、一部をソフトウェア又はファームウェアで実現するようにしてもよい。このように、乗員監視装置２０における処理回路は、ハードウェア、ソフトウェア、ファームウェア、又はこれらの組み合わせによって、上述の機能を実現することができる。 The functions of the image acquisition unit 21, equipment detection unit 22, monitoring area setting unit 23, occupant monitoring unit 24, and warning unit 25 are partially realized by dedicated hardware and partly realized by software or firmware. You may try to do it. As described above, the processing circuit in the occupant monitoring device 20 can realize the above-mentioned functions by hardware, software, firmware, or a combination thereof.
　以上のように、実施の形態１に係る乗員監視装置２０は、画像取得部２１、設備検出部２２、監視領域設定部２３、及び乗員監視部２４を備える。画像取得部２１は、姿勢調整可能なルームミラー１に内蔵された赤外線カメラ７によって撮像された車両１０内の画像を取得する。設備検出部２２は、ルームミラー１の姿勢が本来の目的で使用するために調整された場合に赤外線カメラ７によって撮像される車両１０内の設備をリアウインドウ１２とし、画像取得部２１により取得された画像の予め定められた位置に設備検出領域１３を設定してリアウインドウ１２を設備検出領域１３から検出する。監視領域設定部２３は、設備検出部２２によりリアウインドウ１２が検出された場合、リアウインドウ１２の画像位置を基準にして、画像に対して乗員監視領域１５を設定する。乗員監視部２４は、画像取得部２１により取得された画像のうちの監視領域設定部２３により設定された乗員監視領域１５の画像を用いて運転者１１を監視する。この構成により、乗員監視装置２０は、姿勢調整可能なルームミラー１に内蔵された赤外線カメラ７によって撮像された画像から、監視対象の運転者１１を検出できる。 As described above, the occupant monitoring device 20 according to the first embodiment includes an image acquisition unit 21, an equipment detection unit 22, a monitoring area setting unit 23, and an occupant monitoring unit 24. The image acquisition unit 21 acquires an image of the inside of the vehicle 10 captured by the infrared camera 7 built in the posture-adjustable rearview mirror 1. The equipment detection unit 22 uses the equipment in the vehicle 10 imaged by the infrared camera 7 as the rear window 12 when the posture of the room mirror 1 is adjusted for use for the original purpose, and is acquired by the image acquisition unit 21. The equipment detection area 13 is set at a predetermined position of the image, and the rear window 12 is detected from the equipment detection area 13. When the rear window 12 is detected by the equipment detection unit 22, the monitoring area setting unit 23 sets the occupant monitoring area 15 for the image with reference to the image position of the rear window 12. The occupant monitoring unit 24 monitors the driver 11 using the image of the occupant monitoring area 15 set by the monitoring area setting unit 23 among the images acquired by the image acquisition unit 21. With this configuration, the occupant monitoring device 20 can detect the driver 11 to be monitored from the image captured by the infrared camera 7 built in the posture-adjustable rearview mirror 1.
　また、実施の形態１に係る乗員監視装置２０は、警告部２５を備える。警告部２５は、設備検出部２２がリアウインドウ１２を検出できない場合、又は監視領域設定部２３が設定した乗員監視領域１５の少なくとも一部が画像からはみ出る場合の少なくとも一方であれば、警告機器２６に対して警告を発するよう指示する。この構成により、乗員監視装置２０は、車両１０内に設けられた運転者１１がルームミラー１を介して後方を視認できない状態、又は、乗員監視部２４が運転者１１を監視できない状態である場合に、運転者１１に対してルームミラー１の姿勢を再調整するように促すことができる。したがって、乗員監視装置２０は、ルームミラー１に内蔵された赤外線カメラ７が監視対象の運転者１１を撮像できる姿勢と、ルームミラー１を本来の目的で使用するために適切な姿勢とを両立させることができる。 Further, the occupant monitoring device 20 according to the first embodiment includes a warning unit 25. The warning device 25 is the warning device 26 if the equipment detection unit 22 cannot detect the rear window 12 or at least one of the cases where at least a part of the occupant monitoring area 15 set by the monitoring area setting unit 23 protrudes from the image. Instruct to issue a warning. With this configuration, the occupant monitoring device 20 is in a state in which the driver 11 provided in the vehicle 10 cannot see the rear view through the rearview mirror 1, or the occupant monitoring unit 24 cannot monitor the driver 11. It is possible to urge the driver 11 to readjust the posture of the rearview mirror 1. Therefore, the occupant monitoring device 20 makes both the posture in which the infrared camera 7 built in the rear-view mirror 1 can image the driver 11 to be monitored and the posture appropriate for using the rear-view mirror 1 for the original purpose. be able to.
　なお、車両１０内に荷物等が存在することによりリアウインドウ１２が隠された場合、ステップＳＴ２，ＳＴ１２において、設備検出部２２はリアウインドウ１２を検出できない。その場合、設備検出部２２は、車両１０内の予め定められた第２設備を画像から検出し、第２設備の画像位置を基準にして第１設備であるリアウインドウ１２の画像位置を推定してもよい。例えば、左右のリアサイドウインドウ１４が第２設備である場合、設備検出部２２は、左側のリアサイドウインドウ１４又は右側のリアサイドウインドウ１４の少なくとも一方を画像から検出する。設備検出部２２は、リアサイドウインドウ１４の形状又はリアピラー１６の角度等に基づいて、検出したリアサイドウインドウ１４が車両１０の左側か右側かを判定すればよい。設備検出部２２は、左側のリアサイドウインドウ１４又は右側のリアサイドウインドウ１４の少なくとも一方を画像から検出した場合、検出したリアサイドウインドウ１４の画像位置を基準にして、リアウインドウ１２の画像位置を推定する。なお、左側のリアサイドウインドウ１４とリアウインドウ１２との位置関係、及び右側のリアサイドウインドウ１４とリアウインドウ１２との位置関係は、設備検出部２２に対して予め設定されている。監視領域設定部２３は、設備検出部２２が推定したリアウインドウ１２の画像位置を基準にして、画像に対して乗員監視領域１５を設定する。 If the rear window 12 is hidden due to the presence of luggage or the like in the vehicle 10, the equipment detection unit 22 cannot detect the rear window 12 in steps ST2 and ST12. In that case, the equipment detection unit 22 detects a predetermined second equipment in the vehicle 10 from the image, and estimates the image position of the rear window 12 which is the first equipment with reference to the image position of the second equipment. You may. For example, when the left and right rear side windows 14 are the second equipment, the equipment detection unit 22 detects at least one of the left rear side window 14 and the right rear side window 14 from the image. The equipment detection unit 22 may determine whether the detected rear side window 14 is on the left side or the right side of the vehicle 10 based on the shape of the rear side window 14 or the angle of the rear pillar 16. When at least one of the left rear side window 14 and the right rear side window 14 is detected from the image, the equipment detection unit 22 estimates the image position of the rear window 12 with reference to the detected image position of the rear side window 14. The positional relationship between the rear side window 14 on the left side and the rear window 12 and the positional relationship between the rear side window 14 on the right side and the rear window 12 are preset with respect to the equipment detection unit 22. The monitoring area setting unit 23 sets the occupant monitoring area 15 for the image based on the image position of the rear window 12 estimated by the equipment detection unit 22.
　また、夜間に車両１０が走行している場合等、車外が暗い場合、設備検出部２２が設備検出領域１３からリアウインドウ１２を検出するのが困難な可能性がある。そこで、リアウインドウ１２周辺のリアピラー１６及びルーフライニング１７等は、赤外線を吸収しない素材で構成されていることが望ましい。これにより、赤外線発光部６ａ，６ｂが照射した赤外線は、リアウインドウ１２を透過する一方、リアウインドウ１２の周辺で反射するようになり、リアウインドウ１２とその周辺とで画素値の差が大きくなるため、設備検出部２２におけるリアウインドウ１２の検出が容易になる。 Further, when the outside of the vehicle is dark, such as when the vehicle 10 is traveling at night, it may be difficult for the equipment detection unit 22 to detect the rear window 12 from the equipment detection area 13. Therefore, it is desirable that the rear pillar 16 and the roof lining 17 around the rear window 12 are made of a material that does not absorb infrared rays. As a result, the infrared rays emitted by the infrared light emitting units 6a and 6b pass through the rear window 12 and are reflected around the rear window 12, and the difference in pixel values between the rear window 12 and its surroundings becomes large. Therefore, the rear window 12 can be easily detected by the equipment detection unit 22.
　また、実施の形態１では、乗員監視部２４が運転者１１を監視したが、運転者１１以外の乗員を監視してもよい。その場合、監視領域設定部２３は、リアウインドウ１２の位置を基準とした乗員監視領域１５の位置として、助手席等の監視対象の乗員が搭乗する座席に相当する画像位置に関する情報を予め保持する。また、赤外線カメラ７の撮像範囲は、ルームミラー１の姿勢が本来の目的で使用するために調整された場合に、リアウインドウ１２と監視対象の乗員が搭乗する座席とが含まれる範囲に設定される。
　なお、乗員監視部２４が２人以上の乗員を同時に監視してもよい。その場合、監視領域設定部２３は、リアウインドウ１２の位置を基準として、２つ以上の乗員監視領域１５を画像に設定する。また、赤外線カメラ７の撮像範囲は、ルームミラー１の姿勢が本来の目的で使用するために調整された場合に、リアウインドウ１２と監視対象の乗員が搭乗する全座席とが含まれる範囲に設定される。 Further, in the first embodiment, the occupant monitoring unit 24 monitors the driver 11, but occupants other than the driver 11 may be monitored. In that case, the monitoring area setting unit 23 holds in advance information about the image position corresponding to the seat on which the monitored occupant such as the passenger seat is boarded as the position of the occupant monitoring area 15 based on the position of the rear window 12. .. Further, the imaging range of the infrared camera 7 is set to a range including the rear window 12 and the seat on which the monitored occupant is boarded when the posture of the rearview mirror 1 is adjusted for the original purpose. To.
The occupant monitoring unit 24 may monitor two or more occupants at the same time. In that case, the monitoring area setting unit 23 sets two or more occupant monitoring areas 15 on the image with reference to the position of the rear window 12. Further, the imaging range of the infrared camera 7 is set to a range including the rear window 12 and all the seats on which the monitored occupant is boarded when the posture of the rearview mirror 1 is adjusted for the original purpose. Will be done.
　また、実施の形態１では、乗員監視用の赤外線カメラ７がルームミラー１に内蔵されたが、ルームミラー１以外の車載機器に内蔵されてもよい。赤外線カメラ７を内蔵する車載機器としては、前席の天井に設置されて姿勢調整可能なディスプレイ、又は前席の天井に設置されて開閉するオーバヘッドコンソール等がある。その場合、設備検出部２２は、上記ディスプレイ等の姿勢が本来の目的で使用するために調整された場合に、上記ディスプレイ等に内蔵された赤外線カメラ７によって撮像される車室内の設備を、第１設備とする。また、赤外線カメラ７の撮像範囲は、上記ディスプレイ等の姿勢が本来の目的で使用するために調整された場合に、第１設備と監視対象の乗員が搭乗する座席とが含まれる範囲に設定される。 Further, in the first embodiment, the infrared camera 7 for occupant monitoring is built in the rear-view mirror 1, but it may be built in an in-vehicle device other than the rear-view mirror 1. In-vehicle devices incorporating the infrared camera 7 include a display installed on the ceiling of the front seat and whose posture can be adjusted, an overhead console installed on the ceiling of the front seat and opened and closed, and the like. In that case, the equipment detection unit 22 sets the equipment in the vehicle interior to be imaged by the infrared camera 7 built in the display or the like when the posture of the display or the like is adjusted for the original purpose. 1 facility. Further, the imaging range of the infrared camera 7 is set to a range including the first equipment and the seat on which the monitored occupant is boarded when the posture of the display or the like is adjusted for the original purpose. To.
　また、実施の形態１では、乗員監視用のカメラとして赤外線カメラ７が使用されたが、可視光カメラが使用されてもよい。 Further, in the first embodiment, the infrared camera 7 is used as the camera for occupant monitoring, but a visible light camera may be used.
　また、実施の形態１では、画像取得部２１、設備検出部２２、監視領域設定部２３、乗員監視部２４、及び警告部２５の機能が、車両１０に搭載される乗員監視装置２０に集約された構成であったが、これらの機能がネットワーク上のサーバ装置、スマートフォン等の携帯端末、又は乗員監視装置２０のうちの２つ以上の装置に分散されていてもよい。また、乗員監視装置２０がネットワーク上のサーバ装置として構築されていてもよい。 Further, in the first embodiment, the functions of the image acquisition unit 21, the equipment detection unit 22, the monitoring area setting unit 23, the occupant monitoring unit 24, and the warning unit 25 are integrated into the occupant monitoring device 20 mounted on the vehicle 10. However, these functions may be distributed to two or more of the server device on the network, the mobile terminal such as a smartphone, or the occupant monitoring device 20. Further, the occupant monitoring device 20 may be constructed as a server device on the network.
　なお、本発明はその発明の範囲内において、実施の形態の任意の構成要素の変形、又は実施の形態の任意の構成要素の省略が可能である。 It should be noted that, within the scope of the present invention, it is possible to modify any component of the embodiment or omit any component of the embodiment.
　この発明に係る乗員監視システムは、姿勢調整可能な車載機器に内蔵されたカメラが撮像した画像を用いて乗員を監視するようにしたので、車両、鉄道、船舶、又は航空機を含む移動体等の乗員を監視する乗員監視システムに用いるのに適している。 Since the occupant monitoring system according to the present invention monitors occupants using images captured by a camera built in an in-vehicle device whose attitude can be adjusted, a vehicle, a railroad, a ship, a moving body including an aircraft, or the like can be used. Suitable for use in occupant monitoring systems that monitor occupants.
　１　ルームミラー（車載機器）、２　ミラーケース、３　アーム、４　赤外線フィルタ、５　ハーフミラー、６ａ，６ｂ　赤外線発光部、７　赤外線カメラ、８　仕切り部材、１０　車両、１１　運転者、１２　リアウインドウ（第１設備）、１３　設備検出領域、１４　リアサイドウインドウ、１５　乗員監視領域、１６　リアピラー、１７　ルーフライニング、２０　乗員監視装置、２１　画像取得部、２２　設備検出部、２３　監視領域設定部、２４　乗員監視部、２５　警告部、２６　警告機器、３０　乗員監視システム、１００　処理回路、１０１　プロセッサ、１０２　メモリ。 1 Room mirror (vehicle-mounted device), 2 Mirror case, 3 Arm, 4 Infrared filter, 5 Half mirror, 6a, 6b Infrared light emitting part, 7 Infrared camera, 8 Partition member, 10 Vehicle, 11 Driver, 12 Rear window (No. 1 equipment), 13 equipment detection area, 14 rear side window, 15 occupant monitoring area, 16 rear pillar, 17 roof lining, 20 occupant monitoring device, 21 image acquisition unit, 22 equipment detection unit, 23 monitoring area setting unit, 24 occupant monitoring unit , 25 warning unit, 26 warning equipment, 30 occupant monitoring system, 100 processing circuit, 101 processor, 102 memory.

Claims (7)

  1. 　姿勢調整可能な車載機器に内蔵されたカメラによって撮像された車室内の画像を取得する画像取得部と、
    　前記車載機器の姿勢が本来の目的で使用するために調整された場合に前記カメラによって撮像される前記車室内の設備を第１設備とし、前記画像取得部により取得された画像の予め定められた位置に設備検出領域を設定して前記第１設備を前記設備検出領域から検出する設備検出部と、
    　前記設備検出部により前記第１設備が検出された場合、前記第１設備の画像位置を基準にして、前記画像に対して乗員監視領域を設定する監視領域設定部と、
    　前記画像取得部により取得された画像のうちの前記監視領域設定部により設定された前記乗員監視領域の画像を用いて乗員を監視する乗員監視部とを備える乗員監視装置。     An image acquisition unit that acquires an image of the interior of the vehicle captured by a camera built into an in-vehicle device with adjustable posture,
    When the posture of the in-vehicle device is adjusted for use for the original purpose, the equipment in the vehicle interior captured by the camera is set as the first equipment, and the image acquired by the image acquisition unit is predetermined. An equipment detection unit that sets an equipment detection area at a position and detects the first equipment from the equipment detection area,
    When the first equipment is detected by the equipment detection unit, a monitoring area setting unit that sets an occupant monitoring area for the image based on the image position of the first equipment, and a monitoring area setting unit.
    An occupant monitoring device including an occupant monitoring unit that monitors an occupant using an image of the occupant monitoring area set by the monitoring area setting unit among the images acquired by the image acquisition unit.
  2. 　前記設備検出部が前記第１設備を検出できない場合、又は前記監視領域設定部が設定した前記乗員監視領域の少なくとも一部が前記画像からはみ出る場合の少なくとも一方であれば、前記車室内に設けられた警告機器に対して警告を発するよう指示する警告部を備えることを特徴とする請求項１記載の乗員監視装置。 If the equipment detection unit cannot detect the first equipment, or if at least a part of the occupant monitoring area set by the monitoring area setting unit protrudes from the image, it is provided in the vehicle interior. The occupant monitoring device according to claim 1, further comprising a warning unit for instructing the warning device to issue a warning.
  3. 　前記設備検出部は、前記第１設備を検出できない場合、前記車室内の予め定められた第２設備を前記画像から検出し、前記第２設備の画像位置を基準にして前記第１設備の画像位置を推定することを特徴とする請求項１記載の乗員監視装置。 When the equipment detection unit cannot detect the first equipment, the equipment detection unit detects a predetermined second equipment in the vehicle interior from the image, and an image of the first equipment with reference to the image position of the second equipment. The occupant monitoring device according to claim 1, wherein the position is estimated.
  4. 　前記画像取得部は、姿勢調整可能な前記車載機器であるルームミラーに内蔵された前記カメラによって撮像された前記車室内の画像を取得し、
    　前記設備検出部は、前記画像取得部により取得された画像の予め定められた位置に前記設備検出領域を設定し、前記第１設備であるリアウインドウを前記設備検出領域から検出し、
    　前記監視領域設定部は、前記設備検出部により前記リアウインドウが検出された場合、前記リアウインドウの画像位置を基準にして、前記画像に対して前記乗員監視領域を設定し、
    　前記乗員監視部は、前記画像取得部により取得された画像のうちの前記監視領域設定部により設定された前記乗員監視領域の画像を用いて運転者を監視することを特徴とする請求項１記載の乗員監視装置。     The image acquisition unit acquires an image of the vehicle interior captured by the camera built in the rear-view mirror, which is an in-vehicle device whose posture can be adjusted.
    The equipment detection unit sets the equipment detection area at a predetermined position of the image acquired by the image acquisition unit, and detects the rear window, which is the first equipment, from the equipment detection area.
    When the rear window is detected by the equipment detection unit, the monitoring area setting unit sets the occupant monitoring area for the image with reference to the image position of the rear window.
    The first aspect of claim 1, wherein the occupant monitoring unit monitors the driver using an image of the occupant monitoring area set by the monitoring area setting unit among the images acquired by the image acquisition unit. Crew monitoring device.
  5. 　前記設備検出部は、前記リアウインドウを検出できない場合、左側のリアサイドウインドウ又は右側のリアサイドウインドウの少なくとも一方を前記画像から検出し、検出したリアサイドウインドウの画像位置を基準にして前記リアウインドウの画像位置を推定することを特徴とする請求項４記載の乗員監視装置。 When the rear window cannot be detected, the equipment detection unit detects at least one of the left rear side window and the right rear side window from the image, and the image position of the rear window is based on the detected image position of the rear side window. The occupant monitoring device according to claim 4, wherein the occupant monitoring device is characterized in that.
  6. 　カメラが内蔵された、姿勢調整可能な車載機器と、
    　請求項１記載の乗員監視装置とを備えることを特徴とする乗員監視システム。     In-vehicle device with built-in camera and adjustable posture,
    An occupant monitoring system including the occupant monitoring device according to claim 1.
  7. 　画像取得部が、姿勢調整可能な車載機器に内蔵されたカメラによって撮像された車室内の画像を取得し、
    　設備検出部が、前記車載機器の姿勢が本来の目的で使用するために調整された場合に前記カメラによって撮像される前記車室内の設備を第１設備とし、前記画像取得部により取得された画像の予め定められた位置に設備検出領域を設定して前記第１設備を前記設備検出領域から検出し、
    　監視領域設定部が、前記設備検出部により前記第１設備が検出された場合、前記第１設備の画像位置を基準にして、前記画像に対して乗員監視領域を設定し、
    　乗員監視部が、前記画像取得部により取得された画像のうちの前記監視領域設定部により設定された前記乗員監視領域の画像を用いて乗員を監視する乗員監視方法。     The image acquisition unit acquires the image of the vehicle interior captured by the camera built into the in-vehicle device with adjustable posture.
    When the equipment detection unit adjusts the posture of the in-vehicle device to be used for the original purpose, the equipment in the vehicle interior captured by the camera is set as the first equipment, and the image acquired by the image acquisition unit. The equipment detection area is set at a predetermined position of the above, and the first equipment is detected from the equipment detection area.
    When the first equipment is detected by the equipment detection unit, the monitoring area setting unit sets the occupant monitoring area for the image with reference to the image position of the first equipment.
    A occupant monitoring method in which the occupant monitoring unit monitors an occupant using an image of the occupant monitoring area set by the monitoring area setting unit among the images acquired by the image acquisition unit.
PCT/JP2019/046160 2019-11-26 2019-11-26 Occupant monitoring device, occupant monitoring system, and occupant monitoring method WO2021106073A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2021560799A JP7204941B2 (en) 2019-11-26 2019-11-26 Occupant monitoring device, occupant monitoring system, and occupant monitoring method
PCT/JP2019/046160 WO2021106073A1 (en) 2019-11-26 2019-11-26 Occupant monitoring device, occupant monitoring system, and occupant monitoring method

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2019/046160 WO2021106073A1 (en) 2019-11-26 2019-11-26 Occupant monitoring device, occupant monitoring system, and occupant monitoring method

Publications (1)

Publication Number Publication Date
WO2021106073A1 true WO2021106073A1 (en) 2021-06-03

Family

ID=76129399

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2019/046160 WO2021106073A1 (en) 2019-11-26 2019-11-26 Occupant monitoring device, occupant monitoring system, and occupant monitoring method

Country Status (2)

Country Link
JP (1) JP7204941B2 (en)
WO (1) WO2021106073A1 (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008102752A (en) * 2006-10-19 2008-05-01 Sankyo Kk Monitoring system
JP2010100088A (en) * 2008-10-21 2010-05-06 Toyota Motor Corp Drive assist system
JP2012044404A (en) * 2010-08-18 2012-03-01 Toyota Motor Corp Cabin monitoring device
JP2012230522A (en) * 2011-04-26 2012-11-22 Nissan Motor Co Ltd Image processing apparatus and image processing method

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008102752A (en) * 2006-10-19 2008-05-01 Sankyo Kk Monitoring system
JP2010100088A (en) * 2008-10-21 2010-05-06 Toyota Motor Corp Drive assist system
JP2012044404A (en) * 2010-08-18 2012-03-01 Toyota Motor Corp Cabin monitoring device
JP2012230522A (en) * 2011-04-26 2012-11-22 Nissan Motor Co Ltd Image processing apparatus and image processing method

Also Published As

Publication number Publication date
JPWO2021106073A1 (en) 2021-06-03
JP7204941B2 (en) 2023-01-16

Similar Documents

Publication Publication Date Title
US9139135B2 (en) System and method that minimizes hazards of blind spots while driving
KR101854253B1 (en) Vehicle mirror adjustment
CN110300673B (en) Vehicle periphery monitoring device
JP6384322B2 (en) Vehicle display device
US20120026012A1 (en) Driver vision support system and vehicle including the system
US20100201507A1 (en) Dual-mode vision system for vehicle safety
JP2008269496A (en) Occupant information detection system, occupant restraint system and vehicle
JP2008129948A (en) Occupant detection device, actuator control system, seat belt system, vehicle
US20180335633A1 (en) Viewing direction detector and viewing direction detection system
EP3248839B1 (en) Vehicular display device
JP6393653B2 (en) Vehicle visual recognition device
JP6575404B2 (en) VEHICLE DISPLAY CONTROL DEVICE, VEHICLE DISPLAY SYSTEM, VEHICLE DISPLAY CONTROL METHOD, AND PROGRAM
WO2020031413A1 (en) Recognition processing device, recognition processing method and recognition processing program
JP2020157938A (en) On-vehicle monitoring control device
WO2016042713A1 (en) Electronic mirror device
JP2008120142A (en) Information display system for automobile
KR20180078970A (en) Camera Arranging Structure for Monitoring Surrounding Area of Construction Equipment
WO2021106073A1 (en) Occupant monitoring device, occupant monitoring system, and occupant monitoring method
JP2019110518A (en) Imaging device and imaging system
JP2007253821A (en) Occupant imaging device
JP4840638B2 (en) Vehicle occupant monitoring device
KR102457629B1 (en) Apparatus for real-time condition check of vehicle back seat
JP2020536799A (en) How to operate the safety system for the car seat system and the safety system for the car seat system
JP5119721B2 (en) Peripheral visibility check device for vehicles
JP2017061216A (en) On-board imaging system, vehicle and imaging method

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19954322

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2021560799

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19954322

Country of ref document: EP

Kind code of ref document: A1