WO2022172400A1

WO2022172400A1 - Vehicular monitoring device and vehicular monitoring system

Info

Publication number: WO2022172400A1
Application number: PCT/JP2021/005263
Authority: WO
Inventors: 太郎熊谷
Original assignee: 三菱電機株式会社
Priority date: 2021-02-12
Filing date: 2021-02-12
Publication date: 2022-08-18
Also published as: JP7446492B2; JPWO2022172400A1; DE112021006342T5

Abstract

The present invention provides a vehicular monitoring device with increased accuracy in detecting a target person.　The vehicular monitoring device detects a target person who is an occupant inside the vehicle or a person outside the vehicle, and in the case where a detection result regarding the presence or absence of the target person obtained from sensor information acquired by a sensor different from an imaging device differs from a detection result regarding the presence or absence of the target person obtained from a captured image acquired by the imaging device, the detection result regarding the presence or absence of the target person obtained from the captured image is used to specify whether the target person is present or absent, thereby be capable of increasing the accuracy in detecting the target person.

Description

Vehicle monitoring device and vehicle monitoring system

The present disclosure relates to a vehicle monitoring device and a vehicle monitoring system that detect an occupant inside a vehicle or a person outside the vehicle.

Detecting occupants inside the vehicle or people outside the vehicle, control processing such as airbags and seat belt reminders according to the detection results such as the position of the occupants, notification of the presence of a suspicious person according to the detection results of people outside the vehicle, etc. technology has been developed to Conventionally, regarding the detection of occupants in a vehicle, an imaging device is provided so that the imaging range includes the front and rear seats of the vehicle. In this case, it is detected in which seat the occupant is located (see, for example, Patent Document 1).

Japanese Patent Application Laid-Open No. 2020-50090

When trying to detect such a passenger inside the vehicle or a person outside the vehicle, that is, a target person, it is conceivable to use a sensor different from the imaging device, such as a radio wave sensor or a seating sensor. However, for example, if an attempt is made to detect an occupant in a rear seat using a seat sensor, depending on the posture of the occupant sitting on the seat, the weight of the occupant sitting on the seat cannot be measured correctly. There is a problem that even if it is used, it may not be sufficiently detected.

The present disclosure has been made to solve the above-described problems, and by using a detection result obtained from an imaging device and a detection result obtained from a sensor different from the imaging device, The purpose is to improve detection accuracy.

A first vehicle monitoring device according to the present disclosure includes a first information acquisition unit that acquires sensor information from a sensor whose detection range includes at least the rear seats of a vehicle; a second information acquisition unit that acquires a captured image from the device; an occupant detection unit that detects an occupant in a rear seat from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit; The occupant detection unit obtains from the captured image when the detection result regarding the presence or absence of the rear seat occupant obtained from the sensor information and the detection result regarding the presence or absence of the rear seat occupant obtained from the captured image are different. The presence/absence of a rear-seat passenger is specified using the detection result regarding the presence/absence of a rear-seat passenger.

In addition, a first vehicle monitoring system according to the present disclosure includes a sensor mounted in a vehicle and including at least the rear seats of the vehicle in its detection range, and an imaging sensor mounted in the vehicle and including the front and rear seats of the vehicle in its imaging range. A device, a first information acquisition unit that acquires sensor information from a sensor, a second information acquisition unit that acquires a captured image from an imaging device, and the first information acquisition unit and the second information acquisition unit respectively acquired, and an occupant detection unit that detects an occupant in the rear seat from the sensor information and the captured image. If the detection result regarding the presence/absence of the passenger in the rear seat is different, the presence/absence of the passenger in the rear seat is specified using the detection result regarding the presence/absence of the passenger in the rear seat obtained from the captured image.

Further, a second vehicle monitoring device according to the present disclosure includes a first information acquisition unit that acquires sensor information from a sensor whose detection range includes the outside of the vehicle, and an imaging device that includes the outside of the vehicle in its imaging range. A second information acquisition unit that acquires an image, and a vehicle exterior detection unit that detects a person outside the vehicle from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit, If the detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information differs from the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image, the vehicle exterior detection unit The existence or non-existence of a person outside the vehicle is specified using the detection result regarding the existence or non-existence of the person outside the vehicle.

Further, a second vehicle monitoring system according to the present disclosure includes a sensor mounted on a vehicle and including the exterior of the vehicle in its detection range, an imaging device mounted on the vehicle and including the exterior of the vehicle in its imaging range, and a sensor, A first information acquisition unit that acquires sensor information, a second information acquisition unit that acquires a captured image from an imaging device, and the sensor information and the captured image acquired by the first information acquisition unit and the second information acquisition unit, respectively and a vehicle exterior detection unit that detects a person outside the vehicle, and the vehicle exterior detection unit detects a detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information, and a person outside the vehicle obtained from the captured image. If the detection result regarding the presence or absence of a person is different, the presence or absence of the person outside the vehicle is specified using the detection result regarding the presence or absence of the person outside the vehicle obtained from the captured image.

According to the present disclosure, an occupant inside the vehicle or a person outside the vehicle, that is, a target person is detected using an imaging device and a sensor different from the imaging device, so the detection accuracy of the target person can be improved.

1 is a block diagram showing a configuration example of a vehicle monitoring system according to Embodiment 1; FIG. 4A and 4B are explanatory diagrams showing a detection range of a sensor and an imaging range of an imaging device according to Embodiment 1; FIG. FIG. 5 is an explanatory diagram showing an example of occupant detection using a captured image according to Embodiment 1; 4 is a flowchart showing an operation example of the vehicle monitoring device according to Embodiment 1; 1 is a diagram showing an example hardware configuration of a vehicle monitoring device according to Embodiment 1; FIG. FIG. 11 is an explanatory diagram showing an example of occupant detection using a captured image according to Embodiment 2; FIG. 11 is an explanatory diagram showing an example of occupant detection using a captured image according to Embodiment 2; 9 is a flowchart showing an operation example of the vehicle monitoring device according to Embodiment 2; FIG. 11 is a block diagram showing a configuration example of a vehicle monitoring system according to Embodiment 3; FIG. 11 is an explanatory diagram showing an example of detection of a person outside the vehicle by the vehicle monitoring device according to Embodiment 3; 10 is a flow chart showing an operation example of the vehicle monitoring device according to Embodiment 3; FIG. 11 is a diagram showing a hardware configuration example of a vehicle monitoring device according to Embodiment 3;

Embodiments will be described below based on the drawings.

Embodiment 1.
FIG. 1 is a block diagram showing a configuration example of a vehicle monitoring system 100 according to Embodiment 1. As shown in FIG. The vehicle monitoring system 100 includes a vehicle monitoring device 10, a sensor 20, and an imaging device 30. The vehicle monitoring device 10, the sensor 20, and the imaging device 30 are each mounted on a vehicle. In addition, the vehicle monitoring device 10 is connected to a vehicle-side control device 200 that controls in-vehicle devices such as air conditioners, audio devices, navigation devices, and notification units, and the engine, etc., in the vehicle on which the vehicle monitoring device 10 is mounted. there is

A target person in detection by the vehicle monitoring system 100 according to the present embodiment is a passenger in the rear seat. The vehicle monitoring system 100 may detect not only rear seat passengers, but also front seat passengers as target persons. FIG. 2 is an explanatory diagram showing the detection range of the sensor 20 and the imaging range of the imaging device 30 according to the first embodiment. 2A is a side view of the interior of a vehicle 50 equipped with the vehicle monitoring device 10, and FIG. 2B is a top view of the interior of the vehicle 50 equipped with the vehicle monitoring device 10. FIG. The sensor 20 is a sensor 20 mounted in a vehicle and provided so as to be able to detect at least rear seat passengers among the passengers in the vehicle. The sensor 20 is a sensor 20 different from the imaging device 30, such as a radio wave sensor 21, a weight sensor (seating sensor), a voice sensor, an ultrasonic sensor, etc., which can detect a passenger, for example. Moreover, the sensor 20 is any one of these or a combination of a plurality of them.

FIG. 2A shows an example in which the sensor 20 is a radio wave sensor 21, and the detection range of the radio wave sensor 21 is indicated by area A. The radio wave sensor 21 is, for example, a sensor that transmits millimeter waves and receives reflected waves reflected by a moving object. In the example of FIG. 2A, the sensor 20 is provided on the ceiling of the compartment of the vehicle 50 or the like so that at least the rear seats are included in the detection range. In addition, FIG. 2B shows an example in which an occupant 58 is seated in the seat 53 on the left side of the rear seat. Here, the seat on the left side of the rear seat refers to the seat 53 on the left side with respect to the front of the vehicle among the rear seats, and the seat on the right side of the rear seat refers to the seat 54 on the right side with respect to the front of the vehicle among the rear seats. , and the seat in the middle of the rear seat refers to the seat 55 provided between the left seat and the right seat of the rear seats. Hereinafter, for the sake of explanation, when the term "rear seat" is simply used without any particular notice, it refers to the seat 53 on the left side of the rear seat, the seat 54 on the right side of the rear seat, and the seat 55 in the center of the rear seat. Note that the number of rear seats does not have to be three as in the example of FIG. 2B, and the number of rear seats is arbitrary.

The imaging device 30 is composed of, for example, a wide-angle camera, an infrared camera, etc., and images the inside of the vehicle 50 . Further, the imaging device 30 may be a TOF (Time-of-flight) camera capable of capturing an image reflecting the distance between the imaging device 30 and the subject. The imaging device 30 captures an image of the interior of the vehicle at intervals of, for example, 30 to 60 fps (frames per second), and outputs the captured image to the second information acquisition section 2 of the information acquisition section 11 of the vehicle monitoring device 10 . An image captured by the imaging device 30 is hereinafter referred to as a captured image.

In the example of FIG. 2A, the area B indicates the imaging range of the imaging device 30 . The imaging device 30 includes one or more units, an overhead console, an instrument panel, and a steering wheel so as to include the front seats of the driver's seat 51 and the front passenger seat 52 and at least one of the rear seats in the imaging range. They are installed in columns, room mirrors, etc. Hereinafter, passengers in the vehicle, such as

passengers

56, 57, and 58, who are to be detected by the vehicle monitoring device 10, are collectively referred to as "passengers". That is, the occupants include the driver.

Returning to FIG. 1, the vehicle monitoring device 10 will be described. The vehicle monitoring device 10 includes an information acquisition unit 11 that acquires sensor information and captured images from the sensor 20 and the imaging device 30, respectively, and an occupant detection unit 12 that detects passengers in the rear seats from the sensor information and captured images. .

As shown in FIG. 1, the information acquisition section 11 has a first information acquisition section 1 and a second information acquisition section 2 . The first information acquisition section 1 of the information acquisition section 11 is connected to the sensor 20 and acquires sensor information from the sensor 20 . Here, the sensor information is information regarding the presence or absence of passengers in the vehicle. For example, when the sensor 20 is the radio wave sensor 21, the sensor information includes distance data indicating the distance between the radio wave sensor 21 and the detection target, angle data indicating the angle of the detection target with respect to the radio wave sensor 21, and the like. For example, when the sensor 20 is a seat sensor, the sensor information includes seat data indicating information on the seat provided with the seat sensor, weight data detected by the seat sensor, and the like. Further, for example, when the sensor 20 is a voice sensor, the sensor information is voice data or the like indicating the volume, direction of arrival, etc. of the voice uttered by the passenger. In addition, the sensor information is any one of these or a combination of a plurality of them.

On the other hand, the second information acquisition section 2 of the information acquisition section 11 is connected to the imaging device 30 and acquires a captured image from the imaging device 30 . Although the first information acquisition unit 1 and the second information acquisition unit 2 are shown separately in the example of FIG. You can go with one configuration. The information acquisition unit 11 then outputs the acquired sensor information and captured image to the occupant detection unit 12, which will be described later.

The information acquisition section 11 also has a vehicle information acquisition section 3 connected to the vehicle-side control device 200 . The vehicle information acquisition unit 3 acquires vehicle information such as signals related to starting and stopping of the vehicle from the vehicle-side control device 200 . Then, the vehicle information acquisition unit 3 instructs the first information acquisition unit 1 and the second information acquisition unit 2 to start acquiring the sensor information and the captured image using the vehicle information acquired from the vehicle-side control device 200. A signal or a signal indicating that the acquisition of the sensor information and the captured image is to be terminated is output.

For example, the vehicle information acquisition unit 3 receives information from the vehicle-side control device 200 that the doors are unlocked, the doors are opened, the ignition is turned on, the human sensor is turned on, the shift lever is moved to the drive position, and the vehicle speed is 0 km/h. , the navigation device has started guidance, or the vehicle has left home, the sensor information and the captured image are sent to the first information acquisition unit 1 and the second information acquisition unit 2 output a signal to start acquisition of On the other hand, for example, the vehicle information acquisition unit 3 receives information from the vehicle-side control device 200 that the ignition is turned off, the motion sensor is turned off, the shift lever is moved to the parking position, the navigation device has finished guidance, and the vehicle is heading home. When any of the signals such as return is acquired, a signal to the effect that the acquisition of the sensor information and the captured image is to be terminated is output to the first information acquisition section 1 and the second information acquisition section 2 .

The occupant detection unit 12 of the vehicle monitoring device 10 will be explained. The occupant detection unit 12 includes a first detection unit 4 that detects an occupant using sensor information, a second detection unit 5 that detects an occupant using a captured image, and the first detection unit 4 and the second detection unit 5. A presence/absence determination unit 6 that determines whether or not an occupant is present in the rear seat using the detection result. In addition, as shown in FIG. 1, the first detection unit 4 and the second detection unit 5 of the occupant detection unit 12 are connected to the first information acquisition unit 1 and the second information acquisition unit 2 of the information acquisition unit 11, respectively. there is

The process of detecting passengers in the rear seats by the passenger detection unit 12 will be explained. In the following description, an example in which the first detection unit 4 obtains distance data and angle data obtained from the radio wave sensor 21 as sensor information and detects an occupant in the rear seat will be described.

First, the first detection unit 4 acquires distance data and angle data as sensor information from the first information acquisition unit 1 of the information acquisition unit 11, and determines the size of the detection target and the detection target such as a person or an animal. Calculate the existing range, etc. For example, if the first detection unit 4 finds that the detection target exists on the left side of the rear seat from the calculated size and range of the detection target, the detection result indicates that an occupant has been detected on the left side of the rear seat. is derived, and the detection result is output to a storage unit (not shown) of the vehicle monitoring device 10 .

For example, if it is found that the object to be detected exists on the right side of the rear seat from the calculated size and range of the object to be detected, the first detection unit 4 determines that the occupant has been detected on the right side of the rear seat. The detection result is output to the storage unit. Similarly, for example, if the first detection unit 4 finds that the detection target exists in the center of the rear seat from the calculated size and range of the detection target, the passenger is detected in the center of the rear seat. output the detection result. The occupant detection processing by the first detection unit 4 is not limited to the example described above, and various known algorithms can be used. In the above example, the detection result of the first detection unit 4 is stored in the storage unit, but the first detection unit 4 may output the detection result to the existence determination unit 6 .

Here, the existence of an occupant includes not only the state in which the occupant is strictly sitting on the seat, but also the state in which the occupant is present at the feet of the seat. A small-sized occupant such as an infant may be at the foot of the seat. For example, if the sensor information obtained from the radio wave sensor 21 is used to detect the occupant, it is possible to detect the occupant at the foot of the seat. It's for. Hereinafter, the occupant detection process using the sensor information by the first detection unit 4 will be referred to as the first occupant detection process for the sake of explanation. A detection result of the first occupant detection process by the first detection unit 4 is called a first detection result.

By the way, when using sensor information to detect rear seat occupants, the reliability of the detection results may become an issue. For example, if the sensor 20 is the radio wave sensor 21, the resolution of the distance data and angle data obtained from the radio wave sensor 21 cannot be guaranteed, and multiple occupants sitting in different seats are detected in a row. In some cases, it may not be possible to accurately determine which seat the occupant is in. Further, for example, if the sensor 20 is a seat sensor, depending on the posture of the occupant sitting on the seat, the weight of the occupant sitting on the seat cannot be measured correctly, and it is possible to determine whether the object placed on the seat is a person. It may not be possible to determine whether

On the other hand, when the imaging device 30 is used to detect the occupant, a more reliable detection result can be obtained than when the sensor 20 different from the imaging device 30 is used to detect the occupant. Therefore, regarding the detection of the occupant in the rear seat, when the occupant is detected from the captured image, the occupant detection accuracy can be improved by using the detection result of the occupant detection processing using the captured image.

The occupant detection process using captured images will be explained. FIG. 3 is an explanatory diagram showing an example of occupant detection using captured images according to the first embodiment. FIG. 3 shows captured images acquired from the imaging device 30, and an occupant 56 sitting in the driver's seat 51, an occupant 57 sitting in the passenger's seat 52, and an occupant 57 sitting in the passenger's seat 52 are detected by the occupant detection processing of the second detection unit 5, which will be described later. An example is shown in which an occupant 58 seated on the seat 53 on the left side of the seat and an occupant 60 seated on the seat 55 in the center of the rear seat are respectively detected.

An example of occupant detection processing by the second detection unit 5 will be described. First, the second detection unit 5 analyzes the captured image and detects the face of the passenger in the captured image. Then, the second detection unit 5 acquires an area where the occupant's face is detected (the area indicated by the dashed line in FIG. 3; hereinafter referred to as a face area) and occupant feature information in the face area. Here, the characteristic information of the occupant is, for example, the contrast ratio of the eyes, nose, mouth, and cheeks after normalizing the size of the face. Note that the second detection unit 5 may determine whether or not the face detection is successful using the occupant's characteristic information. In this case, for example, if the luminance distribution of the face region is found to be face-like from the contrast ratio in the face region, the second detection unit 5 determines that face detection has succeeded, and the luminance distribution does not look like a face. , it is determined that the face detection has failed.

After detecting the occupant's face, the second detection unit 5 acquires the coordinates of the face area surrounding the occupant's face, such as a rectangle that contacts the contour of the occupant's face, from the captured image. Here, the coordinates relating to the face area are, for example, the coordinates of each vertex, center, etc. of the rectangle when the face area is rectangular. Further, the second detection unit 5 calculates dimensions such as the width, height, and area of the face area from the coordinates of the face area.

Then, the second detection unit 5 identifies the seat where the occupant is present from the acquired coordinates, width, height, area, etc. of the face region, and detects information regarding the presence or absence of the occupant and the seat where the occupant is present. As a result, it is output to the storage unit of the vehicle monitoring device 10 . The occupant detection process by the second detection unit 5 is not limited to the above example, and various known algorithms can be used.

Also, in the above example, the detection result of the second detection unit 5 is stored in the storage unit, but the second detection unit 5 may output the detection result to the existence determination unit 6. Hereinafter, for the sake of explanation, the occupant detection process using the captured image by the second detection unit 5 will be referred to as the second occupant detection process. A detection result of the second occupant detection process by the second detection unit 5 is called a second detection result.

Next, the presence/absence determination for determining whether or not a passenger is present will be described. The presence/absence determination unit 6 of the present embodiment determines that there is a passenger in the rear seat if the passenger in the rear seat can be detected from the captured image even if the passenger in the rear seat is not detected from the sensor information. In this way, if the detection result regarding the presence or absence of a rear seat passenger obtained from the sensor information differs from the detection result regarding the presence or absence of a rear seat passenger obtained from the captured image, the passenger detection unit 12 The presence or absence of the passenger in the rear seat is specified using the obtained detection result regarding the presence or absence of the passenger in the rear seat.

The operation of the vehicle monitoring device 10 will be described below. FIG. 4 is a flow chart showing an operation example of the vehicle monitoring device 10 according to the first embodiment. Although the flowchart of FIG. 4 does not show a process for terminating the operation of the vehicle monitoring device 10, the vehicle monitoring device 10, for example, the vehicle information acquisition unit 3 receives the vehicle 50 from the vehicle-side control device 200. When the signal indicating that the engine has stopped is acquired, the operations of the first occupant detection process and the second occupant detection process are terminated. In the explanation below, even if multiple rear seats are provided, the rear seats are collectively referred to as rear seats. shall be processed for each seat.

First, after the operation of the vehicle monitoring device 10 is started, the first information acquiring section 1 of the vehicle monitoring device 10 acquires sensor information from the sensor 20 (ST101). Next, the first detection section 4 performs a first occupant detection process using sensor information (ST102). Here, in the process of ST102, the first detection unit 4 performs the first occupant detection process for each of the rear seats, that is, each of the rear seats provided in the vehicle. It should be noted that the determination of whether or not an occupant has been detected for each of the rear seats by the first detection unit 4 is performed by, for example, the distance data acquired as sensor information, the size and presence of the detection target calculated from the angle data, and the like. The determination may be made based on which of the rear seats the detection target is located in, based on the range of detection.

Then, the first detection unit 4 outputs to the storage unit of the vehicle monitoring device 10 the detection result indicating whether or not the passenger in the rear seat has been detected. When an occupant is detected in the rear seat (ST102; YES), the first detection unit 4 stores the detection result indicating that an occupant in the rear seat has been detected in the storage unit of the vehicle monitoring device 10 as the first detection result. output, and the rear seat first flag is turned ON (ST103). On the other hand, when the passenger in the rear seat is not detected (ST102; NO), the first detection unit 4 sets the detection result indicating that the passenger in the rear seat is not detected as the first detection result to the vehicle monitoring device 10. and the rear seat first flag is turned OFF (ST104).

Here, the rear seat first flag stored in the storage unit indicates whether or not an occupant is detected in the rear seat by the first occupant detection process. That is, when the first detection unit 4 outputs the first detection result indicating that the passenger in the rear seat has been detected, the rear seat first flag is turned ON, and the first detection unit 4 detects the passenger in the rear seat. If the first detection result indicating that there is no seat is output, the rear seat first flag is turned OFF.

It should be noted that a plurality or a single number of rear seat first flags are provided according to the number of rear seats. For example, if the number of rear seats is three and the vehicle is provided with a seat on the left side of the rear seat, a seat on the right side of the rear seat, and a seat in the center of the rear seat, the rear seat first flag The first flag, the rear seat right side first flag, and the rear seat center first flag.

Next, the second information acquisition section 2 of the vehicle monitoring device 10 acquires the captured image from the imaging device 30 (ST105). The second detection unit 5 uses the captured image to perform a second occupant detection process (ST106). Here, in the process of ST106, the second detection unit 5 detects the occupant of each rear seat, that is, each of the rear seats provided in the vehicle. It should be noted that the determination of whether or not an occupant has been detected for each of the rear seats by the second detection unit 5 is performed by, for example, characteristic information detected from the captured image, the coordinates of the face region, the width, the height, and the The determination may be made on the basis of which of the rear seats the detection target is located in, based on the area or the like.

Then, the second detection unit 5 outputs the detection result as to whether or not an occupant in the rear seat has been detected to the storage unit of the vehicle monitoring device 10 . When an occupant is detected in the rear seat (ST106; YES), the second detection unit 5 stores the detection result indicating that an occupant in the rear seat has been detected in the storage unit of the vehicle monitoring device 10 as a second detection result. output, and the rear seat second flag is turned ON (ST107). On the other hand, when the passenger in the rear seat is not detected (ST106; NO), the second detection unit 5 sets the detection result indicating that the passenger in the rear seat is not detected as the second detection result to the vehicle monitoring device 10. , and the rear seat second flag is turned OFF (ST108).

Here, the rear seat second flag stored in the storage unit indicates whether or not a passenger is detected in the rear seat by the second passenger detection process. That is, when the second detection unit 5 outputs a second detection result indicating that a rear seat passenger has been detected, the rear seat second flag is turned ON, and the second detection unit 5 detects a rear seat passenger. If the second detection result indicating that there is no seat is output, the rear seat second flag is turned OFF.

It should be noted that a plurality or a single number of rear seat second flags are provided according to the number of rear seats. For example, if the number of rear seats is three, and the vehicle is provided with a seat on the left side of the rear seat, a seat on the right side of the rear seat, and a seat in the center of the rear seat, the second rear seat flag The second flag, the rear seat right side second flag, and the rear seat center second flag.

Next, the presence/absence determination unit 6 of the vehicle monitoring device 10 acquires the first detection result from the first detection unit 4 and acquires the second detection result from the second detection unit 5 . Then, the presence/absence determination unit 6 uses the first detection result and the second detection result to determine whether or not an occupant is present in the rear seat.

The presence/absence determining unit 6 refers to the rear seat first flag as the first detection result, and determines whether or not the rear seat first flag is ON (ST109), and the rear seat occupant is detected from the sensor information. Check whether or not Then, if the rear seat first flag is ON (ST109; YES), that is, if it is confirmed from the sensor information that an occupant in the rear seat is detected, the presence/absence determination unit 6 determines that there is an occupant in the rear seat. It determines and outputs the determination result to the vehicle-side control device 200 (ST110). The determination result by the presence/absence determination unit 6 is output to, for example, a notification control unit (not shown) that controls a seatbelt reminder of the vehicle-side control device 200, and the notification control unit detects that a passenger is present in the rear seat. When it receives the determination result that it should, the function of the seat belt reminder in the rear seat is turned on.

On the other hand, if the rear seat first flag is OFF (ST109; NO), i.e., if it is confirmed from the sensor information that no occupant is detected in the rear seat, the presence/absence determination unit 6 performs ST111 to be described next. Proceed to processing.

The presence/absence determining unit 6 refers to the rear seat second flag as the second detection result, and determines whether or not the rear seat second flag is ON (ST111), and the rear seat occupant is detected from the captured image. Check whether or not Then, if the rear seat second flag is ON (ST111; YES), that is, if it is confirmed that a rear seat occupant is detected from the captured image, the presence/absence determination unit 6 determines that a rear seat occupant is present. It determines and outputs the determination result to the vehicle-side control device 200 (ST110).

On the other hand, if the rear seat second flag is OFF (ST111; NO), that is, if the presence/absence determination unit 6 confirms that the passenger in the rear seat is not detected from the captured image, the passenger is present in the rear seat. It determines not to do so, and outputs the determination result to vehicle-side control device 200 (ST112).

In this way, the occupant detection unit 12 obtains a detection result indicating that no occupant was detected in the rear seat from the sensor information, and obtained a detection result indicating that an occupant in the rear seat was detected from the captured image. In this case, it is specified that there is an occupant in the rear seat. As a result, even if an occupant in the rear seat is not actually detected from the sensor information, even if the occupant in the rear seat is not detected from the captured image, when the occupant in the rear seat is detected from the captured image, Since it is specified that a passenger is present in the rear seat using the detection result obtained from the above, it is possible to prevent failure to detect the passenger in the rear seat and improve the detection accuracy of the passenger in the rear seat, that is, the target person.

It should be noted that the second detection unit 5 may perform the second occupant detection process on the rear seat for which the first detection result indicates that no occupant has been detected. . In this way, detection omissions can be suppressed, and the processing load can be reduced because the second occupant detection processing is performed for the seats where there is a possibility of detection omissions. Further, in the flowchart of FIG. 4, the vehicle monitoring device 10 performs the processing of ST101 to ST104 and then performs the processing of ST105 to ST108, but the flowchart of FIG. 4 is an example. The processing of ST101 to ST104 and the processing of ST105 to ST108 may be performed in parallel.

Next, the hardware configuration that implements the functions of the vehicle monitoring device 10 will be described. FIG. 5 is a diagram showing a hardware configuration example of the vehicle monitoring device 10 according to the first embodiment. First information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first detection unit 4, second detection unit 5, presence/absence determination unit 6, occupant detection unit in vehicle monitoring device 10 12, and the functions of the storage unit are realized by the processing circuit. That is, of the vehicle monitoring device 10, the first information acquisition unit 1, the second information acquisition unit 2, the vehicle information acquisition unit 3, the information acquisition unit 11, the first detection unit 4, the second detection unit 5, the presence/absence determination unit 6, The occupant detection unit 12 and the storage unit may be a processing circuit 10a that is dedicated hardware as shown in FIG. 5A, or may be a processor that executes a program stored in a memory 10c as shown in FIG. 5B. 10b.

As shown in FIG. 5A, first information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first detection unit 4, second detection unit 5, presence/absence determination unit 6, occupant When the detection unit 12 and the storage unit are dedicated hardware, the processing circuit 10a is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-programmable Gate Array), or a combination of these. First information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first detection unit 4, second detection unit 5, presence/absence determination unit 6, occupant detection unit 12, and storage unit Each function of each section may be realized by a processing circuit, or the functions of each section may be collectively realized by one processing circuit.

As shown in FIG. 5B, first information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first detection unit 4, second detection unit 5, presence/absence determination unit 6, occupant When the detection unit 12 and the storage unit are the processor 10b, the functions of each unit are realized by software, firmware, or a combination of software and firmware. Software or firmware is written as a program and stored in the memory 10c. The processor 10b reads and executes the programs stored in the memory 10c to obtain the first information acquisition unit 1, the second information acquisition unit 2, the vehicle information acquisition unit 3, the information acquisition unit 11, the first detection unit 4, It implements the functions of the second detection unit 5, the presence/absence determination unit 6, the occupant detection unit 12, and the storage unit. That is, the first information acquisition unit 1, the second information acquisition unit 2, the vehicle information acquisition unit 3, the information acquisition unit 11, the first detection unit 4, the second detection unit 5, the existence determination unit 6, the occupant detection unit 12, and The storage unit comprises a memory 10c for storing a program which, when executed by the processor 10b, results in the execution of the steps shown in FIG. Further, these programs include a first information acquisition unit 1, a second information acquisition unit 2, a vehicle information acquisition unit 3, an information acquisition unit 11, a first detection unit 4, a second detection unit 5, a presence/absence determination unit 6, an occupant It can also be said that a computer executes the procedures or methods of the detection unit 12 and the storage unit.

Here, the processor 10b is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic device, a processor, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor). The memory 10c may be non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), etc. However, it may be a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a mini disk, a CD (Compact Disc) or a DVD (Digital Versatile Disc).

In addition, the first information acquisition unit 1, the second information acquisition unit 2, the vehicle information acquisition unit 3, the information acquisition unit 11, the first detection unit 4, the second detection unit 5, the presence/absence determination unit 6, the occupant detection unit 12, and Each function of the storage unit may be partly realized by dedicated hardware and partly realized by software or firmware. In this way, the processing circuit 10a in the vehicle monitoring device 10 can realize each function described above by hardware, software, firmware, or a combination thereof. Also, a first information acquisition unit 1, a second information acquisition unit 2, a vehicle information acquisition unit 3, an information acquisition unit 11, a first detection unit 4, a second detection unit 5, a presence/absence determination unit 6, an occupant detection unit 12, and At least part of the functions of the storage unit may be executed by an external server.

In this manner, the vehicle monitoring device 10 includes the first information acquisition unit 1 that acquires sensor information from the sensor 20 whose detection range includes the rear seats of the vehicle, and the imaging device that includes the front and rear seats of the vehicle as its imaging range. From 30, a second information acquisition unit 2 that acquires a captured image, and a passenger detection that detects a passenger in a rear seat from the sensor information and the captured image respectively acquired by the first information acquisition unit 1 and the second information acquisition unit 2. and the occupant detection unit 12, when the detection result regarding the presence or absence of the occupant in the rear seat obtained from the sensor information and the detection result regarding the presence or absence of the occupant in the rear seat obtained from the captured image are different, If the presence or absence of the rear seat occupant is specified using the detection result regarding the presence or absence of the rear seat occupant obtained from the captured image, the rear seat occupant, that is, the target, can be prevented from failing to detect the rear seat occupant. People detection accuracy can be improved.

Embodiment 2.
A vehicle monitoring apparatus 10 according to Embodiment 2 includes, as in Embodiment 1, a first information acquisition unit 1 that acquires sensor information, a second information acquisition unit 2 that acquires a captured image, and a first information acquisition unit 2 that acquires a captured image. An occupant detection unit 12 that detects an occupant in a rear seat from the sensor information and captured image acquired by the unit 1 and the second information acquisition unit 2, respectively. In the present embodiment, when the second detection unit 5 does not detect an occupant in the rear seat from the captured image, the captured image is used to determine whether or not there is an occupant in the rear seat. The processing operation of the vehicle monitoring device 10 is different. The same reference numerals are given to the same components as in the first embodiment, and the description thereof is omitted. Moreover, the target person in the detection of the vehicle monitoring system 100 according to the present embodiment is the passenger in the rear seat. It should be noted that the vehicle monitoring system 100 may detect not only rear-seat passengers, but also front-seat passengers as target persons.

In the first occupant detection process using the sensor 20 different from the imaging device 30, for example, the sensor 20 is the radio wave sensor 21, and if the radio wave sensor 21 detects that the reflected wave is reflected from the moving object present on the seat. For example, it is determined that the detection target exists on the seat. In such a case, even though the moving object present on the seat is, for example, water in a PET bottle and an object other than the occupant, it is possible to determine if the occupant is on the seat based on the reflected wave from the PET bottle. Existence may be falsely detected. Further, for example, when the sensor 20 is a seat sensor, even if the seat sensor can correctly measure the weight of the object on the seat, the object on the seat may be an object other than the occupant depending on the weight of the object on the seat. In spite of this, there is a risk of erroneous detection when an occupant is present in the seat.

As described above, when the sensor 20 different from the imaging device 30 detects an occupant, there is a possibility that the presence of an occupant in the seat may be erroneously detected even though no occupant is actually present in the seat. Therefore, it cannot be said that the occupant detection accuracy is sufficient.

Therefore, in the vehicle monitoring device 10 of the present embodiment, even if the first detection unit 4 indicates the detection result that the passenger is present in the rear seat from the sensor information, the captured image of the second detection unit 5 is used. If the second occupant detection process reveals that no occupant is present in the rear seat, priority is given to the detection result using the captured image, and it is specified that no occupant is present in the rear seat. The occupant detection processing by the second detection unit 5 of the present embodiment will be described below.

In addition to detecting rear seat passengers from the captured image, the second detection unit 5 of the present embodiment detects whether a rear seat passenger is present from the captured image. determine whether or not

FIG. 6 is an explanatory diagram showing an example of occupant detection using captured images according to the second embodiment. In the captured image shown in FIG. 6,

crew members

56 and 57 are respectively seated in a driver's seat 51 and a front passenger seat 52 of a vehicle, and among the rear seats, a passenger 58 and a passenger 58 are seated in a seat 53 on the left side of the rear seat and a seat 54 on the right side of the rear seat. 59 shows an example in which the state in which the person 59 is seated is imaged. Further, in FIG. 6, a child seat 61 is placed on a seat 55 in the center of the rear seat, without an occupant sitting there.

In the example of FIG. 6, for example, when the sensor 20 is a seat sensor, the detection result of the first occupant detection process using the sensor information is obtained by the child seat 61 placed in the seat 55 in the center of the rear seat, as described above. , may indicate that an occupant has been detected in the middle rear seat 55 . On the other hand, it is clear from the captured image that no occupant is present in the seat 55 at the center of the rear seat, and that a child seat 61 is placed in the seat 55 at the center of the rear seat. Then, a detection result indicating that there is an object other than the passenger on the seat 55 at the center of the rear seat, that is, no passenger is present on the seat 55 at the center of the rear seat, is output by the detection processing described below.

For example, the second detection unit 5 acquires the captured image from the second information acquisition unit 2, and extracts the image in the set area from the captured image. The set area is, for example, an area set so as to include a specific seat among the rear seats, such as the seat 55 in the center of the rear seat, as shown in area C in FIG.

Then, the second detection unit 5 compares the extracted image with an image for similarity determination, which is stored in advance in the storage unit or the like, such as an image of the child seat 61 or an image of a PET bottle. By doing so, the object in the extracted image is specified. Here, the identification of the object in the image extracted by the second detection unit 5 can use a known object recognition method such as instance segmentation or template matching.

When the second detection unit 5 identifies that an object in the image extracted corresponding to a specific seat among the rear seats is, for example, a child seat 61 or the like, and is an object other than a passenger, A detection result indicating that there is an object other than the passenger, that is, there is no passenger in the rear seat, is output to the storage unit of the vehicle monitoring device 10 . On the other hand, there are cases where an object on a seat cannot be specified because it is blocked by passengers in other seats than the specific seat. If the object appearing in the extracted image is not specified, the second detection unit 5 outputs the detection result indicating that it is not clear whether or not the passenger is present in the rear seat. output to the storage unit of Note that the detection processing described above is an example, and it is also possible to specify an object in a captured image using various known algorithms.

Furthermore, the process of determining whether or not there is a passenger in the rear seat by the second detection unit 5 is not limited to the above example. In Embodiment 1, an example in which the second detection unit 5 performs face detection processing on a captured image has been described. It is also possible to set and perform face detection processing within the detection area.

When the second detection unit 5 sets a detection region and performs face detection processing within the detection region, for example, the detection region provided in the rear seat is the driver's seat 51 or the passenger's seat 52, that is, the passenger in the front seat. If the passenger in the rear seat is not detected within the detection area while the vehicle is not intruded, it becomes clear that no passenger is seated in the rear seat. This is because when the front seat occupant does not enter the detection area provided in the rear seat, the rear seat occupant is less likely to be blocked by the front seat occupant.

FIG. 7 is an explanatory diagram showing an example of occupant detection using captured images according to the second embodiment. In the captured image shown in FIG. 7,

passengers

56 and 57 are seated in a driver's seat 51 and a front passenger's seat 52 of the vehicle, respectively. , 59 are taken in a seated state. Further, in FIG. 7, neither an occupant nor an object is present in the seat 55 in the middle of the rear seat.

In the example of FIG. 7, the

occupants

56 and 57 in the driver's seat 51 or the passenger's seat 52 do not enter the detection area (area E shown in FIG. 7) provided for the center of the rear seat. The occupant's face is not detected within the detection area provided for the center of the seat. In such a case, even if an occupant is not detected in the center of the rear seat by the detection processing using the captured image, it is clear that there is no occupant in the center of the rear seat.

Furthermore, FIG. 7 shows an example in which an occupant 59 in the seat 54 on the right side of the rear seat is shielded by an occupant 56 in the driver's seat 51 . That is, in the detection area provided for the right side of the rear seat (area D shown in FIG. 7), the occupant 59 on the right side of the rear seat is not detected, while the occupant 56 on the driver's seat 51 is provided on the right side of the rear seat. significantly intrudes into the detection area. In the above case, even if an occupant is not detected on the right side of the rear seat in the detection process using the captured image, it is not clear whether there is an occupant on the right side of the rear seat.

Therefore, the second detection unit 5, for example, as an example of the process of determining whether or not there is an occupant in the rear seat, detects the presence of the driver's seat 51 or the passenger's seat in the detection area provided for the rear seat in the captured image. If the area of the face area detected from the

occupants

56 and 57 in the seat 52 is less than the set threshold value, there is no occupant in the rear seat that is not blocked by the occupant in the front seat. Determine that it does not exist. On the other hand, if the area of the detected face area of the occupant in the driver's seat or front passenger's seat in the detection area provided for the rear seat is greater than or equal to the set threshold value, whether or not the occupant is present in the rear seat. is not clear.

Next, we will explain how to identify the presence or absence of passengers in the rear seats. If the detection result regarding the presence or absence of the passenger in the rear seat obtained from the sensor information and the detection result regarding the presence or absence of the rear seat passenger obtained from the captured image are different, the passenger detection unit 12 detects the presence or absence of the passenger in the rear seat obtained from the captured image. The presence or absence of a passenger in the rear seat is specified using the detection result regarding the presence or absence of the passenger in the seat. That is, even if the presence/absence determination unit 6 obtains a detection result indicating that a passenger is present in the rear seat from the sensor information from the second detection unit 5, the captured image indicates that there is no passenger present in the rear seat. When the result is obtained, it is determined that there is no occupant in the rear seat using the detection result obtained from the captured image. In this way, even if the detection result that there is a passenger in the rear seat obtained from the sensor information is due to an erroneous detection, if it is clear from the captured image that there is no passenger in the rear seat, the rear seat Since it is specified that there is no occupant in the vehicle, erroneous detection in the occupant detection process can be suppressed, and the accuracy of occupant detection can be improved.

The presence/absence determining unit 6 obtains a detection result indicating that a passenger is present in the rear seat from the sensor information, and obtains a detection result indicating that it is unclear whether or not a passenger is present in the rear seat from the captured image. If so, it is determined that there is an occupant in the rear seat.

Next, the operation of the vehicle monitoring device 10 will be described. FIG. 8 is a flow chart showing an operation example of the vehicle monitoring device 10 according to the second embodiment. Here, hereinafter, steps that are the same as the processing of the vehicle monitoring device 10 according to the first embodiment are denoted by the same reference numerals as those shown in FIG. 4, and description thereof will be omitted or simplified. Although the flowchart of FIG. 8 does not show a process for terminating the operation of the vehicle monitoring device 10, the vehicle monitoring device 10, for example, the vehicle information acquisition unit 3 receives the information of the vehicle 50 from the vehicle-side control device 200. When the signal indicating that the engine has stopped is acquired, the operations of the first occupant detection process and the second occupant detection process are terminated. In the explanation below, even if multiple rear seats are provided, the rear seats are collectively referred to as rear seats. shall be processed for each seat.

First, after the operation of the vehicle monitoring device 10 is started, the first information acquiring section 1 of the vehicle monitoring device 10 acquires sensor information from the sensor 20 (ST101). Next, the first detection section 4 performs a first occupant detection process using sensor information (ST102).

Then, the first detection unit 4 outputs the detection result as to whether or not an occupant in the rear seat has been detected to the storage unit of the vehicle monitoring device 10 . When an occupant is detected in the rear seat (ST102; YES), the first detection unit 4 stores the detection result indicating that an occupant in the rear seat has been detected in the storage unit of the vehicle monitoring device 10 as the first detection result. output, and the rear seat first flag is turned ON (ST103). On the other hand, when the passenger in the rear seat is not detected (ST102; NO), the first detection unit 4 sets the detection result indicating that the passenger in the rear seat is not detected as the first detection result to the vehicle monitoring device 10. and the rear seat first flag is turned OFF (ST104).

Next, the second information acquisition section 2 of the vehicle monitoring device 10 acquires the captured image from the imaging device 30 (ST105). The second detection unit 5 uses the captured image to perform a second occupant detection process (ST106).

Then, if no occupant is detected in the rear seat from the captured image, the second detection unit 5 determines whether or not there is an occupant in the rear seat (ST201). Here, the second detection unit 5 determines whether or not an occupant is present in the rear seat by, for example, extracting an image in a region set to include a specific seat among the rear seats in the captured image. , by identifying whether or not the object shown in the extracted image is a passenger.

When the second detection unit 5 determines that there is no passenger in the rear seat (ST201; YES), the second detection unit 5 outputs a detection result indicating that there is no passenger in the rear seat to the storage unit, and turns on the third rear seat flag. (ST202). On the other hand, when the second detection unit 5 determines that it is unclear whether or not there is an occupant in the rear seat (ST201; NO), it stores the detection result indicating that it is unclear whether or not there is an occupant in the rear seat. and the rear seat third flag is turned OFF (ST203).

Here, the rear seat third flag stored in the storage unit indicates whether or not there is an occupant in the rear seat. That is, if the second detection unit 5 outputs a detection result indicating that there is no passenger in the rear seat, the rear seat third flag is turned ON, and the second detection unit 5 clearly determines whether or not there is a passenger in the rear seat. If the detection result that it is not is output, the rear seat third flag is turned OFF.

It should be noted that a plurality or a single number of rear seat third flags are provided according to the number of rear seats. For example, if the number of rear seats is three and the vehicle is provided with a seat on the left side of the rear seat, a seat on the right side of the rear seat, and a seat in the center of the rear seat, the rear seat third flag The 3rd flag, the 3rd flag on the right side of the rear seat, and the 3rd flag on the center of the rear seat.

Next, the presence/absence determination unit 6 of the vehicle monitoring device 10 acquires the second detection result from the second detection unit 5, and the presence/absence determination unit 6 uses the second detection result to determine whether a passenger is present in the rear seat. determine whether or not there is The presence/absence determining unit 6 refers to the rear seat second flag as the second detection result, and determines whether or not the rear seat second flag is ON (ST204), and the passenger in the rear seat is detected from the captured image. Check whether or not

Then, if the rear seat second flag is ON (ST204; YES), that is, if it is confirmed that a rear seat occupant is detected from the captured image, the presence/absence determination unit 6 confirms that a rear seat occupant is present. It is determined that the vehicle is on, and the determination result is output to the vehicle-side control device 200 (ST205). The determination result by the presence/absence determination unit 6 is output to, for example, a notification control unit that controls a seat belt reminder of the vehicle-side control device 200, and the notification control unit outputs the determination result that the passenger is present in the rear seat. When it is received, the function of the seat belt reminder in the rear seat is turned ON.

On the other hand, if the rear seat second flag is OFF (ST204; NO), that is, if no occupant is detected in the rear seat from the captured image, the presence/absence determination unit 6 proceeds to the processing described below. First, the presence/absence determination unit 6 acquires the first detection result from the first detection unit 4 . Then, the presence/absence determination unit 6 refers to the rear seat first flag as the first detection result, determines whether or not the rear seat first flag is ON (ST206), and determines whether or not the rear seat occupant is on the basis of the sensor information. Check whether it has been detected.

If the rear seat first flag is OFF (ST206; NO), that is, if it is confirmed from the sensor information that an occupant in the rear seat is not detected, the presence/absence determination unit 6 determines that there is no occupant in the rear seat. It judges and outputs the judgment result to the vehicle side control device 200 (ST207).

On the other hand, if the rear seat first flag is ON (ST206; YES), that is, if it is confirmed from the sensor information that an occupant in the rear seat has been detected, the presence/absence determining unit 6 performs the processing of ST208 described below. proceed to

The presence/absence determination section 6 refers to the rear seat third flag to determine whether or not the rear seat third flag is ON (ST208), and whether it is found from the captured image that there is no passenger in the rear seat. confirm whether or not Then, if the rear seat third flag is ON (ST208; YES), that is, if it is found from the captured image that there is no passenger in the rear seat, the presence/absence determination unit 6 determines that there is no passenger in the rear seat. It judges and outputs the judgment result to the vehicle side control device 200 (ST207).

On the other hand, if the rear seat third flag is OFF (ST208; NO), the presence/absence determination unit 6 derives the detection result that the passenger is present in the rear seat from the sensor information, and the captured image indicates that the passenger is in the rear seat. When it is confirmed that it is not clear whether or not the passenger is present, it is determined that the passenger is present in the rear seat, and the determination result is output to the vehicle side control device 200 (ST205).

In this way, even if the vehicle monitoring device 10 obtains a detection result indicating that there is a passenger in the rear seat from the sensor information, even though there is no passenger in the rear seat, the captured image indicates that there is no passenger in the rear seat. If it is clear that there is no occupant in the rear seat, it is specified that there is no occupant in the rear seat. Therefore, erroneous detection can be suppressed and the detection accuracy of the occupant in the rear seat, that is, the target person can be improved.

It should be noted that, as in the first embodiment, the second occupant detection process may be performed for the rear seats for which it is indicated that no occupant is present in the first detection result. In this way, since the second occupant detection process is performed on a seat for which there is a possibility of detection failure in the first occupant detection process, occupant detection failure can be suppressed and the processing load of the vehicle monitoring device 10 can be reduced. can be reduced. Further, in the flowchart of FIG. 8, the vehicle monitoring device 10 performs the processing of ST101 to ST104, and then performs the processing of ST105 to ST108 and ST201 to ST203, but the flowchart of FIG. 8 is an example. Yes, for example, ST101 to ST104 and ST105 to ST108 and ST201 to ST203 may be performed in parallel.

In the first and second embodiments, the occupant detection unit 12 detects an occupant in the rear seat and identifies the presence or absence of the occupant in the rear seat. It may be detected and the presence or absence of the passenger in the front seat may be specified. In this case, the sensor 20 may be provided so as to include the front and rear seats of the vehicle in its detection range, and the first detection unit 4 may detect the occupant in the front seat.

Further, in

Embodiments

1 and 2, the presence or absence of a passenger in the front seat is specified by the passenger detection unit 12 by detecting the presence or absence of the passenger in the front seat from the sensor information in the same manner as in the operation example described with reference to the flowchart of FIG. If a detection result indicating that there was no passenger in the front seat was obtained, and a detection result indicating that the passenger in the front seat was detected from the captured image, the presence of the passenger in the front seat could be specified. It is possible to improve the detection accuracy of the passenger in the front seat, that is, the target person.

Furthermore, in

Embodiments

1 and 2, the presence or absence of a passenger in the front seat is identified by the passenger detection unit 12 by detecting the presence or absence of a passenger in the front seat from sensor information in the same manner as in the operation example described using the flowchart of FIG. When a detection result indicating that there is no occupant in the front seat is obtained from the captured image, it is determined that there is no occupant in the front seat. It is possible to suppress erroneous detection and improve the detection accuracy of the passenger in the front seat, that is, the target person.

In

Embodiments

1 and 2, the first detection unit 4 uses sensor information to perform an attribute estimation process for estimating attributes such as the physique, age, or sex of the detected occupant, and to determine the state of the posture and the like. At least one of state estimation processing to estimate may be performed. In this case, if the sensor 20 is composed of a radio wave sensor 21 or an ultrasonic sensor or the like, and the physique, posture, etc. of the occupant are estimated by the first detection unit 4 using distance data, angle data, etc. obtained from the sensor 20, good. Various well-known algorithms can be used for the attribute estimation processing and the state estimation processing described above, and detailed description thereof will be omitted.

Further, in

Embodiments

1 and 2, an example has been described in which the second detection unit 5 detects an occupant using feature information related to the occupant's face, such as the occupant's face area and the positions of facial parts in the captured image. , the second detection unit 5 may detect the occupant using feature information relating to the physique, such as the skeletal points of the occupant in the captured image. Furthermore, the second detection unit 5 performs attribute estimation processing for estimating attributes such as the physique, age, and sex of the detected occupant using feature information about the occupant's face and physique, and also performs an attribute estimation process, and performs posture, drowsiness, physical condition, etc. At least one of state estimation processing for estimating the state of the may be performed. In addition, in the attribute estimation process, passenger authentication may be performed to associate the detected passenger with a specific individual, and the authentication result may be included in the attribute. Various well-known algorithms can be used for the attribute estimation processing and the state estimation processing described above, and detailed description thereof will be omitted.

As described above, the first detection unit 4 uses the sensor information to perform at least one of the occupant attribute estimation process and the state estimation process, and the second detection unit 5 performs the occupant attribute estimation process using the captured image. and state estimation processing. In this way, for example, in the first detection process of the first detection unit 4, the resolution of the sensor information acquired from the radio wave sensor 21 cannot be guaranteed, and a plurality of passengers are detected in a row, and the physique of each passenger is estimated. can be performed, the reliability of attribute estimation results and state estimation results can be improved, such as by estimating the physique of each occupant using the captured image by the second detection unit 5 .

Embodiment 3.
A vehicle monitoring apparatus 80 according to Embodiment 3 includes a first information acquisition section 1 that acquires sensor information and a second information acquisition section 2 that acquires a captured image, as in Embodiment 1. FIG. In the present embodiment, the vehicle monitoring device 80 includes a vehicle exterior detection unit 81 that detects a person outside the vehicle 50 (hereinafter referred to as a person outside the vehicle). different. The same reference numerals are given to the same components as in the first embodiment, and the description thereof is omitted.

FIG. 9 is a block diagram showing a configuration example of the vehicle monitoring system 101 according to the third embodiment. The vehicle monitoring system 101 includes a vehicle monitoring device 80, a sensor 20, and an imaging device 30. The vehicle monitoring device 80, the sensor 20, and the imaging device 30 are mounted on the vehicle 50, respectively.

The vehicle monitoring device 80 also includes a vehicle exterior detection unit 81 that detects a person outside the vehicle. The vehicle exterior detection unit 81 has a first vehicle exterior detection unit 7, a second vehicle exterior detection unit 8, and a vehicle exterior presence/absence determination unit 9, and detects a person outside the vehicle. Details of each configuration of the vehicle exterior detection unit 81 will be described later. That is, the target person in detection by the vehicle monitoring system 101 according to this embodiment is a person outside the vehicle. The vehicle monitoring system 101 may detect not only persons outside the vehicle but also passengers in the front seats or passengers in the rear seats as target persons for detection.

The sensor 20 in the present embodiment is, for example, a sensor 20 different from the imaging device 30, such as a radio wave sensor 21 or an ultrasonic sensor. It is provided on the ceiling of the passenger compartment, the exterior of the vehicle 50, and the like. If the sensor 20 is a radio wave sensor 21 or an ultrasonic sensor, one or more sensors are provided on the ceiling of the passenger compartment or the like so that the detection range includes a set distance from the exterior of the vehicle 50 . Note that the set distance is, for example, about 15 cm, and can be set appropriately as long as it is possible to detect a person existing near the vehicle 50 outside the vehicle.

In addition, one or a plurality of imaging devices 30 are arranged on an overhead console, an instrument panel, a steering column, a room mirror, etc., so as to be able to capture an image of the outside of the vehicle 50 through the windows of the vehicle 50 . FIG. 10 is an explanatory diagram showing an example of detection of a person outside the vehicle by the vehicle monitoring device 80 according to the third embodiment. FIG. 10 is a top view of the inside and outside of a vehicle 50 equipped with a vehicle monitoring device 80, and shows an example in which a person 70 outside the vehicle is present near the right side of the rear seat. A region F in FIG. 10 indicates the imaging range of the imaging device 30, and as shown in FIG. 10, the imaging device 30 includes the outside of the vehicle in the imaging range.

Returning to FIG. 9, the vehicle monitoring device 80 will be described. The information acquisition unit 11 of the vehicle monitoring device 80 has the first information acquisition unit 1 and the second information acquisition unit 2 as in the first embodiment. The first information acquisition section 1 of the information acquisition section 11 is connected to the sensor 20 and acquires sensor information from the sensor 20 . Here, the sensor information is information regarding the presence or absence of a person outside the vehicle. For example, when the sensor 20 is the radio wave sensor 21, the sensor information includes distance data indicating the distance between the radio wave sensor 21 and the detection target, angle data indicating the angle of the detection target with respect to the radio wave sensor 21, and the like. Further, for example, when the sensor 20 is a voice sensor, the sensor information is voice data or the like indicating the volume, direction of arrival, etc. of voice uttered by a person outside the vehicle. In addition, the sensor information is any one of these or a combination of a plurality of them.

On the other hand, the second information acquisition section 2 of the information acquisition section 11 is connected to the imaging device 30 and acquires a captured image from the imaging device 30 . Although the first information acquisition unit 1 and the second information acquisition unit 2 are shown separately in the example of FIG. You can go with one configuration. The information acquisition unit 11 then outputs the acquired sensor information and captured image to the vehicle exterior detection unit 81 .

The first vehicle exterior detection unit 7 of the vehicle exterior detection unit 81 detects a person outside the vehicle using sensor information. The second vehicle exterior detection unit 8 of the vehicle exterior detection unit 81 detects a person outside the vehicle using the captured image. Furthermore, the vehicle exterior presence/absence determination unit 9 of the vehicle exterior detection unit 81 uses the detection results of the first vehicle exterior detection unit 7 and the second vehicle exterior detection unit 8 to determine whether or not a person exists outside the vehicle. A part 9 is provided.

Next, the process of detecting a person outside the vehicle by the vehicle exterior detection unit 81 will be described. In the following description, an example will be described in which the first vehicle exterior detection unit 7 detects a person outside the vehicle using distance data and angle data acquired from the radio wave sensor 21 as sensor information.

First, the first vehicle exterior detection unit 7 acquires distance data and angle data as sensor information from the first information acquisition unit 1 of the information acquisition unit 11, and determines the size of a detection target such as a person, an animal, etc., and the size of the detection target. Calculate the range in which For example, if the first vehicle exterior detection unit 7 finds that the detection target exists near the vehicle 50, such as near the window of the vehicle 50 outside the vehicle, from the calculated size and range of the detection target, A detection result that a person exists outside the vehicle is derived, and the detection result is output to the storage unit of the vehicle monitoring device 80 . Hereinafter, for the sake of explanation, the detection processing using the sensor information by the first vehicle exterior detection unit 7 will be referred to as first vehicle exterior detection processing. Further, the detection result of the first vehicle-exterior detection process by the first vehicle-exterior detection unit 7 is referred to as a first vehicle-exterior detection result.

By the way, when detecting a person outside the vehicle using sensor information, the reliability of the detection result may become an issue. For example, when the sensor 20 is the radio wave sensor 21, depending on the positional relationship between the radio wave sensor 21 and the person outside the vehicle, the person outside the vehicle may be detected larger than necessary, or the body of the person outside the vehicle may not be detected. Therefore, it may not be possible to accurately determine whether the object to be detected is a person.

On the other hand, when the imaging device 30 is used to detect a person outside the vehicle, a highly reliable detection result can be obtained compared to the case where the sensor 20 different from the imaging device 30 is used to detect the person outside the vehicle. . Therefore, regarding the detection of a person outside the vehicle, if a person outside the vehicle is detected from the captured image, the person outside the vehicle, that is, the target person can be detected by using the detection result of the person detection process using the captured image. Can improve accuracy.

An example of detection processing by the second vehicle exterior detection unit 8 will be described. First, the second vehicle exterior detection unit 8 analyzes the captured image and detects the face of a person outside the vehicle in the captured image. Then, the second vehicle exterior detection unit 8 acquires a face area, which is an area in which the face of the person outside the vehicle is detected, and the facial feature information of the person outside the vehicle in the face area. Here, the feature information of the face of the person outside the vehicle is, for example, the contrast ratio of the eyes, nose, mouth, and cheeks after normalizing the size of the face. Note that the second vehicle exterior detection unit 8 may determine whether or not face detection has succeeded using facial feature information of a person outside the vehicle. In this case, the second vehicle exterior detection unit 8 determines that the face detection is successful if the luminance distribution of the face region is found to be face-like from the contrast ratio in the face region, for example, and determines that the face detection is successful. If it is determined that there is no face detection, it may be determined that the face detection has failed.

After detecting the face of the person outside the vehicle, the second outside detection unit 8 acquires the coordinates related to the face area surrounding the face of the person outside the vehicle, such as a rectangle contacting the contour of the person outside the vehicle, from the captured image. do. Here, the coordinates relating to the face area are, for example, the coordinates of each vertex, center, etc. of the rectangle when the face area is rectangular. Further, the second vehicle exterior detection unit 8 calculates dimensions such as the width, height, and area of the face region from the coordinates of the face region.

Then, the second vehicle exterior detection unit 8 identifies whether or not there is a person outside the vehicle from the acquired coordinates, width, height, area, etc. of the face area, and provides information regarding the presence or absence of the person outside the vehicle as the detection result. , to the storage unit of the vehicle monitoring device 80 . The detection processing by the second vehicle exterior detection unit 8 is not limited to the above example, and various known algorithms can be used. Further, the second vehicle exterior detection unit 8 may detect a person outside the vehicle using feature information related to the physique of the person outside the vehicle, such as the skeletal points of the person outside the vehicle in the captured image.

In the above example, the detection result of the second vehicle exterior detection unit 8 is stored in the storage unit. good. Hereinafter, for the sake of explanation, the detection processing using the captured image by the second vehicle exterior detection unit 8 will be referred to as the second vehicle exterior detection processing. Further, the detection result of the second vehicle-exterior detection process by the second vehicle-exterior detection unit 8 is referred to as a second vehicle-exterior detection result.

Next, the presence/absence determination process for determining whether or not there is a person outside the vehicle will be described. The presence/absence determination unit 9 determines that a person exists outside the vehicle if the person outside the vehicle can be detected from the captured image even if the person outside the vehicle is not detected from the sensor information. In this manner, when the detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information differs from the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image, the vehicle exterior detection unit 81 The presence or absence of a person outside the vehicle is specified using the detection result regarding the presence or absence of the person outside the vehicle.

The operation of the vehicle monitoring device 80 will be described below. FIG. 11 is a flow chart showing an operation example of the vehicle monitoring device 80 according to the third embodiment. For example, when the vehicle information acquisition unit 3 acquires a signal indicating that the engine of the vehicle 50 has started from the vehicle-side control device 200, the vehicle monitoring device 80 performs the first vehicle exterior detection process and the second vehicle exterior detection process. to start.

Although the flowchart of FIG. 11 does not show a process for terminating the operation of the vehicle monitoring device 80, the vehicle monitoring device 80 is configured such that the vehicle information acquisition unit 3, for example, controls the vehicle 50 from the vehicle-side control device 200. When the signal indicating that the engine has stopped is acquired, the operations of the first vehicle exterior detection process and the second vehicle exterior detection process are terminated.

First, after the vehicle monitoring device 80 starts operating, the first information acquiring section 1 of the vehicle monitoring device 80 acquires sensor information from the sensor 20 (ST301). Next, the first vehicle exterior detection unit 7 uses the sensor information to perform the first vehicle exterior detection process (ST302). Here, the determination of whether or not a person outside the vehicle has been detected by the first vehicle exterior detection unit 7 is based on, for example, the distance data acquired as sensor information, the size of the detection target calculated from the angle data, the existing range, and the like. , it may be determined whether or not the detected detection target exists outside the vehicle.

Then, the first vehicle exterior detection unit 7 outputs a detection result indicating whether or not a person outside the vehicle has been detected to the storage unit of the vehicle monitoring device 80 . When a person is detected outside the vehicle (ST302; YES), the first outside detection unit 7 stores the detection result indicating that the person outside the vehicle is detected as a first outside detection result in the storage unit of the vehicle monitoring device 80. output, and turn on the vehicle exterior first flag (ST303). On the other hand, when a person is not detected outside the vehicle (ST302; NO), the first vehicle exterior detection unit 7 sets the detection result indicating that a person outside the vehicle is not detected as a first vehicle exterior detection result to the vehicle monitoring device 80. , and the vehicle exterior first flag is turned OFF (ST304).

Here, the first vehicle exterior flag stored in the storage unit indicates whether or not a person has been detected outside the vehicle by the first person detection process. That is, when the first vehicle exterior detection unit 7 outputs a first vehicle exterior detection result indicating that a person outside the vehicle has been detected, the first vehicle exterior flag is turned ON, and the first vehicle exterior detection unit 7 detects a person outside the vehicle. If the first vehicle exterior detection result indicating that the vehicle has not been detected is output, the vehicle exterior first flag is turned OFF.

Next, the second information acquisition section 2 of the vehicle monitoring device 80 acquires the captured image from the imaging device 30 (ST305). The second vehicle exterior detection unit 8 performs a second vehicle exterior detection process using the captured image (ST306). Here, the second vehicle exterior detection unit 8 determines whether or not a person has been detected outside the vehicle, for example, based on the coordinates, width, height, area, etc. of the face area acquired from the captured image. may be performed by determining whether or not it exists in the

Then, the second vehicle exterior detection unit 8 outputs the detection result as to whether or not a person outside the vehicle has been detected to the storage unit of the vehicle monitoring device 80 . When a person is detected outside the vehicle (ST306; YES), the second outside detection unit 8 stores the detection result indicating that the person outside the vehicle is detected as a second outside detection result in the storage unit of the vehicle monitoring device 80. output to turn ON the second flag outside the vehicle (ST307). On the other hand, when a person is not detected outside the vehicle (ST306; NO), the second vehicle exterior detection unit 8 sets the detection result indicating that a person outside the vehicle is not detected as a second vehicle exterior detection result to the vehicle monitoring device 80. , and the second flag outside the vehicle is turned OFF (ST308).

Here, the second vehicle exterior flag stored in the storage unit indicates whether or not a person has been detected outside the vehicle by the second vehicle exterior detection process. That is, when the second vehicle exterior detection unit 8 outputs a second vehicle exterior detection result indicating that a person outside the vehicle has been detected, the second vehicle exterior flag is turned ON, and the second vehicle exterior detection unit 8 detects a person outside the vehicle. If the second vehicle exterior detection result indicating that the vehicle has not been detected is output, the vehicle exterior second flag is turned OFF.

Next, the vehicle-exterior presence/absence determination unit 9 of the vehicle monitoring device 80 acquires the first vehicle-exterior detection result from the first vehicle-exterior detection unit 7 and acquires the second vehicle-exterior detection result from the second vehicle-exterior detection unit 8 . Then, the vehicle exterior presence/absence determination unit 9 determines whether or not a person is present outside the vehicle using the first vehicle exterior detection result and the second vehicle exterior detection result.

The vehicle exterior presence/absence determination unit 9 refers to the vehicle exterior first flag as the first vehicle exterior detection result, and determines whether or not the vehicle exterior first flag is ON (ST309). confirm whether or not If the vehicle outside first flag is ON (ST309; YES), that is, if it is confirmed from the sensor information that a person outside the vehicle has been detected, the vehicle outside presence/absence determining unit 9 determines that a person exists outside the vehicle. , the determination result is output to the vehicle-side control device 200 (ST310).

The determination result by the presence/absence determination unit 9 outside the vehicle is output to, for example, a notification unit (not shown) of the vehicle-side control device 200 that notifies the presence of a person outside the vehicle. Then, when the notification unit receives the determination result that a person exists outside the vehicle, the notification unit notifies the driver or the like of the existence of the person outside the vehicle from the notification unit of the vehicle or a terminal (not shown) possessed by the driver. inform. By doing so, the driver of the vehicle 50 can recognize that a person such as a suspicious person is present in the vicinity of the vehicle.

On the other hand, if the vehicle exterior first flag is OFF (ST309; NO), that is, if it is confirmed from the sensor information that a person has not been detected outside the vehicle, the vehicle exterior presence/absence determination unit 9 performs the processing of ST311 described below. proceed to

The vehicle exterior presence/absence determining unit 9 refers to the vehicle exterior second flag as the second vehicle exterior detection result, and determines whether or not the vehicle exterior second flag is ON (ST311). confirm whether or not If the vehicle exterior second flag is ON (ST311; YES), that is, if it is confirmed that a person outside the vehicle has been detected from the captured image, the vehicle exterior presence/absence determination unit 9 determines that a person exists outside the vehicle. , the determination result is output to the vehicle-side control device 200 (ST310).

On the other hand, if the vehicle exterior second flag is OFF (ST311; NO), that is, if it is confirmed that a person outside the vehicle has not been detected from the captured image, the vehicle exterior presence/absence determination unit 9 determines that there is no person outside the vehicle. It determines and outputs the determination result to the vehicle-side control device 200 (ST312).

In this way, when the vehicle exterior detection unit 81 obtains a detection result indicating that a person outside the vehicle has not been detected from the sensor information and obtains a detection result indicating that a person outside the vehicle has been detected from the captured image, Identify the presence of a person outside the vehicle. As a result, even if a person is actually present outside the vehicle, even if the presence of a person outside the vehicle is not detected from the sensor information, the presence of a person outside the vehicle is detected from the captured image. Furthermore, since it is determined that there is a person outside the vehicle using the detection result obtained from the captured image, it is possible to prevent omissions in detecting the person outside the vehicle and improve the detection accuracy of the person outside the vehicle, that is, the target person.

Further, in the flowchart of FIG. 11, the vehicle monitoring device 80 performs the processing of ST301 to ST304 and then performs the processing of ST305 to ST308, but the flowchart of FIG. 11 is an example. The processing of ST301 to ST304 and the processing of ST305 to ST308 may be performed in parallel.

In the vehicle monitoring device 80 of the present embodiment, as in the case of the second embodiment, the presence/absence determination unit 9 detects from the second vehicle exterior detection unit 8 the detection result indicating that a person is present outside the vehicle based on the sensor information. Even if it is determined that there is no person outside the vehicle by using the detection result obtained from the imaged image, if a detection result indicating that there is no person outside the vehicle is obtained from the imaged image. good.

That is, for example, even if the presence/absence determination unit 9 outside the vehicle obtains a detection result indicating that a detection target exists outside the vehicle detected from the sensor information from the first detection unit 7, the second detection unit 8 outside the vehicle When it is specified from the captured image that the object to be detected is an object other than a person, it may be determined that there is no person outside the vehicle. In this way, even if the detection result obtained from the sensor information that there is a person outside the vehicle is due to an erroneous detection, if it is clear from the captured image that there is no person outside the vehicle, the person outside the vehicle will be detected. False detection can be suppressed and the detection accuracy of the person outside the vehicle, that is, the target person can be improved.

Further, the vehicle monitoring device 80 may perform the first vehicle exterior detection process and the second vehicle exterior detection process even when the engine of the vehicle is stopped. In this way, the notification control unit of the vehicle-side control device 200 can notify the presence of a person outside the vehicle to the driver or the like from the notification unit of the vehicle or the terminal held by the driver. A driver can be notified that a suspicious person exists in the vicinity of the vehicle before a criminal act is committed on the vehicle.

Next, the hardware configuration that implements the functions of the vehicle monitoring device 80 will be described. FIG. 12 is a diagram showing a hardware configuration example of a vehicle monitoring device 80 according to Embodiment 3. As shown in FIG. First information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first vehicle exterior detection unit 7, second vehicle exterior detection unit 8, vehicle exterior presence/absence determination unit 9 in vehicle monitoring device 80, The functions of the vehicle exterior detection unit 81 and the storage unit are implemented by a processing circuit. That is, the first information acquisition unit 1, the second information acquisition unit 2, the vehicle information acquisition unit 3, the information acquisition unit 11, the first vehicle exterior detection unit 7, the second vehicle exterior detection unit 8, and the vehicle exterior presence/absence determination of the vehicle monitoring device 80 The unit 9, the vehicle exterior detection unit 81, and the storage unit may be a processing circuit 80a that is dedicated hardware as shown in FIG. It may be the executing processor 80b.

As shown in FIG. 12A, a first information acquisition unit 1, a second information acquisition unit 2, a vehicle information acquisition unit 3, an information acquisition unit 11, a first vehicle exterior detection unit 7, a second vehicle exterior detection unit 8, and a vehicle exterior presence/absence determination unit. 9. If the vehicle exterior detection unit 81 and the storage unit are dedicated hardware, the processing circuit 80a may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit ), FPGA (Field-programmable Gate Array), or a combination thereof. First information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first vehicle exterior detection unit 7, second vehicle exterior detection unit 8, vehicle exterior presence/absence determination unit 9, vehicle exterior detection unit 81, Each function of each unit of the storage unit and the storage unit may be realized by a processing circuit, or the functions of each unit may be collectively realized by one processing circuit.

As shown in FIG. 12B, a first information acquisition unit 1, a second information acquisition unit 2, a vehicle information acquisition unit 3, an information acquisition unit 11, a first vehicle exterior detection unit 7, a second vehicle exterior detection unit 8, and a vehicle exterior presence/absence determination unit. 9. When the outside detection unit 81 and the storage unit are the processor 80b, the function of each unit is realized by software, firmware, or a combination of software and firmware. Software or firmware is written as a program and stored in the memory 80c. The processor 80b reads out and executes the programs stored in the memory 80c to obtain the first information acquisition unit 1, the second information acquisition unit 2, the vehicle information acquisition unit 3, the information acquisition unit 11, and the first outside detection unit 7. , the second vehicle exterior detection unit 8, the vehicle exterior presence/absence determination unit 9, the vehicle exterior detection unit 81, and the storage unit. That is, first information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first vehicle exterior detection unit 7, second vehicle exterior detection unit 8, vehicle exterior presence/absence determination unit 9, vehicle exterior detection unit 81, and storage comprises memory 80c for storing a program which, when executed by processor 80b, results in the execution of the steps shown in FIG. Further, these programs include a first information acquisition unit 1, a second information acquisition unit 2, a vehicle information acquisition unit 3, an information acquisition unit 11, a first vehicle exterior detection unit 7, a second vehicle exterior detection unit 8, and a vehicle exterior presence/absence determination unit. 9, the vehicle exterior detection unit 81, and the storage unit.

Here, the processor 80b is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic device, a processor, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor). The memory 80c may be non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, EPROM (Erasable Programmable ROM), EEPROM (Electrically EPROM), etc. However, it may be a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a mini disk, a CD (Compact Disc) or a DVD (Digital Versatile Disc).

First information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first vehicle exterior detection unit 7, second vehicle exterior detection unit 8, vehicle exterior presence/absence determination unit 9, vehicle exterior detection unit 81 and each function of the storage unit may be partly realized by dedicated hardware and partly realized by software or firmware. Thus, the processing circuit 80a in the vehicle monitoring device 80 can realize each of the functions described above by hardware, software, firmware, or a combination thereof. Also, a first information acquisition unit 1, a second information acquisition unit 2, a vehicle information acquisition unit 3, an information acquisition unit 11, a first vehicle exterior detection unit 7, a second vehicle exterior detection unit 8, a vehicle exterior presence/absence determination unit 9, and a vehicle exterior detection unit 81 and at least part of the functions of the storage unit may be executed by an external server.

In this way, the vehicle monitoring device 80 includes the first information acquisition unit 1 that acquires sensor information from the sensor 20 whose detection range is outside the vehicle, and the second information acquisition unit 1 that acquires the captured image from the imaging device 30 whose imaging range is outside the vehicle. 2 information acquisition unit 2, and a vehicle exterior detection unit 81 that detects a person outside the vehicle from the sensor information and the captured image respectively acquired by the first information acquisition unit 1 and the second information acquisition unit 2, and the vehicle exterior detection unit If the detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information is different from the detection result regarding the presence or absence of the person outside the vehicle obtained from the captured image, 81 determines whether the person outside the vehicle is present or not obtained from the captured image. If the presence or absence of a person outside the vehicle is specified using the detection result, detection failure of the person outside the vehicle can be prevented, and the detection accuracy of the person outside the vehicle, that is, the target person can be improved.

In addition, each embodiment disclosed in this specification can be freely combined within its scope, and each embodiment can be modified or omitted as appropriate. . For example, the vehicle monitoring device may include both the occupant detection unit 12 and the vehicle exterior detection unit 81 to detect both the occupant and the person outside the vehicle.

1 first information acquisition unit, 2 second information acquisition unit, 3 vehicle information acquisition unit, 4 first detection unit, 5 second detection unit, 6 presence/absence determination unit, 7 first vehicle exterior detection unit, 8 second vehicle exterior detection unit , 9 vehicle presence/absence determination unit, 10, 80 vehicle monitoring device, 10a, 80a processing circuit, 10b, 80b processor, 10c, 80c memory, 11 information acquisition unit, 12 occupant detection unit, 20 sensor, 21 radio wave sensor, 30 imaging device , 50 Vehicle, 51 Driver's seat, 52 Passenger seat, 53, 54, 55 Seats, 56, 57, 58, 59, 60 Passengers, 61 Child seat, 70 People outside the vehicle, 81 Outside detection unit, 100, 101 Vehicle monitoring system.

Claims

A first information acquisition unit that acquires sensor information from a sensor that includes at least a rear seat of the vehicle in its detection range;
a second information acquisition unit that acquires a captured image from an imaging device that includes an imaging range of the front seats and the rear seats of the vehicle;
an occupant detection unit that detects an occupant in the rear seat from the sensor information and the captured image obtained by the first information acquisition unit and the second information acquisition unit, respectively;
The occupant detection unit detects the captured image when a detection result regarding the presence or absence of the passenger in the backseat obtained from the sensor information is different from a detection result regarding the presence or absence of the backseat passenger obtained from the captured image. A vehicle monitoring device, wherein the presence or absence of the passenger in the rear seat is specified using the detection result regarding the presence or absence of the passenger in the rear seat obtained from.
The occupant detection unit obtains a detection result indicating that no occupant is present in the backseat from the sensor information, and obtains a detection result indicating that an occupant is present in the backseat from the captured image. 2. The vehicle monitoring device according to claim 1, wherein it is specified that an occupant is present in a rear seat.
The occupant detection unit obtains a detection result indicating that an occupant is present in the backseat from the sensor information, and obtains a detection result indicating that no occupant is present in the backseat from the captured image. 3. The vehicle monitoring device according to claim 1, wherein it is specified that there is no passenger in the rear seat.
The occupant detection unit obtains a detection result indicating that an occupant is present in the rear seat from the sensor information, and obtains a detection result indicating that an object other than the occupant is present in the rear seat from the captured image. 4. The vehicle monitoring device according to any one of claims 1 to 3, wherein the vehicle monitoring device determines that there is no passenger in the rear seat when the vehicle is in the rear seat.
The occupant detection unit obtains a detection result indicating that an occupant is present in the rear seat from the sensor information, and determines from the captured image that the occupant is present in the rear seat that is not blocked by the occupant in the front seat. 5. The vehicle monitoring device according to any one of claims 1 to 4, wherein when a detection result indicating that no passenger is present is obtained, it is determined that there is no passenger in the rear seat.
The first information acquisition unit acquires sensor information from a sensor whose detection range includes the front seat in addition to the passenger in the rear seat,
Detecting the front seat occupant in addition to the rear seat occupant from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit,
The occupant detection unit detects the presence or absence of the occupant in the front seat obtained from the sensor information and the detection result regarding the presence or absence of the occupant in the front seat obtained from the captured image. The vehicle monitor according to any one of claims 1 to 5, wherein the presence or absence of the passenger in the rear seat is specified using the detection result regarding the presence or absence of the passenger in the front seat obtained from Device.
The occupant detection unit obtains a detection result indicating that no occupant is present in the front seat from the sensor information, and obtains a detection result indicating that an occupant is present in the front seat from the captured image. 7. The vehicle monitoring device according to claim 6, wherein it is specified that an occupant is present in the rear seat.
The occupant detection unit obtains a detection result indicating that an occupant is present in the front seat from the sensor information, and obtains a detection result indicating that no occupant is present in the front seat from the captured image. 8. The vehicle monitoring device according to claim 6, wherein it is specified that there is no passenger in the rear seat.
The occupant detection unit uses the sensor information to estimate the attribute of the occupant and the state of the occupant for the occupant present in the seat for which a detection result indicating the presence of the occupant is obtained from the sensor information. and performing at least one of the above, and using the imaged image to estimate the attributes of the occupant and determine the state of the occupant, for the occupant present in the seat for which a detection result indicating the presence of the occupant is obtained from the imaged image. The vehicle monitoring device according to any one of claims 1 to 8, wherein at least one of estimation is performed.
A first information acquisition unit that acquires sensor information from a sensor that includes the outside of the vehicle in its detection range;
a second information acquisition unit that acquires a captured image from an imaging device that includes an imaging range of the outside of the vehicle;
A vehicle exterior detection unit that detects a person outside the vehicle from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit,
When the detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information is different from the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image, the vehicle exterior detection unit detects the presence or absence of the person outside the vehicle. A vehicle monitoring device that identifies the presence or absence of a person outside the vehicle using a detection result regarding the presence or absence of the person outside the vehicle obtained from a captured image.
When the vehicle exterior detection unit obtains a detection result indicating that a person does not exist outside the vehicle from the sensor information, and obtains a detection result indicating that a person exists outside the vehicle from the captured image. 11. The vehicle monitoring apparatus according to claim 10, wherein the vehicle monitoring apparatus identifies that a person exists outside the vehicle.
When the vehicle exterior detection unit obtains a detection result indicating that a person exists outside the vehicle from the sensor information, and obtains a detection result indicating that a person does not exist outside the vehicle from the captured image. 12. The vehicle monitoring device according to claim 10 or 11, wherein the vehicle monitoring device determines that there is no person outside the vehicle.
a sensor mounted on a vehicle and including at least a rear seat of the vehicle in a detection range;
an imaging device mounted on the vehicle and including the front seats and the rear seats of the vehicle in an imaging range;
A first information acquisition unit that acquires sensor information from the sensor;
a second information acquisition unit that acquires a captured image from the imaging device;
an occupant detection unit that detects an occupant in the rear seat from the sensor information and the captured image obtained by the first information acquisition unit and the second information acquisition unit, respectively;
The occupant detection unit detects the captured image when a detection result regarding the presence or absence of the passenger in the backseat obtained from the sensor information is different from a detection result regarding the presence or absence of the backseat passenger obtained from the captured image. A vehicle monitoring system, wherein the presence or absence of the passenger in the rear seat is specified using the detection result regarding the presence or absence of the passenger in the rear seat obtained from the above.
a sensor mounted on a vehicle and including the outside of the vehicle in its detection range;
an imaging device mounted on the vehicle and including the exterior of the vehicle in an imaging range;
A first information acquisition unit that acquires sensor information from the sensor;
a second information acquisition unit that acquires a captured image from the imaging device;
A vehicle exterior detection unit that detects a person outside the vehicle from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit,
When the detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information is different from the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image, the vehicle exterior detection unit detects the presence or absence of the person outside the vehicle. A vehicle monitoring system that identifies the presence or absence of a person outside the vehicle using a detection result regarding the presence or absence of the person outside the vehicle obtained from a captured image.