JP7446492B2 - Vehicle monitoring device, vehicle monitoring system, and vehicle monitoring method - Google Patents

Description

The present disclosure relates to a vehicle monitoring device and a vehicle monitoring system that detect an occupant inside a vehicle or a person outside the vehicle.

Detects occupants inside the vehicle or people outside the vehicle, controls airbags, seatbelt reminders, etc. according to the detection results such as the position of the occupant, and reports the presence of a suspicious person according to the detection results of the person outside the vehicle. Technology has been developed to do this. Conventionally, when it comes to detecting occupants inside a vehicle, an imaging device is installed to include the front and rear seats of the vehicle in the imaging range, and detection is performed based on the position, size, etc. of the occupant's head in the captured image obtained from the imaging device. It was detected in which seat the occupant was present (for example, see Patent Document 1).

Japanese Patent Application Publication No. 2020-50090

When attempting to detect such an occupant inside the vehicle or a person outside the vehicle, that is, a target person, it is conceivable to use a sensor different from the imaging device, such as a radio wave sensor or a seating sensor. However, for example, when trying to detect a passenger in the rear seat using a seating sensor, depending on the posture of the passenger in the seat, it may not be possible to accurately measure the weight of the passenger sitting in the seat. Even if it were used, there was a problem that it might not be able to detect it sufficiently.

The present disclosure has been made to solve the above-mentioned problems, and uses detection results obtained from an imaging device and detection results obtained from a sensor different from the imaging device to detect a target person. The purpose is to improve detection accuracy.

A first vehicle monitoring device according to the present disclosure includes a first information acquisition unit that acquires sensor information from a sensor whose detection range includes at least the rear seats of a vehicle, and an imaging unit whose imaging range includes the front seats and rear seats of the vehicle. a second information acquisition unit that acquires a captured image from the device; an occupant detection unit that detects a rear seat occupant from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit; If the detection result regarding the presence or absence of a rear seat occupant obtained from the sensor information differs from the detection result regarding the presence or absence of a rear seat occupant obtained from the captured image, the occupant detection unit detects the presence or absence of the rear seat occupant obtained from the captured image. The presence or absence of a rear seat occupant is determined using the detection results regarding the presence or absence of a rear seat occupant.

Further, the first vehicle monitoring system according to the present disclosure includes a sensor that is mounted on a vehicle and whose detection range includes at least the rear seats of the vehicle, and an image sensor that is mounted on the vehicle and whose imaging range includes the front seats and rear seats of the vehicle. a first information acquisition unit that acquires sensor information from the device, a sensor, a second information acquisition unit that acquires a captured image from the imaging device, and the first information acquisition unit and the second information acquisition unit each acquire, an occupant detection section that detects an occupant in the rear seat from the sensor information and the captured image, and the occupant detection section detects the presence or absence of the occupant in the rear seat obtained from the sensor information and the detection result obtained from the captured image. If the detection result regarding the presence or absence of a rear seat occupant is different from the detection result regarding the presence or absence of a rear seat occupant, the presence or absence of a rear seat occupant is determined using the detection result regarding the presence or absence of a rear seat occupant obtained from the captured image.

Further, the second vehicle monitoring device according to the present disclosure includes a first information acquisition unit that acquires sensor information from a sensor whose detection range includes the exterior of the vehicle, and an image pickup unit that acquires sensor information from an imaging device whose imaging range includes the exterior of the vehicle. A second information acquisition unit that acquires an image; and an exterior detection unit that detects a person outside the vehicle from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit, If the detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information differs from the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image, the vehicle exterior detection unit detects the presence or absence of a person outside the vehicle obtained from the captured image. The presence or absence of a person outside the vehicle is determined using the detection result regarding the presence or absence of a person outside the vehicle.

Further, the second vehicle monitoring system according to the present disclosure includes a sensor that is mounted on a vehicle and whose detection range includes the outside of the vehicle, an imaging device that is mounted on the vehicle and whose imaging range includes the outside of the vehicle, and a sensor that includes: A first information acquisition unit that acquires sensor information; a second information acquisition unit that acquires a captured image from an imaging device; and a first information acquisition unit that acquires sensor information and a captured image that are acquired by the first information acquisition unit and the second information acquisition unit, respectively. , an outside detection unit that detects a person outside the vehicle, and the outside detection unit detects the presence or absence of a person outside the vehicle obtained from the sensor information, and the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image. If the detection result regarding the presence or absence of a person is different from the detection result regarding the presence or absence of a person outside the vehicle, the presence or absence of the person outside the vehicle is specified using the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image.

According to the present disclosure, since an occupant inside the vehicle or a person outside the vehicle, that is, a target person, is detected using an imaging device and a sensor different from the imaging device, the detection accuracy of the target person can be improved.

1 is a block diagram showing a configuration example of a vehicle monitoring system according to Embodiment 1. FIG. FIG. 2 is an explanatory diagram showing a detection range of a sensor and an imaging range of an imaging device according to the first embodiment. FIG. 3 is an explanatory diagram showing an example of passenger detection using captured images according to the first embodiment. 3 is a flowchart illustrating an example of the operation of the vehicle monitoring device according to the first embodiment. 1 is a diagram showing an example of a hardware configuration of a vehicle monitoring device according to a first embodiment; FIG. FIG. 7 is an explanatory diagram showing an example of occupant detection using captured images according to Embodiment 2; FIG. 7 is an explanatory diagram showing an example of occupant detection using captured images according to Embodiment 2; 7 is a flowchart illustrating an example of the operation of the vehicle monitoring device according to Embodiment 2. FIG. FIG. 3 is a block diagram showing a configuration example of a vehicle monitoring system according to a third embodiment. FIG. 7 is an explanatory diagram showing an example of detection of a person outside the vehicle by the vehicle monitoring device according to Embodiment 3; 7 is a flowchart illustrating an example of the operation of the vehicle monitoring device according to Embodiment 3. FIG. FIG. 7 is a diagram illustrating an example of a hardware configuration of a vehicle monitoring device according to a third embodiment.

Embodiments will be described below based on the drawings.

Embodiment 1.
FIG. 1 is a block diagram showing a configuration example of a vehicle monitoring system 100 according to the first embodiment. The vehicle monitoring system 100 includes a vehicle monitoring device 10, a sensor 20, and an imaging device 30, and the vehicle monitoring device 10, the sensor 20, and the imaging device 30 are each mounted on a vehicle. The vehicle monitoring device 10 is also connected to a vehicle-side control device 200 that controls in-vehicle devices such as air conditioning equipment, audio equipment, navigation equipment, and notification units, and the engine in the vehicle in which the vehicle monitoring device 10 is installed. There is.

The target person for detection by the vehicle monitoring system 100 according to the present embodiment is a passenger in the rear seat. Note that the vehicle monitoring system 100 does not detect only rear seat occupants, and may include front seat occupants as target persons. FIG. 2 is an explanatory diagram showing the detection range of the sensor 20 and the imaging range of the imaging device 30 according to the first embodiment. FIG. 2A is a side view of the interior of the vehicle 50 equipped with the vehicle monitoring device 10, and FIG. 2B is a diagram of the interior of the vehicle 50 equipped with the vehicle monitoring device 10 viewed from above. The sensor 20 is a sensor 20 that is installed in a vehicle and is capable of detecting at least a rear seat occupant among the occupants inside the vehicle. The sensor 20 is a sensor 20 different from the imaging device 30, such as a radio wave sensor 21, a weight sensor (seating sensor), an audio sensor, or an ultrasonic sensor, which can detect an occupant, for example. Moreover, the sensor 20 is any one of these or a combination of a plurality of them.

FIG. 2A shows an example in which the sensor 20 is a radio wave sensor 21, and the detection range of the radio wave sensor 21 is indicated by area A. Note that the radio wave sensor 21 is, for example, a sensor that transmits millimeter waves and receives reflected waves reflected from a moving object. In the example of FIG. 2A, the sensor 20 is provided on the ceiling of the cabin of the vehicle 50, etc. so that the detection range includes at least the rear seat. Further, FIG. 2B shows an example in which an occupant 58 is seated on the seat 53 on the left side of the rear seat. Here, the seat on the left side of the rear seat refers to the seat 53 on the left side with respect to the front of the vehicle among the rear seats, and the seat on the right side of the rear seat refers to the seat 54 on the right side of the rear seat with respect to the front of the vehicle. The term "rear center seat" refers to the seat 55 provided between the left rear seat and the right rear seat among the rear seats. Hereinafter, for the sake of explanation, unless otherwise specified, the term "rear seat" refers to the seat 53 on the left side of the rear seat, the seat 54 on the right side of the rear seat, and the seat 55 in the center of the rear seat collectively. Note that the number of rear seats does not need to be three as in the example of FIG. 2B, and the number of rear seats is arbitrary.

The imaging device 30 includes, for example, a wide-angle camera, an infrared camera, etc., and images the inside of the vehicle 50. Furthermore, the imaging device 30 may be a TOF (Time-of-flight) camera that can capture an image that reflects the distance between the imaging device 30 and the subject. The imaging device 30 images the interior of the vehicle at intervals of 30 to 60 fps (frames per second), for example, and outputs the captured images to the second information acquisition unit 2 of the information acquisition unit 11 included in the vehicle monitoring device 10. Hereinafter, the image captured by the imaging device 30 will be referred to as a captured image.

In the example of FIG. 2A, the imaging range of the imaging device 30 is shown as region B. The imaging device 30 includes one or more imaging devices, an overhead console, an instrument panel, and a steering wheel so that the imaging range includes the front seats of the driver's seat 51 and the passenger seat 52, and at least one of the rear seats. Placed in columns, room mirrors, etc. Hereinafter, the occupants in the vehicle, such as the occupants 56, 57, and 58, who are the detection targets of the vehicle monitoring device 10, will also be collectively referred to as "occupants." That is, the occupants include the driver.

Returning to FIG. 1, the vehicle monitoring device 10 will be explained. The vehicle monitoring device 10 includes an information acquisition unit 11 that acquires sensor information and captured images from the sensor 20 and the imaging device 30, respectively, and an occupant detection unit 12 that detects a rear seat occupant from the sensor information and captured images. .

As shown in FIG. 1, the information acquisition section 11 includes a first information acquisition section 1 and a second information acquisition section 2. The first information acquisition unit 1 of the information acquisition unit 11 is connected to the sensor 20 and acquires sensor information from the sensor 20. Here, the sensor information is information regarding the presence or absence of an occupant in the vehicle. For example, when the sensor 20 is the radio wave sensor 21, the sensor information includes distance data indicating the distance between the radio wave sensor 21 and the detection target, angle data indicating the angle of the detection target with respect to the radio wave sensor 21, and the like. For example, when the sensor 20 is a seating sensor, the sensor information is seat data indicating information about the seat where the seating sensor is installed, weight data detected by the seating sensor, and the like. For example, when the sensor 20 is a voice sensor, the sensor information is voice data indicating the volume, direction of arrival, etc. of the voice uttered by the occupant. Note that the sensor information is any one of these or a combination of a plurality of them.

On the other hand, the second information acquisition unit 2 of the information acquisition unit 11 is connected to the imaging device 30 and acquires a captured image from the imaging device 30. In the example of FIG. 1, the first information acquisition unit 1 and the second information acquisition unit 2 are illustrated separately, but the acquisition of captured images from the imaging device 30 and the acquisition of sensor information from the sensor 20 are A single configuration may be used. The information acquisition unit 11 then outputs the acquired sensor information and captured image to the occupant detection unit 12, which will be described later.

Further, the information acquisition unit 11 includes a vehicle information acquisition unit 3 connected to the vehicle-side control device 200. The vehicle information acquisition unit 3 acquires vehicle information such as signals related to starting, stopping, etc. of the vehicle from the vehicle-side control device 200. Then, the vehicle information acquisition unit 3 uses the vehicle information acquired from the vehicle-side control device 200 to instruct the first information acquisition unit 1 and the second information acquisition unit 2 to start acquiring sensor information and captured images. A signal or a signal indicating that acquisition of sensor information and captured images is to be ended is output.

For example, the vehicle information acquisition unit 3 receives information from the vehicle-side control device 200 such that the door is unlocked, the door is opened, the ignition is turned on, the human sensor is turned on, the shift lever is moved to the drive position, and the vehicle speed is 0 km/h. When any one of the signals is acquired, such as that the navigation device has started guidance, or the vehicle has left home, the sensor information and the captured image are sent to the first information acquisition unit 1 and the second information acquisition unit 2. Outputs a signal to start acquiring the data. On the other hand, for example, the vehicle information acquisition unit 3 receives information from the vehicle-side control device 200 indicating that the ignition is turned off, the human sensor is turned off, the shift lever is moved to the parking position, the navigation device has finished guiding, and the vehicle is heading home. When any one of the signals such as "return" is acquired, a signal indicating that the acquisition of the sensor information and the captured image is to be ended is output to the first information acquisition section 1 and the second information acquisition section 2.

The occupant detection unit 12 of the vehicle monitoring device 10 will be explained. The occupant detection section 12 includes a first detection section 4 that detects an occupant using sensor information, a second detection section 5 that detects an occupant using a captured image, and a first detection section 4 and a second detection section 5. The vehicle also includes a presence/absence determination section 6 that determines whether or not an occupant is present in the rear seat using the detection result. Note that, as shown in FIG. 1, the first detection section 4 and the second detection section 5 of the occupant detection section 12 are connected to the first information acquisition section 1 and the second information acquisition section 2 of the information acquisition section 11, respectively. There is.

The process of detecting a rear seat occupant by the occupant detection unit 12 will be described. In the following description, an example will be given in which the first detection unit 4 acquires distance data and angle data acquired from the radio wave sensor 21 as sensor information, and detects an occupant in the rear seat.

First, the first detection unit 4 acquires distance data and angle data as sensor information from the first information acquisition unit 1 of the information acquisition unit 11, and determines the size of the detection target and the detection target such as a person or animal. Calculate the existing range etc. For example, if the first detection unit 4 determines that the detection target exists on the left side of the rear seat based on the calculated size of the detection target and the range in which it exists, the first detection unit 4 generates a detection result indicating that an occupant is detected on the left side of the rear seat. is derived, and the detection result is output to a storage unit (not shown) of the vehicle monitoring device 10.

In addition, if the first detection unit 4 determines that the detection target exists on the right side of the rear seat based on the calculated size of the detection target and the range in which it exists, the first detection unit 4 determines that an occupant has been detected on the right side of the rear seat. Output the detection results to the storage unit. Similarly, for example, if the first detection unit 4 determines that the detection target exists in the center of the rear seat based on the calculated size of the detection target and the range in which it exists, the first detection unit 4 detects that an occupant is detected in the center of the rear seat. Outputs the detection result. Note that the occupant detection processing by the first detection unit 4 is not limited to the above-mentioned example, and various known algorithms can be used. Further, in the above example, the detection result of the first detection unit 4 is stored in the storage unit, but the first detection unit 4 may output the detection result to the presence/absence determination unit 6.

Here, the presence of an occupant includes not only a state in which an occupant is strictly sitting on a seat, but also a state in which an occupant is present at the foot of the seat. There is a possibility that small-sized occupants, such as infants, are present at the foot of the seat. For example, if the occupant is detected using the sensor information acquired from the radio wave sensor 21, it is possible to detect the occupant present at the foot of the seat. It's for a reason. Hereinafter, for the sake of explanation, the occupant detection process using the sensor information by the first detection unit 4 will be referred to as the first occupant detection process. Further, the detection result of the first occupant detection process by the first detection unit 4 is referred to as a first detection result.

By the way, when detecting a passenger in a rear seat using sensor information, the reliability of the detection result may become a problem. For example, if the sensor 20 is a radio wave sensor 21, the resolution of distance data, angle data, etc. acquired from the radio wave sensor 21 cannot be guaranteed, and multiple occupants seated in different seats may be detected in succession. In some cases, it may not be possible to accurately determine which seat the occupant is in. For example, if the sensor 20 is a seating sensor, the weight of the occupant sitting on the seat may not be accurately measured depending on the posture of the occupant, and it may be difficult to accurately measure the weight of the occupant seated on the seat. It may not be possible to determine whether

On the other hand, when the occupant is detected using the imaging device 30, a more reliable detection result can be obtained than when the occupant is detected using the sensor 20 different from the imaging device 30. Therefore, regarding the detection of a rear seat occupant, when the occupant is detected from the captured image, the accuracy of occupant detection can be improved by using the detection result of the occupant detection processing using the captured image.

Passenger detection processing using captured images will be described. FIG. 3 is an explanatory diagram showing an example of occupant detection using captured images according to the first embodiment. FIG. 3 shows a captured image acquired from the imaging device 30, in which an occupant detection process of the second detection unit 5, which will be described later, detects an occupant 56 seated in the driver's seat 51, an occupant 57 seated in the passenger seat 52, and a passenger 57 seated in the passenger seat 52. An example is shown in which an occupant 58 seated on the seat 53 on the left side of the seats and an occupant 60 seated on the seat 55 in the center of the rear seat are detected.

An example of occupant detection processing by the second detection unit 5 will be described. First, the second detection unit 5 analyzes the captured image and detects the face of the occupant in the captured image. Then, the second detection unit 5 acquires an area where the face of the occupant has been detected (an area indicated by a broken line in FIG. 3; hereinafter referred to as a face area) and characteristic information of the occupant in the face area. Here, the occupant characteristic information is, for example, the contrast ratio of the eyes, nose, mouth, and cheeks after normalizing the size of the face. Note that the second detection unit 5 may determine whether or not face detection has been successful using the occupant's characteristic information. In this case, for example, if the brightness distribution of the face area is found to be similar to a face based on the contrast ratio within the face area, the second detection unit 5 determines that face detection has been successful, and the brightness distribution is not similar to a face. If it is determined that the face detection has failed, it may be determined that the face detection has failed.

After detecting the occupant's face, the second detection unit 5 acquires coordinates of a facial area surrounding the occupant's face, such as a rectangle touching the outline of the occupant's face, from the captured image. Here, the coordinates related to the face area are, for example, when the face area is rectangular, the coordinates of each vertex, center, etc. of the rectangle. Furthermore, the second detection unit 5 calculates dimensions such as the width, height, and area of the face area from the coordinates of the face area.

Then, the second detection unit 5 identifies the seat where the occupant is present from the coordinates, width, height, area, etc. of the acquired face area, and detects information regarding the presence or absence of the occupant and the seat where the occupant is present. As a result, it is output to the storage section of the vehicle monitoring device 10. Note that the occupant detection processing by the second detection unit 5 is not limited to the above-mentioned example, and various known algorithms can be used.

Further, in the above example, the detection result of the second detection section 5 is stored in the storage section, but the second detection section 5 may output the detection result to the presence/absence determination section 6. Hereinafter, for the sake of explanation, the occupant detection process using the captured image by the second detection unit 5 will be referred to as the second occupant detection process. Furthermore, the detection result of the second occupant detection process by the second detection unit 5 is referred to as a second detection result.

Next, the presence/absence determination for determining whether or not an occupant is present will be explained. The presence/absence determining unit 6 of the present embodiment determines that there is a passenger in the rear seat if the passenger in the rear seat can be detected from the captured image even if the passenger in the rear seat is not detected from the sensor information. In this way, if the detection result regarding the presence or absence of a rear seat occupant obtained from the sensor information differs from the detection result regarding the presence or absence of a rear seat occupant obtained from the captured image, the occupant detection unit 12 detects the presence or absence of the rear seat occupant from the captured image. The presence or absence of a rear seat occupant is determined using the obtained detection results regarding the presence or absence of a rear seat occupant.

The operation of the vehicle monitoring device 10 will be explained below. FIG. 4 is a flowchart showing an example of the operation of the vehicle monitoring device 10 according to the first embodiment. Further, although the flowchart of FIG. 4 does not show the process of terminating the operation of the vehicle monitoring device 10, the vehicle monitoring device 10 is configured such that, for example, the vehicle information acquisition unit 3 When a signal indicating that the engine has stopped is acquired, the operations of the first occupant detection process and the second occupant detection process are ended. In addition, in the following explanation, even if there are multiple rear seats, the rear seats are collectively referred to as rear seats, but if there are multiple rear seats, the rear seats are referred to as rear seats. The process shall be performed for each seat.

First, after the operation of the vehicle monitoring device 10 starts, the first information acquisition unit 1 of the vehicle monitoring device 10 acquires sensor information from the sensor 20 (ST101). Next, the first detection unit 4 performs a first occupant detection process using the sensor information (ST102). Here, in the process of ST102, the first detection unit 4 performs the first occupant detection process on each of the rear seats, that is, on each of the rear seats provided in the vehicle. Note that the first detection unit 4 determines whether or not an occupant is detected for each of the rear seats based on, for example, the size and presence of the detection target calculated from the distance data and angle data acquired as sensor information. The determination may be made based on which seat among the rear seats the detected object is present, based on the range etc.

The first detection unit 4 then outputs a detection result indicating whether or not a rear seat occupant has been detected to the storage unit of the vehicle monitoring device 10. When an occupant is detected in the rear seat (ST102; YES), the first detection unit 4 stores the detection result indicating that the occupant in the rear seat has been detected as the first detection result in the storage unit of the vehicle monitoring device 10. output, and turns on the rear seat first flag (ST103). On the other hand, if the occupant is not detected in the rear seat (ST102; NO), the first detection unit 4 sends the detection result indicating that the occupant is not detected to the vehicle monitoring device 10 as the first detection result. , and turns off the first flag in the rear seat (ST104).

Here, the rear seat first flag stored in the storage unit indicates whether or not an occupant is detected in the rear seat by the first occupant detection process. That is, if the first detection unit 4 outputs the first detection result indicating that the rear seat occupant has been detected, the rear seat first flag is turned ON, and the first detection unit 4 outputs the first detection result indicating that the rear seat occupant has been detected. If the first detection result indicating that the vehicle was not detected is output, the first rear seat flag is turned OFF.

In addition, it is assumed that a plurality of rear seat first flags or a single rear seat flag are provided depending on the number of rear seats. For example, if the number of rear seats is three, and the vehicle has a seat on the left side of the rear seat, a seat on the right side of the rear seat, and a seat in the center of the rear seat, the first flag for the rear seat is the left side of the rear seat. The first flag, the first flag on the right side of the rear seat, and the first flag in the center of the rear seat.

Next, the second information acquisition unit 2 of the vehicle monitoring device 10 acquires a captured image from the imaging device 30 (ST105). The second detection unit 5 performs a second occupant detection process using the captured image (ST106). Here, in the process of ST106, the second detection unit 5 detects the occupant in each of the rear seats, that is, each of the rear seats provided in the vehicle. Note that the second detection unit 5 determines whether or not an occupant is detected for each of the rear seats based on, for example, characteristic information detected from the captured image, coordinates of the face area, width, height, and The determination may be made based on which seat among the rear seats the detected object is present based on the area or the like.

Then, the second detection unit 5 outputs the detection result as to whether or not a rear seat occupant is detected to the storage unit of the vehicle monitoring device 10. If an occupant is detected in the rear seat (ST106; YES), the second detection unit 5 stores the detection result indicating that the occupant in the rear seat has been detected as a second detection result in the storage unit of the vehicle monitoring device 10. output, and turns on the second rear seat flag (ST107). On the other hand, when the occupant is not detected in the rear seat (ST106; NO), the second detection unit 5 transmits the detection result indicating that the occupant in the rear seat is not detected to the vehicle monitoring device 10 as a second detection result. , and turns off the second rear seat flag (ST108).

Here, the second rear seat flag stored in the storage unit indicates whether or not an occupant is detected in the rear seat by the second occupant detection process. That is, if the second detection unit 5 outputs a second detection result indicating that a rear seat occupant has been detected, the second rear seat flag is turned ON, and the second detection unit 5 outputs a second detection result indicating that a rear seat occupant has been detected. If the second detection result indicating that the vehicle was not detected is output, the second rear seat flag is turned OFF.

In addition, it is assumed that a plurality of rear seat second flags or a single rear seat flag are provided depending on the number of rear seats. For example, if the number of rear seats is three, and the vehicle has a seat on the left side of the rear seat, a seat on the right side of the rear seat, and a seat in the center of the rear seat, the second rear seat flag is The second flag, the second flag on the right side of the rear seat, and the second flag in the center of the rear seat.

Next, the presence/absence determination unit 6 of the vehicle monitoring device 10 acquires the first detection result from the first detection unit 4 and the second detection result from the second detection unit 5. Then, the presence/absence determination unit 6 determines whether or not an occupant is present in the rear seat using the first detection result and the second detection result.

The presence/absence determining unit 6 refers to the rear seat first flag as the first detection result and determines whether or not the rear seat first flag is ON (ST109), and determines whether or not the rear seat occupant is detected from the sensor information. Check whether the Then, if the rear seat first flag is ON (ST109; YES), that is, if it is confirmed that a rear seat occupant is detected from the sensor information, the presence/absence determining unit 6 determines that there is a rear seat occupant. The determination result is output to the vehicle-side control device 200 (ST110). Note that the determination result by the presence/absence determination unit 6 is output to a notification control unit (not shown) of the vehicle-side control device 200 that controls, for example, a seatbelt reminder, and the notification control unit determines whether there is an occupant in the rear seat. If the determination result is received, the seatbelt reminder function for the rear seats is turned on.

On the other hand, if the rear seat first flag is OFF (ST109; NO), that is, if it is confirmed from the sensor information that no occupant is detected in the rear seat, the presence/absence determining unit 6 performs ST111, which will be described next. Proceed to processing.

The presence/absence determining unit 6 refers to the second rear seat flag as the second detection result and determines whether or not the second rear seat flag is ON (ST111), and determines whether the rear seat occupant is detected from the captured image. Check whether it is correct or not. Then, if the second rear seat flag is ON (ST111; YES), that is, if it is confirmed that the passenger in the rear seat is detected from the captured image, the presence/absence determining unit 6 determines that there is a passenger in the rear seat. The determination result is output to the vehicle-side control device 200 (ST110).

On the other hand, if the second rear seat flag is OFF (ST111; NO), that is, if it is confirmed that no passenger in the rear seat is detected from the captured image, the presence/absence determining unit 6 determines that there is a passenger in the rear seat. It is determined not to do so, and the determination result is output to the vehicle-side control device 200 (ST112).

In this way, the occupant detection unit 12 obtained a detection result indicating that an occupant was not detected in the rear seat from the sensor information, and obtained a detection result indicating that an occupant in the rear seat was detected from the captured image. If so, it is determined that there is a passenger in the rear seat. As a result, even if a rear seat occupant is actually present in the rear seat, even if the rear seat occupant is not detected from the sensor information, if a rear seat occupant is detected from the captured image, the captured image Since the detection results obtained from the above are used to identify the presence of an occupant in the rear seat, it is possible to prevent failure to detect the occupant in the rear seat and improve the detection accuracy of the occupant in the rear seat, that is, the target person.

Note that the second detection unit 5 may perform the second occupant detection process on a seat among the rear seats in which the first detection result indicates that no occupant was detected. . In this way, detection failures can be suppressed, and the processing load can be reduced because the second occupant detection process is performed for seats where there is a possibility of detection failure. Furthermore, in the flowchart of FIG. 4, the vehicle monitoring device 10 performs the processing of ST101 to ST104 and then the processing of ST105 to ST108. However, the flowchart of FIG. 4 is an example, and for example, The processing in ST101 to ST104 and the processing in ST105 to ST108 may be performed in parallel.

Next, a hardware configuration that implements the functions of the vehicle monitoring device 10 will be described. FIG. 5 is a diagram showing an example of the hardware configuration of the vehicle monitoring device 10 according to the first embodiment. First information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first detection unit 4, second detection unit 5, presence/absence determination unit 6, occupant detection unit in vehicle monitoring device 10 12 and the functions of the storage section are realized by a processing circuit. That is, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first detection section 4, the second detection section 5, the presence/absence determination section 6, of the vehicle monitoring device 10, The occupant detection unit 12 and the storage unit may be a processing circuit 10a that is dedicated hardware as shown in FIG. 5A, or a processor that executes a program stored in a memory 10c as shown in FIG. 5B. 10b.

As shown in FIG. 5A, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first detection section 4, the second detection section 5, the presence/absence determination section 6, the occupant When the detection unit 12 and the storage unit are dedicated hardware, the processing circuit 10a may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), or an FPGA. (Field-programmable Gate Array), or a combination of these. First information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first detection unit 4, second detection unit 5, presence/absence determination unit 6, occupant detection unit 12, and storage unit The functions of each part may be realized by a processing circuit, or the functions of each part may be realized by a single processing circuit.

As shown in FIG. 5B, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first detection section 4, the second detection section 5, the presence/absence determination section 6, the occupant When the detection section 12 and the storage section are the processor 10b, the functions of each section are realized by software, firmware, or a combination of software and firmware. Software or firmware is written as a program and stored in the memory 10c. The processor 10b reads and executes the program stored in the memory 10c, thereby controlling the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first detection section 4, The functions of the second detection section 5, presence/absence determination section 6, occupant detection section 12, and storage section are realized. That is, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first detection section 4, the second detection section 5, the presence/absence determination section 6, the occupant detection section 12, and The storage unit includes a memory 10c for storing a program that, when executed by the processor 10b, results in each step shown in FIG. 4 being executed. In addition, these programs include the first information acquisition unit 1, the second information acquisition unit 2, the vehicle information acquisition unit 3, the information acquisition unit 11, the first detection unit 4, the second detection unit 5, the presence/absence determination unit 6, and the occupant It can also be said that the computer is caused to execute the procedures or methods of the detection unit 12 and the storage unit.

Here, the processor 10b is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic device, a processor, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor). The memory 10c may be a nonvolatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM). However, it may be a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a mini disk, a CD (Compact Disc), or a DVD (Digital Versatile Disc).

Note that the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first detection section 4, the second detection section 5, the presence/absence determination section 6, the occupant detection section 12, and Regarding each function of the storage unit, some may be realized by dedicated hardware, and some may be realized by software or firmware. In this way, the processing circuit 10a in the vehicle monitoring device 10 can implement the above-mentioned functions using hardware, software, firmware, or a combination thereof. Further, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first detection section 4, the second detection section 5, the presence/absence determination section 6, the occupant detection section 12, and At least some of the functions of the storage unit may be performed by an external server.

In this way, the vehicle monitoring device 10 includes the first information acquisition unit 1 that acquires sensor information from the sensor 20 whose detection range includes the rear seats of the vehicle, and the imaging device whose imaging range includes the front and rear seats of the vehicle. 30, a second information acquisition unit 2 that acquires a captured image, and an occupant detection unit that detects a rear seat occupant from the sensor information and captured images respectively acquired by the first information acquisition unit 1 and the second information acquisition unit 2. If the detection result regarding the presence or absence of a rear seat occupant obtained from the sensor information is different from the detection result regarding the presence or absence of a rear seat occupant obtained from the captured image, the occupant detection section 12 includes: If the detection result regarding the presence or absence of a rear seat occupant obtained from a captured image is used to identify the presence or absence of a rear seat occupant, it will be possible to prevent failure to detect a rear seat occupant, and to identify the presence or absence of a rear seat occupant. The accuracy of detecting people can be improved.

Embodiment 2.
Similarly to the first embodiment, the vehicle monitoring device 10 according to the second embodiment includes a first information acquisition unit 1 that acquires sensor information, a second information acquisition unit 2 that acquires captured images, and a first information acquisition unit 1 that acquires sensor information. The vehicle also includes an occupant detection section 12 that detects an occupant in the rear seat from the sensor information and captured images acquired by the section 1 and the second information acquisition section 2, respectively. In the present embodiment, when the second detection unit 5 does not detect a passenger in the rear seat from the captured image, the second detection unit 5 uses the captured image to determine whether or not there is a passenger in the rear seat. The processing operations of the vehicle monitoring device 10 are different. Components that are the same as those in Embodiment 1 are given the same reference numerals, and their descriptions will be omitted. Further, the target person for detection by the vehicle monitoring system 100 according to the present embodiment is a rear seat occupant. Note that the vehicle monitoring system 100 does not detect only rear seat occupants, and may include front seat occupants as target persons.

In the first occupant detection process using a sensor 20 different from the imaging device 30, for example, the sensor 20 is a radio wave sensor 21, and the radio wave sensor 21 detects that a reflected wave is reflected from a moving object present on the seat. For example, it is determined that there is a detection target on the seat. In such a case, even though the moving object on the seat is, for example, water in a plastic bottle, which is not an occupant, it is determined that the occupant is on the seat based on the reflected waves from the plastic bottle. There is a possibility that it will be falsely detected if it exists. For example, if the sensor 20 is a seating sensor, even if the seating sensor is able to correctly measure the weight of an object on the seat, depending on the weight of the object on the seat, the object on the seat may be an object other than the occupant. However, there is a risk of false detection if there is an occupant in the seat.

As described above, when an occupant is detected using a sensor 20 that is different from the imaging device 30, there is a risk that it will be falsely detected that an occupant is present in the seat, even though there is actually no occupant present in the seat. However, the accuracy of passenger detection cannot be said to be sufficient.

Therefore, in the vehicle monitoring device 10 of the present embodiment, even if the first detection unit 4 indicates a detection result that a passenger is present in the rear seat based on the sensor information, the second detection unit 5 uses the captured image. If it is determined through the second occupant detection process that there is no occupant in the rear seat, priority is given to the detection result using the captured image, and it is determined that there is no occupant in the rear seat. The occupant detection process by the second detection unit 5 of this embodiment will be described below.

In addition to detecting the occupant in the rear seat from the captured image, the second detection unit 5 of the present embodiment detects the presence of an occupant in the rear seat from the captured image if the occupant in the rear seat is not detected from the captured image. Determine whether or not.

FIG. 6 is an explanatory diagram showing an example of occupant detection using captured images according to the second embodiment. In the captured image shown in FIG. 6, occupants 56 and 57 are seated in the driver's seat 51 and passenger seat 52, respectively, of the vehicle, and occupants 58 and 57 are seated in the rear left seat 53 and the rear right seat 54, respectively. 59 is imaged in a seated state. Furthermore, in FIG. 6, no passenger is seated on the seat 55 at the center of the rear seat, but a child seat 61 is arranged.

In the example of FIG. 6, for example, when the sensor 20 is a seating sensor, the detection result of the first occupant detection process using sensor information is detected by the child seat 61 placed on the seat 55 in the center of the rear seat, as described above. , it may indicate that an occupant has been detected in the seat 55 in the center of the rear seat. On the other hand, the captured image shows that there is no occupant in the seat 55 in the center of the rear seat, and that a child seat 61 is placed in the seat 55 in the center of the rear seat. Through the detection processing described below, a detection result indicating that an object other than an occupant is present in the rear center seat 55, that is, an occupant is not present in the rear center seat 55, is output.

The second detection unit 5, for example, acquires a captured image from the second information acquisition unit 2, and extracts an image in a set area from the captured image. Note that the set area is, for example, an area set to include a specific seat among the rear seats, such as the seat 55 in the center of the rear seat, as shown in area C in FIG.

Then, the second detection unit 5 compares the extracted image with an image for similarity determination stored in advance in a storage unit, such as an image of the child seat 61 or an image of a plastic bottle. By doing this, the object in the extracted image is identified. Here, the second detection unit 5 can identify the object in the extracted image using a known object recognition method such as instance segmentation or template matching.

If the second detection unit 5 identifies that the object in the image extracted corresponding to a specific seat among the rear seats is, for example, the child seat 61, and is an object other than the passenger, the second detection unit 5 detects the object in the rear seat. A detection result indicating that an object other than the occupant is present, that is, there is no occupant in the rear seat, is output to the storage unit of the vehicle monitoring device 10. On the other hand, there are cases where it is impossible to identify an object on a seat because it is obscured by occupants in seats other than the specific seat. If the object shown in the image extracted in this way is not identified, the second detection unit 5 sends a detection result to the vehicle monitoring device 10 indicating that it is not clear whether or not there is a passenger in the rear seat. output to the storage section. Note that the above-mentioned detection processing is just an example, and it is also possible to specify the object in the captured image using various known algorithms.

Furthermore, the process of determining whether or not a passenger is present in the rear seat by the second detection unit 5 is not limited to the above-mentioned example. In the first embodiment, an example has been described in which the second detection unit 5 performs face detection processing on the captured image. It is also possible to set this and perform face detection processing within the detection area.

When the second detection unit 5 sets a detection area and performs face detection processing within the detection area, for example, the driver's seat 51 or the front passenger seat 52, that is, the front seat occupant, is detected in the detection area provided in the rear seat. If a rear seat occupant is not detected within the detection area without intrusion, it becomes clear that no passenger is seated in the rear seat. This is because if the front seat occupant has not entered the detection area provided in the rear seat, there is a low possibility that the rear seat occupant will be shielded by the front seat occupant.

FIG. 7 is an explanatory diagram showing an example of occupant detection using captured images according to the second embodiment. In the captured image shown in FIG. 7, occupants 56 and 57 are seated in the driver's seat 51 and passenger seat 52, respectively, of the vehicle, and occupants 58 are seated in the rear left seat 53 and the rear right seat 54, respectively. , 59 are photographed in a seated state. Furthermore, in FIG. 7, neither an occupant nor an object is present in the seat 55 in the center of the rear seat.

In the example of FIG. 7, the occupants 56 and 57 in the driver's seat 51 or the passenger seat 52 have not entered the detection area (area E shown in FIG. 7) provided at the center of the rear seat, and The occupant's face is not detected within the detection area provided at the center of the seat. In such a case, even if an occupant is not detected in the center of the rear seat in the detection process using the captured image, it is clear that there is no occupant in the center of the rear seat.

Further, FIG. 7 shows an example in which an occupant 59 present in the seat 54 on the right side of the rear seat is shielded by an occupant 56 in the driver's seat 51. In other words, in the detection area provided for the right side of the rear seat (area D shown in FIG. 7), the passenger 59 on the right side of the rear seat is not detected, while the passenger 56 in the driver's seat 51 is not detected in the detection area provided for the right side of the rear seat. The detection area has been significantly invaded. In the above case, even if an occupant is not detected on the right side of the rear seat in the detection process using the captured image, it is not clear whether or not there is an occupant on the right side of the rear seat.

Therefore, the second detection unit 5 detects, for example, the driver's seat 51 or the passenger's seat that occupies the detection area provided for the rear seat in the captured image, as an example of a process of determining whether or not there is an occupant in the rear seat. If the area of the face area detected from the occupants 56 and 57 in the seat 52 is less than the set threshold, there is no occupant in the rear seat that is not shielded by the occupant in the front seat, that is, there is no occupant in the rear seat. Determine that it does not exist. On the other hand, if the area of the face area detected from the driver or front passenger seat occupant in the detection area provided for the rear seat is greater than or equal to the set threshold, it is determined that there is an occupant in the rear seat. It is judged that it is not clear.

Next, identification of the presence or absence of a passenger in the rear seat will be described. If the detection result regarding the presence or absence of a rear seat occupant obtained from the sensor information is different from the detection result regarding the presence or absence of a rear seat occupant obtained from the captured image, the occupant detection unit 12 detects the presence or absence of the rear seat occupant obtained from the captured image. The presence or absence of a passenger in the rear seat is determined using the detection results regarding the presence or absence of a passenger in the seat. That is, even if the presence/absence determining unit 6 obtains a detection result indicating that an occupant is present in the rear seat from the sensor information from the second detecting unit 5, the presence/absence determination unit 6 may detect a detection result indicating that there is no occupant in the rear seat from the captured image. If a result is obtained, it is determined that there is no occupant in the rear seat using the detection result obtained from the captured image. In this way, even if the detection result that there is a passenger in the rear seat obtained from the sensor information is due to a false detection, if it is clear from the captured image that there is no passenger in the rear seat, then the Since it is determined that no occupant is present in the vehicle, false detections in the occupant detection process can be suppressed and the accuracy of occupant detection can be improved.

The presence/absence determining unit 6 obtains a detection result indicating that there is an occupant in the rear seat from the sensor information, and obtains a detection result indicating that it is not clear whether or not there is an occupant in the rear seat from the captured image. If so, it is determined that there is an occupant in the rear seat.

Next, the operation of the vehicle monitoring device 10 will be explained. FIG. 8 is a flowchart showing an example of the operation of the vehicle monitoring device 10 according to the second embodiment. Hereinafter, steps that are the same as those in the process of the vehicle monitoring device 10 according to the first embodiment will be given the same reference numerals as those shown in FIG. 4, and the explanation will be omitted or simplified. Further, although the flowchart of FIG. 8 does not show the process of terminating the operation of the vehicle monitoring device 10, the vehicle monitoring device 10 is configured such that, for example, the vehicle information acquisition unit 3 When a signal indicating that the engine has stopped is acquired, the operations of the first occupant detection process and the second occupant detection process are ended. In addition, in the following explanation, even if there are multiple rear seats, the rear seats are collectively referred to as rear seats, but if there are multiple rear seats, the rear seats are referred to as rear seats. The process shall be performed for each seat.

First, after the operation of the vehicle monitoring device 10 starts, the first information acquisition unit 1 of the vehicle monitoring device 10 acquires sensor information from the sensor 20 (ST101). Next, the first detection unit 4 performs a first occupant detection process using the sensor information (ST102).

The first detection unit 4 then outputs the detection result as to whether or not a rear seat occupant has been detected to the storage unit of the vehicle monitoring device 10. When an occupant is detected in the rear seat (ST102; YES), the first detection unit 4 stores the detection result indicating that the occupant in the rear seat has been detected as the first detection result in the storage unit of the vehicle monitoring device 10. output, and turns on the rear seat first flag (ST103). On the other hand, if the occupant is not detected in the rear seat (ST102; NO), the first detection unit 4 sends the detection result indicating that the occupant is not detected to the vehicle monitoring device 10 as the first detection result. , and turns off the first flag in the rear seat (ST104).

Next, the second information acquisition unit 2 of the vehicle monitoring device 10 acquires a captured image from the imaging device 30 (ST105). The second detection unit 5 performs a second occupant detection process using the captured image (ST106).

Then, the second detection unit 5 outputs the detection result as to whether or not a rear seat occupant is detected to the storage unit of the vehicle monitoring device 10. If an occupant is detected in the rear seat (ST106; YES), the second detection unit 5 stores the detection result indicating that the occupant in the rear seat has been detected as a second detection result in the storage unit of the vehicle monitoring device 10. output, and turns on the second rear seat flag (ST107). On the other hand, when the occupant is not detected in the rear seat (ST106; NO), the second detection unit 5 transmits the detection result indicating that the occupant in the rear seat is not detected to the vehicle monitoring device 10 as a second detection result. , and turns off the second rear seat flag (ST108).

Then, if an occupant is not detected in the rear seat from the captured image, the second detection unit 5 determines whether or not there is an occupant in the rear seat (ST201). Here, the second detection unit 5 determines whether or not there is a passenger in the rear seat by, for example, extracting an image in a region set to include a specific seat among the rear seats in the captured image. This can be done by specifying whether or not the object shown in the extracted image is a passenger.

If the second detection unit 5 determines that there is no occupant in the rear seat (ST201; YES), it outputs a detection result indicating that there is no occupant in the rear seat to the storage unit, and turns on the third flag in the rear seat. (ST202). On the other hand, when the second detection unit 5 determines that it is not clear whether or not there is a passenger in the rear seat (ST201; NO), the second detection unit 5 stores the detection result indicating that it is not clear whether or not there is a passenger in the rear seat. and turns off the third flag in the rear seat (ST203).

Here, the third rear seat flag stored in the storage unit indicates whether or not there is an occupant in the rear seat. That is, if the second detection unit 5 outputs a detection result indicating that there is no passenger in the rear seat, the third flag in the rear seat is turned ON, and the second detection unit 5 clearly indicates whether or not there is a passenger in the rear seat. If a detection result indicating that the vehicle is not in use is output, the third rear seat flag is turned OFF.

In addition, it is assumed that a plurality of rear seat third flags or a single rear seat third flag are provided depending on the number of rear seats. For example, if the number of rear seats is three, and the vehicle has a seat on the left side of the rear seat, a seat on the right side of the rear seat, and a seat in the center of the rear seat, the third flag for the rear seat is the left side of the rear seat. The third flag, the third flag on the right side of the rear seat, and the third flag in the middle of the rear seat.

Next, the presence/absence determination section 6 of the vehicle monitoring device 10 acquires the second detection result from the second detection section 5, and the presence/absence determination section 6 uses the second detection result to determine whether an occupant is present in the rear seat. Determine whether or not there is. The presence/absence determining unit 6 refers to the second rear seat flag as the second detection result and determines whether or not the second rear seat flag is ON (ST204), and determines whether or not the rear seat occupant is detected from the captured image. Check whether it is correct or not.

Then, if the second rear seat flag is ON (ST204; YES), that is, if it is confirmed that the passenger in the rear seat is detected from the captured image, the presence/absence determining unit 6 determines that there is a passenger in the rear seat. It is determined that it is, and the determination result is output to the vehicle-side control device 200 (ST205). The determination result by the presence/absence determining section 6 is output to the notification control section of the vehicle-side control device 200 that controls, for example, a seatbelt reminder, and the notification control section outputs the determination result that there is an occupant in the rear seat. If received, turn on the seatbelt reminder function for the rear seats.

On the other hand, if the second rear seat flag is OFF (ST204; NO), that is, if no occupant is detected in the rear seat from the captured image, the presence/absence determining unit 6 proceeds to the process described next. First, the presence/absence determination section 6 acquires the first detection result from the first detection section 4 . Then, the presence/absence determining unit 6 refers to the rear seat first flag as the first detection result and determines whether the rear seat first flag is ON (ST206), and based on the sensor information, the rear seat occupant is Check whether it has been detected.

If the rear seat first flag is OFF (ST206; NO), that is, if it is confirmed from the sensor information that no passenger in the rear seat is detected, the presence/absence determining unit 6 determines that there is no passenger in the rear seat. The determination result is output to the vehicle-side control device 200 (ST207).

On the other hand, if the rear seat first flag is ON (ST206; YES), that is, if it is confirmed that a passenger in the rear seat has been detected from the sensor information, the presence/absence determination unit 6 performs the process of ST208, which will be described next. Proceed to.

The presence/absence determination unit 6 refers to the third rear seat flag and determines whether the third rear seat flag is ON (ST208), and determines whether or not there is no occupant in the rear seat from the captured image. Check whether or not. Then, if the third rear seat flag is ON (ST208; YES), that is, if it is determined from the captured image that there is no passenger in the rear seat, the presence/absence determining unit 6 determines that there is no passenger in the rear seat. The determination result is output to the vehicle-side control device 200 (ST207).

On the other hand, if the third rear seat flag is OFF (ST208; NO), the presence/absence determination unit 6 derives a detection result that there is an occupant in the rear seat from the sensor information, and the captured image indicates that there is an occupant in the rear seat. If it is confirmed that it is not clear whether or not there is an occupant, it is determined that there is an occupant in the rear seat, and the determination result is output to the vehicle-side control device 200 (ST205).

In this way, even if the vehicle monitoring device 10 obtains a detection result from the sensor information that there is a passenger in the rear seat even though there is no passenger in the rear seat, the captured image does not indicate that there is a passenger in the rear seat. If it is clear that there is no passenger present in the rear seat, it is determined that there is no passenger present in the rear seat, thereby suppressing false detection and improving the accuracy of detecting the passenger in the rear seat, that is, the target person.

Note that, similarly to the first embodiment, the second occupant detection process may be performed on a seat among the rear seats for which the first detection result indicates that no occupant is present. In this way, since the second occupant detection process is performed for a seat where there is a possibility that detection may be missed in the first occupant detection process, it is possible to suppress failure to detect an occupant, and reduce the processing load on the vehicle monitoring device 10. It can be reduced. Furthermore, in the flowchart of FIG. 8, the vehicle monitoring device 10 performs the processing of ST101 to ST104 and then the processing of ST105 to ST108 and ST201 to ST203, but the flowchart of FIG. 8 is an example. For example, ST101 to ST104, ST105 to ST108, and ST201 to ST203 may be performed in parallel.

In the first and second embodiments, an example has been described in which the occupant detection unit 12 detects an occupant in the rear seat and identifies the presence or absence of the occupant in the rear seat. However, the occupant detection unit 12 detects the occupant in the front seat. The presence or absence of a passenger in the front seat may also be determined by detection. In this case, the sensor 20 may be provided so that the detection range includes the front and rear seats of the vehicle, and the first detection unit 4 may perform detection processing for the occupant in the front seat.

In addition, in the first and second embodiments, the identification of the presence or absence of a front seat occupant by the occupant detection unit 12 is similar to the operation example described using the flowchart of FIG. If a detection result is obtained indicating that a passenger in the front seat was detected from the captured image, it is possible to identify the presence of a passenger in the front seat. It is possible to improve the accuracy of detecting the front seat occupant, that is, the target person.

Furthermore, in Embodiments 1 and 2, the identification of the presence or absence of a front seat occupant by the occupant detection unit 12 is similar to the operation example described using the flowchart of FIG. If a detection result indicating that there is no occupant in the front seat is obtained from the captured image, it is possible to identify that there is no occupant in the front seat. It is possible to suppress false detections and improve the detection accuracy of the front seat occupant, that is, the target person.

In the first and second embodiments, the first detection unit 4 performs an attribute estimation process that uses sensor information to estimate attributes such as the detected occupant's physique, age, or gender, and the state such as posture. At least one of the state estimation processes may be performed. In this case, the sensor 20 is configured with a radio wave sensor 21 or an ultrasonic sensor, and the first detection unit 4 estimates the occupant's physique, posture, etc. using distance data, angle data, etc. acquired from the sensor 20. good. The above-described attribute estimation processing and state estimation processing can use various known algorithms, and detailed description thereof will be omitted.

Furthermore, in the first and second embodiments, an example was described in which the second detection unit 5 detects an occupant using feature information regarding the occupant's face, such as the facial area of the occupant and the position of facial parts in the captured image. The second detection unit 5 may detect the occupant using characteristic information regarding the physique, such as the skeletal points of the occupant in the captured image. Further, the second detection unit 5 performs attribute estimation processing to estimate attributes such as the detected occupant's physique, age, and gender, and posture, drowsiness, physical condition, etc., using characteristic information regarding the occupant's face and physique. At least one of state estimation processing for estimating the state of may be performed. Note that in the attribute estimation process, occupant authentication may be performed to link the detected occupant with a specific individual, and the authentication result may be included in the attributes. The above-described attribute estimation processing and state estimation processing can use various known algorithms, and detailed explanations will be omitted.

As described above, the first detection unit 4 performs at least one of occupant attribute estimation processing and state estimation processing using sensor information, and second detection unit 5 performs occupant attribute estimation processing using captured images. and state estimation processing. In this case, for example, in the first detection process of the first detection unit 4, the resolution of the sensor information acquired from the radio wave sensor 21 cannot be guaranteed, and multiple occupants are detected in succession, and the physique of each occupant is estimated. Even if this cannot be performed, the reliability of the attribute estimation results and the state estimation results can be improved, such as by being able to estimate the above-mentioned physiques of the respective occupants using the captured images by the second detection unit 5.

Embodiment 3.
Vehicle monitoring device 80 according to Embodiment 3, like Embodiment 1, includes a first information acquisition section 1 that acquires sensor information and a second information acquisition section 2 that acquires captured images. In this embodiment, the vehicle monitoring device 80 includes an outside detection unit 81 that detects a person outside the vehicle 50 (hereinafter referred to as a person outside the vehicle), and is different from the first embodiment in that it detects a person outside the vehicle. different. Components that are the same as those in Embodiment 1 are given the same reference numerals, and their descriptions will be omitted.

FIG. 9 is a block diagram showing a configuration example of a vehicle monitoring system 101 according to the third embodiment. The vehicle monitoring system 101 includes a vehicle monitoring device 80, a sensor 20, and an imaging device 30, and the vehicle monitoring device 80, the sensor 20, and the imaging device 30 are each mounted on the vehicle 50.

The vehicle monitoring device 80 also includes an outside detection section 81 that detects a person outside the vehicle. The vehicle exterior detection unit 81 includes a first vehicle exterior detection unit 7, a second vehicle exterior detection unit 8, and an exterior presence determination unit 9, and detects a person outside the vehicle. The details of each component included in the vehicle exterior detection section 81 will be described later. That is, the target person for detection by the vehicle monitoring system 101 according to the present embodiment is a person outside the vehicle. Note that the vehicle monitoring system 101 does not only detect a person outside the vehicle, but may also include a front seat occupant or a rear seat occupant as target persons for detection.

The sensor 20 in this embodiment is a sensor 20 different from the imaging device 30, such as a radio wave sensor 21 or an ultrasonic sensor, and is designed to include the outside of the vehicle in its detection range, that is, to be able to detect a person outside the vehicle. It is provided on the ceiling of the vehicle interior, the exterior of the vehicle 50, etc. When the sensor 20 is a radio wave sensor 21 or an ultrasonic sensor, one or more sensors are installed on the ceiling of the vehicle interior, etc. so that the detection range includes a range separated by a set distance from the exterior of the vehicle 50. Note that the set distance is, for example, about 15 cm, and can be set as appropriate as long as a person existing near the vehicle 50 outside the vehicle can be detected.

Further, one or more imaging devices 30 are disposed in an overhead console, an instrument panel, a steering column, a rearview mirror, etc. so as to be able to capture an image of the outside of the vehicle through a window of the vehicle 50. FIG. 10 is an explanatory diagram showing an example of detection of a person outside the vehicle by the vehicle monitoring device 80 according to the third embodiment. FIG. 10 is a top view of the interior and exterior of a vehicle 50 equipped with a vehicle monitoring device 80, and shows an example in which a person 70 outside the vehicle is present near the right side of the rear seat. Further, region F in FIG. 10 indicates the imaging range of the imaging device 30, and as shown in FIG. 10, the imaging device 30 includes the outside of the vehicle in the imaging range.

Returning to FIG. 9, the vehicle monitoring device 80 will be explained. The information acquisition unit 11 of the vehicle monitoring device 80 includes a first information acquisition unit 1 and a second information acquisition unit 2, as in the first embodiment. The first information acquisition unit 1 of the information acquisition unit 11 is connected to the sensor 20 and acquires sensor information from the sensor 20. Here, the sensor information is information regarding the presence or absence of a person outside the vehicle. For example, when the sensor 20 is the radio wave sensor 21, the sensor information includes distance data indicating the distance between the radio wave sensor 21 and the detection target, angle data indicating the angle of the detection target with respect to the radio wave sensor 21, and the like. For example, when the sensor 20 is a voice sensor, the sensor information is voice data indicating the volume, direction of arrival, etc. of the voice uttered by a person outside the vehicle. Note that the sensor information is any one of these or a combination of a plurality of them.

On the other hand, the second information acquisition unit 2 of the information acquisition unit 11 is connected to the imaging device 30 and acquires a captured image from the imaging device 30. Note that in the example of FIG. 9, the first information acquisition unit 1 and the second information acquisition unit 2 are illustrated separately, but the acquisition of the captured image from the captured image and the acquisition of sensor information from the sensor 20 are carried out in one step. It may be done in one configuration. The information acquisition unit 11 then outputs the acquired sensor information and captured image to the vehicle exterior detection unit 81.

The first outside detection section 7 of the outside detection section 81 detects a person outside the vehicle using sensor information. Further, the second outside detection unit 8 of the outside detection unit 81 detects a person outside the vehicle using the captured image. Further, the vehicle exterior presence/absence determination section 9 of the vehicle exterior detection section 81 uses the detection results of the first vehicle exterior detection section 7 and the second vehicle exterior detection section 8 to determine whether or not a person exists outside the vehicle. 9.

Next, a process of detecting a person outside the vehicle by the vehicle exterior detection unit 81 will be described. In the following description, an example will be given in which the first outside detection unit 7 detects a person outside the vehicle using distance data and angle data acquired from the radio wave sensor 21 as sensor information.

First, the first outside detection unit 7 acquires distance data and angle data as sensor information from the first information acquisition unit 1 of the information acquisition unit 11, and determines the size of the detection target such as a person or animal. Calculate the range etc. where exists. For example, if the first vehicle exterior detection unit 7 determines that the detection target exists near the vehicle 50, such as near the window of the vehicle 50 outside the vehicle, based on the calculated size of the detection target and the range in which it exists, A detection result indicating that a person exists outside the vehicle is derived, and the detection result is output to the storage unit of the vehicle monitoring device 80. Hereinafter, for the sake of explanation, the detection process using the sensor information by the first vehicle exterior detection unit 7 will be referred to as the first vehicle exterior detection process. Moreover, the detection result of the first vehicle exterior detection process by the first vehicle exterior detection unit 7 is referred to as a first vehicle exterior detection result.

By the way, when detecting a person outside the vehicle using sensor information, the reliability of the detection result may become a problem. For example, if the sensor 20 is a radio sensor 21, depending on the positional relationship between the radio sensor 21 and the person outside the vehicle, the person outside the vehicle may be detected larger than necessary, or the body of the person outside the vehicle may not be detected. Therefore, it may not be possible to accurately determine whether the detection target is a person or not.

On the other hand, when detecting a person outside the vehicle using the imaging device 30, a more reliable detection result can be obtained than when detecting a person outside the vehicle using a sensor 20 different from the imaging device 30. . Therefore, regarding the detection of a person outside the vehicle, when a person outside the vehicle is detected from the captured image, the detection result of the detection process of the person outside the vehicle using the captured image is used to detect the person outside the vehicle, that is, the target person. Accuracy can be improved.

An example of a detection process by the second outside-vehicle detection unit 8 will be described. First, the second outside-of-vehicle detection unit 8 analyzes the captured image and detects the face of a person outside the vehicle in the captured image. Then, the second outside-of-vehicle detection unit 8 acquires a face area, which is an area where the face of a person outside the vehicle has been detected, and characteristic information of the face of the person outside the vehicle in the face area. Here, the facial feature information of the person outside the vehicle is, for example, the contrast ratio of the eyes, nose, mouth, and cheeks after normalizing the size of the face. Note that the second outside-vehicle detection unit 8 may determine whether or not face detection has been successful using facial feature information of a person outside the vehicle. In this case, if the second outside-vehicle detection unit 8 determines that the brightness distribution of the face area is similar to a face based on the contrast ratio within the face area, it determines that face detection has been successful and the brightness distribution is similar to a face. If it is determined that there is no face detection, it may be determined that face detection has failed.

When detecting the face of a person outside the vehicle, the second outside detection unit 8 acquires coordinates related to a face area surrounding the face of the person outside the vehicle, such as a rectangle touching the outline of the face of the person outside the vehicle, from the captured image. do. Here, the coordinates related to the face area are, for example, when the face area is rectangular, the coordinates of each vertex, center, etc. of the rectangle. Furthermore, the second vehicle exterior detection unit 8 calculates dimensions such as the width, height, and area of the face area from the coordinates of the face area.

Then, the second vehicle exterior detection unit 8 identifies whether or not there is a person outside the vehicle based on the coordinates, width, height, area, etc. of the acquired face area, and provides information regarding the presence or absence of the person outside the vehicle as a detection result. , is output to the storage unit of the vehicle monitoring device 80. Note that the detection processing by the second outside-vehicle detection unit 8 is not limited to the above-mentioned example, and various known algorithms can be used. Further, the second outside-vehicle detection unit 8 may detect a person outside the vehicle using characteristic information regarding the physique, such as a skeletal point of the person outside the vehicle in the captured image.

Further, in the above example, the detection result of the second outside detection section 8 is stored in the storage section, but the second outside detection section 8 may also output the detection result to the presence/absence determination section 9 outside the vehicle. good. Hereinafter, for the sake of explanation, the detection process using the captured image by the second vehicle exterior detection unit 8 will be referred to as the second vehicle exterior detection process. Furthermore, the detection result of the second exterior detection process by the second exterior detection unit 8 is referred to as a second exterior detection result.

Next, the presence/absence determination process for determining whether or not a person exists outside the vehicle will be described. Even if the person outside the vehicle is not detected from the sensor information, the presence/absence determination unit 9 determines that there is a person outside the vehicle if the person outside the vehicle can be detected from the captured image. In this way, when the detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information differs from the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image, the vehicle exterior detection unit 81 detects the presence or absence of the person outside the vehicle obtained from the captured image. The presence or absence of a person outside the vehicle is determined using the detection results regarding the presence or absence of a person outside the vehicle.

The operation of the vehicle monitoring device 80 will be explained below. FIG. 11 is a flowchart showing an example of the operation of the vehicle monitoring device 80 according to the third embodiment. For example, when the vehicle information acquisition unit 3 acquires a signal indicating that the engine of the vehicle 50 has started from the vehicle-side control device 200, the vehicle monitoring device 80 performs the operations of the first exterior detection process and the second exterior detection process. Start.

Further, although the flowchart of FIG. 11 does not show the process of terminating the operation of the vehicle monitoring device 80, the vehicle monitoring device 80, for example, When a signal indicating that the engine has stopped is acquired, the operations of the first exterior detection process and the second exterior detection process are terminated.

First, after the operation of the vehicle monitoring device 80 starts, the first information acquisition unit 1 of the vehicle monitoring device 80 acquires sensor information from the sensor 20 (ST301). Next, the first vehicle exterior detection unit 7 performs a first vehicle exterior detection process using the sensor information (ST302). Here, the first outside detection unit 7 determines whether a person outside the vehicle has been detected based on, for example, the size and existing range of the detection target calculated from the distance data and angle data acquired as sensor information. This may be done by determining whether or not the detected object exists outside the vehicle.

The first vehicle exterior detection unit 7 then outputs a detection result indicating whether or not a person outside the vehicle has been detected to the storage unit of the vehicle monitoring device 80. When a person is detected outside the vehicle (ST302; YES), the first outside detection unit 7 stores the detection result indicating that a person outside the vehicle has been detected in the storage unit of the vehicle monitoring device 80 as the first outside detection result. output and turn on the first flag outside the vehicle (ST303). On the other hand, if a person is not detected outside the vehicle (ST302; NO), the first outside detection unit 7 transmits the detection result indicating that no person outside the vehicle has been detected to the vehicle monitoring device 80 as the first outside detection result. The first flag outside the vehicle is turned OFF (ST304).

Here, the first flag outside the vehicle stored in the storage unit indicates whether a person is detected outside the vehicle by the first person detection process. That is, if the first outside detection unit 7 outputs the first outside detection result indicating that a person outside the vehicle has been detected, the first outside flag is turned ON, and the first outside detection unit 7 outputs the first outside detection result indicating that a person outside the vehicle has been detected. If the first outside-vehicle detection result indicating that there is no such thing is output, the first outside-vehicle flag is turned OFF.

Next, the second information acquisition unit 2 of the vehicle monitoring device 80 acquires a captured image from the imaging device 30 (ST305). The second vehicle exterior detection unit 8 performs a second vehicle exterior detection process using the captured image (ST306). Here, the determination as to whether or not a person is detected outside the vehicle by the second outside detection unit 8 is based on, for example, the coordinates, width, height, area, etc. of the face area acquired from the captured image. This can be done by determining whether or not it exists.

Then, the second outside-vehicle detection unit 8 outputs the detection result of whether a person outside the vehicle has been detected to the storage unit of the vehicle monitoring device 80. When a person is detected outside the vehicle (ST306; YES), the second outside detection unit 8 stores the detection result indicating that a person outside the vehicle has been detected in the storage unit of the vehicle monitoring device 80 as the second outside detection result. output and turn on the second flag outside the vehicle (ST307). On the other hand, if a person is not detected outside the vehicle (ST306; NO), the second vehicle exterior detection unit 8 sends a detection result indicating that a person outside the vehicle was not detected to the vehicle monitoring device 8 as a second vehicle exterior detection result. The second flag outside the vehicle is turned OFF (ST308).

Here, the second outside-vehicle flag stored in the storage unit indicates whether or not a person has been detected outside the vehicle by the second outside-vehicle detection process. That is, if the second outside detection unit 8 outputs the second outside detection result indicating that a person outside the vehicle has been detected, the second outside flag is turned ON, and the second outside detection unit 8 outputs the second outside detection result indicating that a person outside the vehicle has been detected. If the second outside-vehicle detection result indicating that there is no such thing is output, the second outside-vehicle flag is turned OFF.

Next, the vehicle exterior presence/absence determining section 9 of the vehicle monitoring device 80 acquires the first vehicle exterior detection result from the first vehicle exterior detection section 7 and the second vehicle exterior detection result from the second vehicle exterior detection section 8 . Then, the vehicle exterior presence/absence determining unit 9 determines whether or not a person exists outside the vehicle using the first vehicle exterior detection result and the second vehicle exterior detection result.

The vehicle exterior presence/absence determination unit 9 refers to the vehicle exterior first flag as the first vehicle exterior detection result and determines whether or not the vehicle exterior first flag is ON (ST309), and determines whether a person outside the vehicle is detected from the sensor information. Check whether or not. Then, if the first flag outside the vehicle is ON (ST309; YES), that is, if it is confirmed that a person outside the vehicle has been detected from the sensor information, the outside presence determination unit 9 determines that there is a person outside the vehicle. , and outputs the determination result to the vehicle-side control device 200 (ST310).

Note that the determination result by the presence/absence determination unit 9 outside the vehicle is output to a notification unit (not shown) of the vehicle-side control device 200 that notifies, for example, that a person exists outside the vehicle. When the notification unit receives the determination result that there is a person outside the vehicle, the notification unit notifies the driver etc. of the presence of a person outside the vehicle from the notification unit of the vehicle or a terminal (not shown) owned by the driver. inform. In this way, the driver of the vehicle 50 can recognize that a person such as a suspicious person exists near the vehicle.

On the other hand, if the outside-vehicle first flag is OFF (ST309; NO), that is, if it is confirmed from the sensor information that no person has been detected outside the vehicle, the outside-vehicle existence determination unit 9 performs the process of ST311, which will be described next. Proceed to.

The vehicle exterior presence determination unit 9 refers to the vehicle exterior second flag as the second vehicle exterior detection result, determines whether the vehicle exterior second flag is ON (ST311), and determines whether a person outside the vehicle is detected from the captured image. Check whether or not. Then, if the second outside flag is ON (ST311; YES), that is, if it is confirmed that a person outside the vehicle has been detected from the captured image, the outside presence/absence determining unit 9 determines that there is a person outside the vehicle. , and outputs the determination result to the vehicle-side control device 200 (ST310).

On the other hand, if the second outside flag is OFF (ST311; NO), that is, if it is confirmed that a person outside the vehicle is not detected from the captured image, the outside presence/absence determining unit 9 determines that there is no person outside the vehicle. The determination result is output to the vehicle-side control device 200 (ST312).

In this way, when the vehicle exterior detection unit 81 obtains a detection result indicating that a person outside the vehicle was not detected from the sensor information and a detection result indicating that a person outside the vehicle was detected from the captured image, Identify the presence of a person outside the vehicle. As a result, even if the presence of a person is not detected from the sensor information even though there is actually a person outside the vehicle, if the presence of a person is detected from the captured image. In addition, since it is determined that there is a person outside the vehicle using the detection result obtained from the captured image, it is possible to prevent failure to detect a person outside the vehicle and improve the accuracy of detecting a person outside the vehicle, that is, a target person.

Further, in the flowchart of FIG. 11, the vehicle monitoring device 80 performs the processing of ST301 to ST304 and then the processing of ST305 to ST308. However, the flowchart of FIG. 11 is an example, and for example, The processing in ST301 to ST304 and the processing in ST305 to ST308 may be performed in parallel.

Note that in the vehicle monitoring device 80 according to the present embodiment, similarly to the second embodiment, the outside-of-vehicle presence/absence determination section 9 receives a detection result from the second outside-of-vehicle detection section 8 indicating that a person is present outside the vehicle based on sensor information. However, if a detection result indicating that there is no person outside the vehicle is obtained from the captured image, even if the detection result obtained from the captured image is used to determine that there is no person outside the vehicle. good.

That is, for example, even if the vehicle exterior presence/absence determination section 9 obtains a detection result from the first vehicle exterior detection section 7 indicating that a detection target exists outside the vehicle detected from the sensor information, the second vehicle exterior detection section 8 If it is determined from the captured image that the detection target is an object other than a person, it may be determined that there is no person outside the vehicle. In this way, even if the detection result that there is a person outside the vehicle obtained from the sensor information is due to a false detection, if it is clear from the captured image that there is no person outside the vehicle, then there is no person outside the vehicle. Since the system determines that a person does not exist, it is possible to suppress false detections and improve the accuracy of detecting a person outside the vehicle, that is, a target person.

Further, the vehicle monitoring device 80 may perform the first exterior detection process and the second exterior detection process even when the vehicle engine is stopped. In this way, the notification control unit of the vehicle-side control device 200 can notify the driver etc. of the presence of a person outside the vehicle from the notification unit of the vehicle or the terminal owned by the driver, so that the vehicle can be prevented from being stolen, etc. To notify a driver of the presence of a suspicious person near the vehicle before a criminal act is committed against the vehicle.

Next, a hardware configuration that implements the functions of the vehicle monitoring device 80 will be described. FIG. 12 is a diagram showing an example of the hardware configuration of the vehicle monitoring device 80 according to the third embodiment. In the vehicle monitoring device 80, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first outside detection section 7, the second outside detection section 8, the presence/absence determination section 9 outside the vehicle, The functions of the vehicle exterior detection section 81 and the storage section are realized by a processing circuit. That is, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first outside detection section 7, the second outside detection section 8, and the presence/absence determination outside the vehicle of the vehicle monitoring device 80. The unit 9, the vehicle exterior detection unit 81, and the storage unit may be a processing circuit 80a, which is dedicated hardware, as shown in FIG. 12A, or may be a processing circuit 80a, which is dedicated hardware, as shown in FIG. It may be the processor 80b that executes it.

As shown in FIG. 12A, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first outside detection section 7, the second outside detection section 8, and the presence/absence determination section outside the vehicle. 9. When the outside detection unit 81 and the storage unit are dedicated hardware, the processing circuit 80a may be, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, or an ASIC (Application Specific Integrated Circuit). ), FPGA (Field-programmable Gate Array), or a combination of these. First information acquisition unit 1, second information acquisition unit 2, vehicle information acquisition unit 3, information acquisition unit 11, first vehicle exterior detection unit 7, second vehicle exterior detection unit 8, vehicle exterior presence/absence determination unit 9, vehicle exterior detection unit 81, The functions of each part of the storage section and the storage section may be realized by a processing circuit, or the functions of each part may be realized by a single processing circuit.

As shown in FIG. 12B, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first outside detection section 7, the second outside detection section 8, and the presence/absence determination section outside the vehicle. 9. When the outside detection section 81 and the storage section are the processor 80b, the functions of each section are realized by software, firmware, or a combination of software and firmware. Software or firmware is written as a program and stored in memory 80c. The processor 80b reads and executes the program stored in the memory 80c, thereby controlling the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, and the first outside detection section 7. , a second outside-vehicle detection section 8, an outside-vehicle existence determination section 9, an outside-vehicle detection section 81, and a storage section. That is, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first outside detection section 7, the second outside detection section 8, the presence/absence determination section 9 outside the vehicle, and the outside detection section 81, and a storage unit includes a memory 80c for storing a program that, when executed by the processor 80b, results in each step shown in FIG. 4 being executed. In addition, these programs include a first information acquisition section 1, a second information acquisition section 2, a vehicle information acquisition section 3, an information acquisition section 11, a first vehicle exterior detection section 7, a second vehicle exterior detection section 8, and an exterior presence/absence determination section. 9. It can also be said that the computer is caused to execute the procedures or methods of the vehicle external detection section 81 and the storage section.

Here, the processor 80b is, for example, a CPU (Central Processing Unit), a processing device, an arithmetic device, a processor, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor). The memory 80c may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM). However, it may be a magnetic disk such as a hard disk or a flexible disk, or an optical disk such as a mini disk, a CD (Compact Disc), or a DVD (Digital Versatile Disc).

Note that the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first outside detection section 7, the second outside detection section 8, the presence/absence determination section 9 outside the vehicle, and the outside detection section 81 and each function of the storage unit, some may be realized by dedicated hardware, and some may be realized by software or firmware. In this way, the processing circuit 80a in the vehicle monitoring device 80 can implement the above-mentioned functions using hardware, software, firmware, or a combination thereof. In addition, the first information acquisition section 1, the second information acquisition section 2, the vehicle information acquisition section 3, the information acquisition section 11, the first vehicle exterior detection section 7, the second vehicle exterior detection section 8, the vehicle exterior presence/absence determination section 9, the vehicle exterior detection section 81 and at least some of the functions of the storage unit may be executed by an external server.

In this way, the vehicle monitoring device 80 includes the first information acquisition unit 1 that acquires sensor information from the sensor 20 whose detection range includes the outside of the vehicle, and the first information acquisition unit 1 that acquires captured images from the imaging device 30 whose imaging range includes the outside of the vehicle. 2 information acquisition section 2, and a vehicle exterior detection section 81 that detects a person outside the vehicle from the sensor information and captured images acquired by the first information acquisition section 1 and the second information acquisition section 2, respectively. 81 is a detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image when the detection result regarding the presence or absence of the person outside the vehicle obtained from the sensor information is different from the detection result regarding the presence or absence of the person outside the vehicle obtained from the captured image. By using the detection results to identify the presence or absence of a person outside the vehicle, it is possible to prevent failure to detect a person outside the vehicle and improve the accuracy of detecting a person outside the vehicle, that is, a target person.

Further, each embodiment disclosed in this specification can be freely combined within its scope, and each embodiment can be modified or omitted as appropriate. . For example, the vehicle monitoring device may include both the occupant detection section 12 and the vehicle exterior detection section 81, and may detect both the occupant and the person outside the vehicle.

1 First information acquisition unit, 2 Second information acquisition unit, 3 Vehicle information acquisition unit, 4 First detection unit, 5 Second detection unit, 6 Presence/absence determination unit, 7 First vehicle exterior detection unit, 8 Second vehicle exterior detection unit , 9 Existence determination unit outside the vehicle, 10, 80 Vehicle monitoring device, 10a, 80a Processing circuit, 10b, 80b Processor, 10c, 80c Memory, 11 Information acquisition unit, 12 Occupant detection unit, 20 Sensor, 21 Radio wave sensor, 30 Imaging device , 50 Vehicle, 51 Driver's Seat, 52 Passenger Seat, 53, 54, 55 Seat, 56, 57, 58, 59, 60 Passenger, 61 Child Seat, 70 Person Outside the Vehicle, 81 External Detection Unit, 100, 101 Vehicle Monitoring System.

Claims (22)

a first information acquisition unit that acquires sensor information from a sensor whose detection range includes at least a rear seat of the vehicle;
a second information acquisition unit that acquires a captured image from an imaging device whose imaging range includes the front seat and the rear seat of the vehicle;
an occupant detection unit that detects an occupant in the rear seat from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit;
When the detection result regarding the presence or absence of the occupant in the rear seat obtained from the sensor information is different from the detection result regarding the presence or absence of the occupant in the rear seat obtained from the captured image, the occupant detection unit detects the presence or absence of the occupant in the captured image. A vehicle monitoring device characterized in that the presence or absence of a passenger in the rear seat is specified using a detection result regarding the presence or absence of a passenger in the rear seat obtained from the above. When the occupant detection unit obtains a detection result indicating that an occupant is not present in the rear seat from the sensor information and obtains a detection result indicating that an occupant is present in the rear seat from the captured image, the The vehicle monitoring device according to claim 1, wherein the vehicle monitoring device specifies that a passenger is present in a rear seat. When the occupant detection unit obtains a detection result indicating that an occupant is present in the rear seat from the sensor information and obtains a detection result indicating that an occupant is not present in the rear seat from the captured image, The vehicle monitoring device according to claim 1 or 2, wherein the vehicle monitoring device specifies that no occupant is present in the rear seat. The occupant detection unit obtains a detection result indicating that an occupant is present in the rear seat from the sensor information, and obtains a detection result indicating that an object other than the occupant exists in the rear seat from the captured image. The vehicle monitoring device according to any one of claims 1 to 3, wherein the vehicle monitoring device specifies that there is no occupant in the rear seat if the vehicle monitor detects an occupant in the rear seat. If the presence or absence of the occupant in the rear seat cannot be determined from the captured image, the occupant detection unit uses the sensor information to identify the presence or absence of the occupant in the rear seat.
The vehicle monitoring device according to any one of claims 1 to 4, characterized in that: The occupant detection unit obtains a detection result indicating that an occupant is present in the rear seat from the sensor information, and detects from the captured image that an occupant is present in the rear seat that is not obscured by the occupant in the front seat. The vehicle monitoring device according to any one of claims 1 to 5 , wherein when a detection result indicating that no passenger is present is obtained, it is specified that there is no passenger in the rear seat. a first information acquisition unit that acquires sensor information from a sensor whose detection range includes a front seat of the vehicle;
a second information acquisition unit that acquires a captured image from an imaging device that includes the front seat in its imaging range;
an occupant detection unit that detects an occupant in the front seat from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit;
When the detection result regarding the presence or absence of the occupant in the front seat obtained from the sensor information is different from the detection result regarding the presence or absence of the occupant in the front seat obtained from the captured image, the occupant detection unit detects the presence or absence of the occupant in the captured image. The presence or absence of the occupant in the front seat is identified using the detection result regarding the presence or absence of the occupant in the front seat obtained from
A vehicle monitoring device characterized by: When the occupant detection unit obtains a detection result indicating that an occupant is not present in the front seat from the sensor information and obtains a detection result indicating that an occupant is present in the front seat from the captured image, the The vehicle monitoring device according to claim 7, wherein the vehicle monitoring device specifies that a passenger is present in the front seat . When the occupant detection unit obtains a detection result indicating that an occupant is present in the front seat from the sensor information and obtains a detection result indicating that an occupant is not present in the front seat from the captured image, the The vehicle monitoring device according to claim 7 or 8, wherein the vehicle monitoring device specifies that no occupant is present in the front seat . If the presence or absence of the occupant in the front seat cannot be determined from the captured image, the occupant detection unit uses the sensor information to identify the presence or absence of the occupant in the front seat.
The vehicle monitoring device according to any one of claims 7 to 9. The occupant detection unit estimates the attributes of the occupant and the condition of the occupant, using the sensor information, for an occupant present in a seat for which a detection result indicating the presence of an occupant is obtained from the sensor information. At least one of the above is performed, and the captured image is used to estimate the attributes of the occupant and to determine the condition of the occupant for the occupant present in the seat for which a detection result indicating the presence of the occupant is obtained from the captured image. The vehicle monitoring device according to any one of claims 1 to 10, wherein the vehicle monitoring device performs at least one of estimation. a first information acquisition unit that acquires sensor information from a sensor whose detection range includes the outside of the vehicle;
a second information acquisition unit that acquires a captured image from an imaging device whose imaging range includes the outside of the vehicle;
a vehicle exterior detection unit that detects a person outside the vehicle from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit;
When the detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information is different from the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image, the vehicle exterior detection unit detects the presence or absence of the person outside the vehicle. A vehicle monitoring device characterized in that the presence or absence of a person outside the vehicle is specified using a detection result regarding the presence or absence of a person outside the vehicle obtained from a captured image. When the vehicle exterior detection unit obtains a detection result indicating that a person does not exist outside the vehicle from the sensor information, and obtains a detection result indicating that a person exists outside the vehicle from the captured image; 13. The vehicle monitoring device according to claim 12, wherein the vehicle monitoring device specifies that a person is present outside the vehicle. When the vehicle exterior detection unit obtains a detection result indicating that a person exists outside the vehicle from the sensor information, and obtains a detection result indicating that a person does not exist outside the vehicle from the captured image; 14. The vehicle monitoring device according to claim 12 or 13, wherein the vehicle monitoring device specifies that no person exists outside the vehicle. a sensor mounted on a vehicle whose detection range includes at least the rear seat of the vehicle;
an imaging device that is mounted on the vehicle and includes a front seat and a rear seat of the vehicle in an imaging range;
a first information acquisition unit that acquires sensor information from the sensor;
a second information acquisition unit that acquires a captured image from the imaging device;
an occupant detection unit that detects an occupant in the rear seat from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit;
When the detection result regarding the presence or absence of the occupant in the rear seat obtained from the sensor information is different from the detection result regarding the presence or absence of the occupant in the rear seat obtained from the captured image, the occupant detection unit detects the presence or absence of the occupant in the captured image. A vehicle monitoring system characterized in that the presence or absence of a passenger in the rear seat is specified using a detection result regarding the presence or absence of a passenger in the rear seat obtained from the above. a sensor mounted on a vehicle whose detection range includes a front seat of the vehicle;
an imaging device that is mounted on the vehicle and includes the front seat in its imaging range;
a first information acquisition unit that acquires sensor information from the sensor;
a second information acquisition unit that acquires a captured image from the imaging device;
an occupant detection unit that detects an occupant in the front seat from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit;
When the detection result regarding the presence or absence of the occupant in the front seat obtained from the sensor information is different from the detection result regarding the presence or absence of the occupant in the front seat obtained from the captured image, the occupant detection unit detects the presence or absence of the occupant in the captured image. The presence or absence of the occupant in the front seat is identified using the detection result regarding the presence or absence of the occupant in the front seat obtained from
A vehicle monitoring system featuring: The vehicle further includes a vehicle-side control device that acquires a result of identifying the presence or absence of an occupant by the occupant detection unit,
The vehicle-side control device includes a notification control unit that notifies the passenger using the identification result of the presence or absence of the passenger.
The vehicle monitoring system according to claim 15 or 16, characterized in that: A sensor installed in a vehicle and whose detection range includes the outside of the vehicle;
an imaging device that is mounted on the vehicle and whose imaging range includes the outside of the vehicle;
a first information acquisition unit that acquires sensor information from the sensor;
a second information acquisition unit that acquires a captured image from the imaging device;
a vehicle exterior detection unit that detects a person outside the vehicle from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit;
When the detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information is different from the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image, the vehicle exterior detection unit detects the presence or absence of the person outside the vehicle. A vehicle monitoring system characterized in that the presence or absence of a person outside the vehicle is specified using a detection result regarding the presence or absence of a person outside the vehicle obtained from a captured image. further comprising a vehicle-side control device that acquires a identification result from the vehicle exterior detection unit regarding the presence or absence of a person outside the vehicle;
The vehicle-side control device notifies the presence of a person outside the vehicle via a notification unit or a terminal owned by the driver of the vehicle, using the identification result regarding the presence or absence of a person outside the vehicle. do
The vehicle monitoring system according to claim 18. a step of acquiring sensor information from a sensor whose detection range includes at least the rear seat of the vehicle by the first information acquisition unit;
a second information acquisition unit acquiring a captured image from an imaging device whose imaging range includes the front seat and the rear seat of the vehicle;
an occupant detection unit detecting an occupant in the rear seat from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit;
In the step of detecting the occupant in the rear seat, the occupant detection section detects the presence or absence of the occupant in the rear seat obtained from the sensor information and the presence or absence of the occupant in the rear seat obtained from the captured image. If the detection result regarding the presence or absence of the rear seat occupant is different from the detection result regarding the presence or absence of the rear seat occupant, the presence or absence of the rear seat occupant is determined using the detection result regarding the presence or absence of the rear seat occupant obtained from the captured image.
A vehicle monitoring method characterized by: a step of acquiring sensor information from a sensor whose detection range includes the front seat of the vehicle by the first information acquisition unit;
a second information acquisition unit acquiring a captured image from an imaging device whose imaging range includes the front seat of the vehicle;
detecting an occupant in the front seat by an occupant detection unit from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit;
In the step of detecting the occupant in the front seat, a detection result regarding the presence or absence of the occupant in the front seat obtained from the sensor information is different from a detection result regarding the presence or absence of the occupant in the front seat obtained from the captured image. In this case, the presence or absence of the passenger in the front seat is determined using a detection result regarding the presence or absence of the passenger in the front seat obtained from the captured image.
A vehicle monitoring method characterized by: a step of acquiring sensor information from a sensor whose detection range includes the outside of the vehicle by the first information acquisition unit;
a second information acquisition unit acquiring a captured image from an imaging device whose imaging range includes the outside of the vehicle;
detecting a person outside the vehicle from the sensor information and the captured image respectively acquired by the first information acquisition unit and the second information acquisition unit by a vehicle exterior detection unit;
In the step of detecting a person outside the vehicle, the outside detection unit detects a detection result regarding the presence or absence of a person outside the vehicle obtained from the sensor information and a detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image. If the detection result regarding the presence or absence of a person is different, the presence or absence of a person outside the vehicle is determined using the detection result regarding the presence or absence of a person outside the vehicle obtained from the captured image.
A vehicle monitoring method characterized by:

Images