US20200098128A1 - Indoor monitoring device - Google Patents
Indoor monitoring device Download PDFInfo
- Publication number
- US20200098128A1 US20200098128A1 US16/578,574 US201916578574A US2020098128A1 US 20200098128 A1 US20200098128 A1 US 20200098128A1 US 201916578574 A US201916578574 A US 201916578574A US 2020098128 A1 US2020098128 A1 US 2020098128A1
- Authority
- US
- United States
- Prior art keywords
- occupant
- head
- seating position
- detected
- detection unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V40/00—Recognition of biometric, human-related or animal-related patterns in image or video data
- G06V40/10—Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands
- G06V40/16—Human faces, e.g. facial parts, sketches or expressions
- G06V40/161—Detection; Localisation; Normalisation
- G06V40/164—Detection; Localisation; Normalisation using holistic features
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- G06K9/00838—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
- G06V20/593—Recognising seat occupancy
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30196—Human being; Person
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30268—Vehicle interior
Definitions
- An embodiment of this disclosure relates to an indoor monitoring device.
- the above-described techniques are techniques for obtaining information on an occupant in a cabin by using the single imaging unit having no distance measurement function, it is difficult to detect the occupant's seating position in both front-rear and left-right directions.
- An indoor monitoring device of an embodiment includes, as an example, an image obtainment unit obtaining a captured image obtained by imaging an inside of a cabin of a vehicle by an imaging unit, and a position detection unit determining whether or not an occupant's head is present in the captured image, detecting a size of the head, a head position of the head, and a key point position of a key point other than the head of the occupant in the captured image in a case where the head is present in the captured image, and detecting a seating position of the occupant in the cabin based on the size of the head, the head position, and the key point position. Accordingly, as an example, it is possible to detect the seating position of the occupant in the cabin with high accuracy by using the single imaging unit that has no distance measurement function.
- FIG. 1 is a perspective view illustrating an example of a state where a part of the cabin of a vehicle equipped with an indoor monitoring device according to the present embodiment is seen through;
- FIG. 2 is a plan view of an example of the vehicle according to the present embodiment
- FIG. 3 is a block diagram illustrating an example of the functional configuration of the vehicle according to the present embodiment
- FIG. 4 is a block diagram illustrating an example of the functional configuration of an ECU of the vehicle according to the present embodiment
- FIG. 5 is a diagram illustrating an example of the processing that the vehicle according to the present embodiment executes in order to detect an occupant's seating position
- FIG. 6 is a flowchart illustrating an example of the flow of the processing that the vehicle according to the present embodiment executes in order to detect an occupant's seating position.
- a vehicle equipped with an indoor monitoring device may be an automobile (internal combustion engine automobile) having an internal combustion engine (engine) as a drive source, an automobile (such as an electric vehicle and a fuel cell vehicle) having an electric motor (motor) as a drive source, or an automobile (hybrid car) having both an internal combustion engine and an electric motor as drive sources.
- the vehicle can be equipped with various transmission devices and various devices (such as systems and components) necessary for driving an internal combustion engine or an electric motor.
- a device related to driving a vehicle wheel of the vehicle can be variously set in terms of method, number, layout, and the like.
- FIG. 1 is a perspective view illustrating an example of a state where a part of the cabin of the vehicle equipped with the indoor monitoring device according to the present embodiment is seen through.
- a vehicle 1 is provided with a vehicle body 2 , a steering unit 4 , an acceleration operation unit 5 , a braking operation unit 6 , a shift operation unit 7 , and a monitor device 11 .
- the vehicle body 2 has a cabin 2 a that an occupant enters.
- the seat 2 b, 2 h, 2 i, and 2 j on which an occupant in the cabin 2 a sits are provided in the cabin 2 a.
- the seat 2 b is a seat on which a driver sits as an occupant.
- the seat 2 h is a passenger seat provided next to the seat 2 b, which is a driver's seat.
- the seats 2 i and 2 j are rear seats provided behind the seats 2 b and 2 h.
- the steering unit 4 the acceleration operation unit 5 , the braking operation unit 6 , the shift operation unit 7 , and the like are provided in the cabin 2 a in a state where a driver as an occupant faces the seat 2 b.
- the steering unit 4 is, for example, a steering wheel projecting from a dashboard 24 .
- the acceleration operation unit 5 is, for example, an accelerator pedal positioned under a driver's foot.
- the braking operation unit 6 is, for example, a brake pedal positioned under a driver's foot.
- the shift operation unit 7 is, for example, a shift lever projecting from a center console.
- the monitor device 11 is provided in, for example, the middle portion of the dashboard 24 in the width direction (that is, the left-right direction) of the vehicle.
- the monitor device 11 may have a function of a navigation system, an audio system, or the like.
- the monitor device 11 has a display device 8 , a voice output device 9 , and an operation input unit 10 .
- the monitor device 11 may have various operation input units such as a switch, a dial, a joystick, and a push button.
- a liquid crystal display (LCD), an organic electroluminescent display (OELD), or the like constitutes the display device 8 capable of displaying various images based on image data.
- a speaker or the like constitutes the voice output device 9 outputting various voices based on voice data.
- the voice output device 9 may be provided at a different position other than the monitor device 11 in the cabin 2 a.
- a touch panel or the like constitutes the operation input unit 10 allowing an occupant to input various types of information.
- the operation input unit 10 is provided on the display screen of the display device 8 and is capable of transmitting an image displayed on the display device 8 .
- the operation input unit 10 allows an occupant to visually recognize an image displayed on the display screen of the display device 8 .
- the operation input unit 10 receives the input of various types of information by an occupant by detecting a touch operation that the occupant performs on the display screen of the display device 8 .
- FIG. 2 is a plan view of an example of the vehicle according to the present embodiment.
- the vehicle 1 is a four-wheeled vehicle or the like and has two (left and right) front wheels 3 F and two (left and right) rear wheels 3 R. All or some of the four vehicle wheels 3 are steerable.
- the vehicle 1 is equipped with a plurality of imaging units 15 (in-vehicle cameras).
- the vehicle 1 is equipped with, for example, four imaging units 15 a to 15 d .
- the imaging unit 15 is a digital camera having an imaging element such as a charge coupled device (CCD) and a CMOS image sensor (CIS).
- the imaging unit 15 is capable of imaging the surroundings of the vehicle 1 at a predetermined frame rate.
- the imaging unit 15 outputs a captured image obtained by the surroundings of the vehicle 1 being imaged.
- Each of the imaging units 15 has a wide-angle lens or a fisheye lens and is capable of imaging, for example, a range of 140° to 220° in the horizontal direction.
- the optical axis of the imaging unit 15 may be set obliquely downward.
- the imaging unit 15 a is positioned in, for example, a rear end portion 2 e of the vehicle body 2 and is provided in a wall portion below the rear window of a rear hatch door 2 k .
- the imaging unit 15 a is capable of imaging the region behind the vehicle 1 among the surroundings of the vehicle 1 .
- the imaging unit 15 b is positioned in, for example, a right end portion 2 f of the vehicle body 2 and is provided on a door mirror 2 g on the right side.
- the imaging unit 15 b is capable of imaging the region beside the vehicle among the surroundings of the vehicle 1 .
- the imaging unit 15 c is positioned on the front side of the vehicle body 2 or the like, that is, in a front end portion 2 c in the front-rear direction of the vehicle 1 or the like.
- the imaging unit 15 c is provided on a front bumper, a front grill, or the like.
- the imaging unit 15 c is capable of imaging the region in front of the vehicle 1 among the surroundings of the vehicle 1 .
- the imaging unit 15 d is positioned on the left side of the vehicle body 2 or the like, that is, in a left end portion 2 d in the vehicle width direction or the like.
- the imaging unit 15 d is provided on the door mirror 2 g on the left side.
- the imaging unit 15 d is capable of imaging the region beside the vehicle 1 among the surroundings of the vehicle 1 .
- an in-vehicle camera 16 (imaging unit) is provided in the cabin 2 a.
- the in-vehicle camera 16 is provided so as to be capable of imaging the inside of the cabin 2 a.
- the in-vehicle camera 16 is provided so as to be capable of imaging the entire inside of the cabin 2 a (at least an object on the seats 2 b, 2 h, 2 i , and 2 j ) from a point in front of the seats 2 b, 2 h, 2 i, and 2 j.
- the in-vehicle camera 16 is an infrared camera and is capable of imaging the inside of the cabin 2 a at a predetermined frame rate.
- the in-vehicle camera 16 outputs an IR image (hereinafter, referred to as a captured image) obtained by the inside of the cabin 2 a being imaged.
- an IR camera is used as the in-vehicle camera 16 in the present embodiment, the embodiment disclosed here is not limited thereto insofar as the in-vehicle camera 16 is an imaging unit capable of imaging the entire inside of the cabin 2 a.
- the in-vehicle camera 16 may be a visible light camera.
- FIG. 3 is a block diagram illustrating an example of the functional configuration of the vehicle according to the present embodiment.
- the vehicle 1 is provided with a steering system 13 , a brake system 18 , a steering angle sensor 19 , an accelerator sensor 20 , a shift sensor 21 , a vehicle wheel sensor 22 , an in-vehicle network 23 , and an electronic control unit (ECU) 14 .
- the monitor device 11 , the steering system 13 , the brake system 18 , the steering angle sensor 19 , the accelerator sensor 20 , the shift sensor 21 , the vehicle wheel sensor 22 , and the ECU 14 are electrically connected via the in-vehicle network 23 , which is a telecommunication line.
- a controller area network (CAN) or the like constitutes the in-vehicle network 23 .
- CAN controller area network
- the steering system 13 is an electric power steering system, a steer by wire (SBW) system, or the like.
- the steering system 13 has an actuator 13 a and a torque sensor 13 b.
- the steering system 13 steers the vehicle wheel 3 by being electrically controlled by the ECU 14 or the like, operating the actuator 13 a, and performing steering force compensation by applying torque to the steering unit 4 .
- the torque sensor 13 b detects the torque that a driver gives to the steering unit 4 and transmits the result of the detection to the ECU 14 .
- the brake system 18 includes an anti-lock brake system (ABS) controlling the lock of the brake of the vehicle 1 , an anti-slip device (electronic stability control: ESC) suppressing a side slip of the vehicle 1 during cornering, an electric brake system assisting the brake by brake force enhancement, and a brake by wire (BBW).
- ABS anti-lock brake system
- ESC electronic stability control
- BBW brake by wire
- the brake system 18 has an actuator 18 a and a brake sensor 18 b.
- the brake system 18 is electrically controlled by the ECU 14 or the like and applies a braking force to the vehicle wheel 3 via the actuator 18 a.
- the brake system 18 detects, for example, the lock of the brake, the idle rotation of the vehicle wheel 3 , and the indication of a side slip from the rotational difference between the left and right vehicle wheels 3 or the like and executes control to suppress the lock of the brake, the idle rotation of the vehicle wheel 3 , and a side slip.
- the brake sensor 18 b is a displacement sensor detecting the position of the brake pedal as a movable portion of the braking operation unit 6 and transmits the result of the brake pedal position detection to the ECU 14 .
- the steering angle sensor 19 is a sensor detecting the amount of steering of the steering unit 4 such as the steering wheel.
- a Hall element or the like constitutes the steering angle sensor 19 detecting the rotation angle of the rotating part of the steering unit 4 as the steering amount and transmitting the result of the detection to the ECU 14 .
- the accelerator sensor 20 is a displacement sensor detecting the position of the accelerator pedal as a movable portion of the acceleration operation unit 5 and transmits the result of the detection to the ECU 14 .
- the shift sensor 21 is a sensor detecting the position of a movable portion (such as a bar, an arm, and a button) of the shift operation unit 7 and transmits the result of the detection to the ECU 14 .
- the vehicle wheel sensor 22 which has a Hall element or the like, is a sensor detecting the amount of rotation of the vehicle wheel 3 and the number of rotations of the vehicle wheel 3 per unit time and transmits the result of the detection to the ECU 14 .
- a computer or the like constitutes the ECU 14 responsible for overall control of the vehicle 1 by means of hardware-software cooperation.
- the ECU 14 is provided with a central processing unit (CPU) 14 a, a read only memory (ROM) 14 b, a random access memory (RAM) 14 c, a display control unit 14 d, a voice control unit 14 e, and a solid state drive (SSD) 14 f .
- the CPU 14 a, the ROM 14 b, and the RAM 14 c may be provided in the same circuit board.
- the CPU 14 a reads a program stored in a non-volatile storage device such as the ROM 14 b and executes various types of arithmetic processing in accordance with the program. For example, the CPU 14 a executes image processing with respect to image data to be displayed on the display device 8 , processing for detecting an occupant's seating position based on a captured image obtained by the in-vehicle camera 16 , and the like.
- the ROM 14 b stores, for example, various programs and parameters necessary for executing the programs.
- the RAM 14 c temporarily stores various data used in the operation at the CPU 14 a.
- the display control unit 14 d mainly executes, for example, image processing with respect to image data obtained from the imaging unit 15 and output to the CPU 14 a and conversion of image data obtained from the CPU 14 a into display image data to be displayed on the display device 8 among the arithmetic processing at the ECU 14 .
- the voice control unit 14 e mainly executes processing of a voice obtained from the CPU 14 a and output to the voice output device 9 among the arithmetic processing at the ECU 14 .
- the SSD 14 f which is a rewritable non-volatile storage unit, keeps storing data obtained from the CPU 14 a even in a case where the ECU 14 is turned off.
- FIG. 4 is a block diagram illustrating an example of the functional configuration of the ECU of the vehicle according to the present embodiment.
- the ECU 14 is provided with an image obtainment unit 400 , a position detection unit 401 , a position determination unit 402 , and a control unit 403 .
- the ECU 14 realizes the functions of the image obtainment unit 400 , the position detection unit 401 , the position determination unit 402 , and the control unit 403 by, for example, a processor such as the CPU 14 a mounted on a circuit board executing a periphery monitoring program stored in a storage medium such as the ROM 14 b and the SSD 14 f .
- Hardware such as a circuit may partially or wholly constitute the image obtainment unit 400 , the position detection unit 401 , the position determination unit 402 , and the control unit 403 .
- the ECU 14 functions as an example of the indoor monitoring device.
- the image obtainment unit 400 obtains a captured image obtained by the in-vehicle camera 16 imaging the entire inside of the cabin 2 a.
- the position detection unit 401 determines whether or not an occupant's head is present in the captured image obtained by the image obtainment unit 400 . In a case where it is determined that an occupant's head is present in the captured image, the position detection unit 401 detects the size of the occupant's head, the position of the occupant's head (hereinafter, referred to as a head position), and the position of a key point other than the occupant's head (hereinafter, referred to as a key point position) in the captured image.
- the key point is a preset non-head point in the occupant's body.
- the key points are the occupant's joint position (such as neck, shoulder, elbow, wrist, hip joint, and knee) and the occupant's crown or chest.
- the position detection unit 401 detects the occupant's seating position in the cabin 2 a based on the size of the occupant's head, the occupant's head position, and the key point position.
- the seating position of an occupant in the cabin 2 a can be detected with high accuracy, even in a case where the occupant's seating position is detected by means of a captured image obtained by the inside of the cabin 2 a being imaged by the in-vehicle camera 16 that does not have a distance measurement function for measuring the distance to an object, since the positions of the occupant in the cabin 2 a in both of the front-rear and left-right directions can also be detected.
- a plurality of cameras such as a camera for imaging an occupant seated in the front seat (seats 2 b and 2 h ) and a camera for imaging an occupant seated in the rear seat ( 2 i and 2 j ) and there is no need to provide a camera having a distance measurement function such as a time of flight (TOF) camera in addition to the in-vehicle camera 16 , and thus the system for detecting the seating position of an occupant in the cabin 2 a can be reduced in size and cost.
- TOF time of flight
- the position detection unit 401 detects the seating position of the occupant in the cabin 2 a in the left-right direction based on the head position and the key point position. As a result, the seating position of the occupant in the cabin 2 a in the left-right direction can be detected with high accuracy.
- the position detection unit 401 detects whether the occupant is seated on the left side or the right side as the seating position in the left-right direction.
- the position detection unit 401 detects whether the occupant is seated on the left side, in the middle, or on the right side as the seating position in the left-right direction.
- the position detection unit 401 detects two seating positions (such as the left side and the right side) or three seating positions (such as the left side, the middle, and the right side) as the left-right-direction seating positions of an occupant in the cabin 2 a is described in the present embodiment, the embodiment disclosed here is not limited thereto and four or more seating positions can be detected as the left-right-direction seating positions in the cabin 2 a.
- the position detection unit 401 detects the front-rear-direction seating position of an occupant in the cabin 2 a based on the presence or absence of the key point and the size of his or her head in a captured image. As a result, the front-rear-direction seating position of an occupant in the cabin 2 a can be detected with high accuracy. In the present embodiment, the position detection unit 401 detects whether an occupant is seated in the front seat (seats 2 b and 2 h ) or the rear seat (seats 2 i and 2 j ) in the cabin 2 a as the front-rear-direction seating position.
- the position detection unit 401 detects two seating positions (such as the front seat and the rear seat) as the front-rear-direction seating positions of an occupant in the cabin 2 a is described in the present embodiment, the embodiment disclosed here is not limited thereto and three or more seating positions can be detected as the front-rear-direction seating positions in the cabin 2 a.
- the position detection unit 401 detects the front-rear-direction position of the seat 2 b, 2 h, 2 i, or 2 j in which an occupant is seated (hereinafter, referred to as a seat position) based on the size of the head of the occupant (seating position-detected occupant) in a captured image.
- the position detection unit 401 pre-stores a seat position table in a storage medium such as the RAM 14 c.
- Associated in the seat position table for each of the seats 2 b, 2 h, 2 i, and 2 j are the seat positions to which the seats 2 b, 2 h, 2 i, and 2 j can be moved and the reference sizes of the head at the seat positions (hereinafter, referred to as reference sizes).
- the position detection unit 401 detects, as the seat position of the seat in which an occupant is seated, the seat position stored in association with the reference size corresponding to the occupant's head size among the reference sizes of the seat detected as the occupant's seating position in the seat position table. As a result, a more detailed position of an occupant in the cabin 2 a can be detected, and thus control processing for the vehicle 1 related to the position of the occupant can be executed with high accuracy.
- the position detection unit 401 detects an occupant's posture based on the size of the head of the occupant (seating position-detected occupant) in a captured image. In the present embodiment, the position detection unit 401 detects that an occupant is bent forward in a case where the occupant's head size exceeds the upper limit of the reference size of the seat detected as the occupant's seating position in the seat position table.
- the position detection unit 401 detects that an occupant is in a supine state in a case where the occupant's head size is exceeded by the lower limit of the reference size of the seat detected as the occupant's seating position in the seat position table. As a result, a more detailed position of an occupant in the cabin 2 a can be detected, and thus control processing for the vehicle 1 related to the position of the occupant in the cabin 2 a can be executed with high accuracy.
- the position determination unit 402 determines the detected seating position as the occupant's seating position until the occupant alights. After an occupant sits on a seat and wears a seat belt, the occupant is unlikely to move to another seat and the seating position is unlikely to change. Even if the occupant's seating position detected by means of a captured image temporarily changes due to a change in the posture of the occupant or the like after the seat belt worn by the occupant is detected from the captured image, the possibility of returning to the original seating position is high.
- the position determination unit 402 determines the detected seating position as the occupant's seating position until the occupant alights.
- the position determination unit 402 determines the detected seating position as the occupant's seating position until the occupant alights.
- the control unit 403 executes control processing for the vehicle 1 related to an occupant's position based on the occupant's seating position detected by the position detection unit 401 .
- the control unit 403 executes airbag control processing, seat belt reminder control processing, and the like based on an occupant's seating position detected by the position detection unit 401 .
- FIG. 5 is a diagram illustrating an example of the processing that the vehicle according to the present embodiment executes in order to detect an occupant's seating position.
- the vertical axis is the Y axis and the horizontal axis is the X axis.
- the captured image 500 is obtained by the image obtainment unit 400 and then the position detection unit 401 sets, with respect to the obtained captured image 500 , determination regions 501 to 503 corresponding to seating position candidates in the cabin 2 a as illustrated in FIG. 5 .
- the embodiment disclosed here is not limited thereto and one determination region, two determination regions, or four or more determination regions may be set with respect to the captured image 500 .
- the determination regions 501 to 503 are regions for determining whether or not an occupant's head is present.
- the determination region 501 is a region corresponding to the driver's seat (seat 2 b ) and the rear seat (right side and middle) as seating position candidates.
- the determination region 502 is a region corresponding to the rear seat (right side, middle, and left side) as a seating position candidate.
- the determination region 503 is a region corresponding to the passenger seat (seat 2 h ) and the rear seat (left side and middle) as seating position candidates.
- the position detection unit 401 detects occupant heads H 1 to H 5 from each of the determination regions 501 to 503 by image processing. In other words, the position detection unit 401 determines whether or not occupant heads H 1 to H 5 are present in each of the determination regions 501 to 503 . Then, the position detection unit 401 determines whether or not an occupant has entered the cabin 2 a based on the presence or absence of the heads H 1 to H 5 in each of the determination regions 501 to 503 .
- the position detection unit 401 detects head positions HP 1 to HP 5 and the sizes of the heads H 1 to H 5 and the occupant's key point positions KP 1 to KP 12 in the captured image 500 by image processing.
- the position detection unit 401 detects head coordinates as the head positions HP 1 to HP 5 of the heads H 1 to H 5 .
- the head coordinates are the coordinates of the heads H 1 to H 5 in an XY coordinate system in which the vertical axis of the captured image 500 is the X axis and the horizontal axis of the captured image 500 is the X axis.
- the position detection unit 401 detects key point position coordinates, which are the coordinates of key points in the XY coordinate system, as the key point positions KP 1 to KP 12 .
- the position detection unit 401 selects a candidate for the occupant's seating position for each of the plurality of heads H 1 and H 2 from the seating position candidates corresponding to the determination region 501 (driver's seat, rear seat right side, and rear seat middle) based on the sizes of the heads H 1 and H 2 , the head positions HP 1 and HP 2 , and the key point positions KP 1 to KP 5 and detects the selected seating position candidate as the occupant's seating position.
- the position detection unit 401 detects the driver's seat as the occupant's seating position corresponding to the head H 1 among the driver's seat and the rear seat (right side and middle), which are seating position candidates, since the head H 1 is larger in size than the head H 2 and the key point positions KP 1 to KP 3 of the occupant to which the head H 1 belongs are not hidden by the seat 2 b as illustrated in FIG. 5 .
- the position detection unit 401 determines that the head H 1 is larger in size than the head H 2 in a case where the size of the head H 1 exceeds the size of the head H 2 by a predetermined threshold value or more.
- the predetermined threshold value is a head size difference at which it is determined that front-rear-direction seating positions differ from each other.
- the position detection unit 401 detects the right side of the rear seat as the occupant's seating position corresponding to the head H 2 among the driver's seat and the rear seat (right side and middle), which are seating position candidates, since the head H 1 is larger in size than the head H 2 and only the key point positions KP 4 and KP 5 on the left side of the occupant to which the head H 2 belongs are detected (that is, the key point position on the right side of the occupant to which the head H 2 belongs is hidden by the seat 2 b ) as illustrated in FIG. 5 .
- the position detection unit 401 determines that the head H 2 is smaller in size than the head H 1 in a case where the size of the head H 2 is exceeded by the size of the head H 1 by a predetermined threshold value or more.
- the position detection unit 401 selects the occupant seating position candidates of the respective heads from the driver's seat and the rear seat (middle and right side), which are seating position candidates corresponding to the determination region 501 , based on the respective head positions HP 1 and HP 5 and key point positions KP 1 to KP 3 and KP 9 to KP 11 of the plurality of heads H 1 and H 5 and detects the selected seating position candidates as occupant seating positions.
- the position detection unit 401 detects the left-right-direction seating positions of the occupants having the heads H 1 and H 5 based on the positional relationship between the head position HP 1 and the key point positions KP 1 to KP 3 of the occupant having the head H 1 and the head position HP 5 and the key point positions KP 9 to KP 11 of the occupant having the head H 5 .
- the occupant having the head H 1 is likely to be positioned to the right of the occupant having the head H 5 in a case where the head position HP 1 and the key point positions KP 1 to KP 3 of the occupant having the head H 1 are present to the left of the head position HP 5 and the key point positions KP 9 to KP 11 of the occupant having the head H 5 in the captured image 500 .
- the position detection unit 401 detects the driver's seat as the seating position of the occupant having the head H 1 .
- the passenger seat present to the left of the driver's seat is detected as the seating position.
- the position detection unit 401 selects a candidate for the seating position of the occupant having the head H 3 from the seating position candidates corresponding to the determination region 502 (rear seat right side, middle, and left side) based on the head position HP 3 and the key point positions KP 6 , KP 7 , and KP 12 and detects the selected seating position candidate as an occupant seating position.
- the position detection unit 401 detects the middle of the rear seat as the occupant's seating position corresponding to the head H 3 in the rear seat (right side, middle, and left side), which is a seating position candidate, since the key point positions KP 6 , KP 7 , and KP 12 of the occupant having the head H 3 are hidden neither by the seat 2 b nor by the seat 2 h as illustrated in FIG. 5 .
- the position detection unit 401 selects the occupant seating position candidates of the respective heads from the seating position candidates corresponding to the determination region 502 (rear seat right side, middle, and left side) based on the respective head positions and key point positions of the plurality of heads and detects the selected seating position candidates as occupant seating positions.
- the position detection unit 401 detects the right side of the rear seat as the seating position of the occupant having the one head.
- the position detection unit 401 detects the left side of the rear seat as the seating position of the occupant having the one head.
- the position detection unit 401 selects a candidate for the occupant's seating position for each of the plurality of heads H 4 and H 5 from the seating position candidates corresponding to the determination region 503 (passenger seat, rear seat middle, and rear seat left side) based on the sizes of the heads H 4 and H 5 , the head positions HP 4 and HP 5 , and the key point positions KP 8 to KP 11 and detects the selected seating position candidate as the occupant's seating position.
- the position detection unit 401 detects the passenger seat as the occupant's seating position corresponding to the head H 5 among the passenger seat and the rear seat (middle and left side), which are seating position candidates, since the head H 5 is larger in size than the head H 4 and the key point positions KP 9 to KP 11 of the occupant to which the head H 5 belongs are not hidden by the seat 2 h as illustrated in FIG. 5 .
- the position detection unit 401 determines that the head H 5 is larger in size than the head H 4 in a case where the size of the head H 5 exceeds the size of the head H 4 by a predetermined threshold value or more.
- the position detection unit 401 detects the left side of the rear seat as the occupant's seating position corresponding to the head H 4 among the passenger seat and the rear seat (left side and middle), which are seating position candidates, since the head H 5 is larger in size than the head H 4 and only the key point position KP 8 on the right side of the occupant to which the head H 4 belongs is detected (that is, the key point position on the left side of the occupant to which the head H 4 belongs is hidden by the seat 2 h ) as illustrated in FIG. 5 .
- the position detection unit 401 determines that the head H 2 is smaller in size than the head H 1 in a case where the size of the head H 2 is exceeded by the size of the head H 1 by a predetermined threshold value or more.
- the position detection unit 401 selects the occupant seating position candidates of the respective heads from the passenger seat and the rear seat (middle and left side), which are seating position candidates corresponding to the determination region 503 , based on the respective head positions HP 1 and HP 5 and key point positions KP 1 to KP 3 and KP 9 to KP 11 of the plurality of heads H 1 and H 5 and detects the selected seating position candidates as occupant seating positions.
- the position detection unit 401 detects the left-right-direction seating positions of the occupants having the heads H 1 and H 5 based on the positional relationship between the head position HP 1 and the key point positions KP 1 to KP 3 of the occupant having the head H 1 and the head position HP 5 and the key point positions KP 9 to KP 11 of the occupant having the head H 5 .
- the occupant having the head H 5 is likely to be positioned to the left of the occupant having the head H 1 in a case where the head position HP 5 and the key point positions KP 9 to KP 11 of the occupant having the head H 5 are present to the right of the head position HP 1 and the key point positions KP 1 to KP 3 of the occupant having the head H 1 in the captured image 500 .
- the position detection unit 401 detects the passenger seat as the seating position of the occupant having the head H 5 .
- the passenger seat present to the right of the passenger seat is detected as the seating position as described above.
- FIG. 6 is a flowchart illustrating an example of the flow of the processing that the vehicle according to the present embodiment executes in order to detect an occupant's seating position.
- the position detection unit 401 determines whether or not an occupant's head is present in the obtained captured image (Step S 601 ). In a case where an occupant's head is not present in the obtained captured image (Step S 601 : No), the position detection unit 401 waits until the image obtainment unit 400 obtains a new IR image.
- the position detection unit 401 detects the size and the position of the occupant's head and the key point position in the captured image (Step S 602 ).
- the position detection unit 401 detects the left-right-direction seating position of the occupant in the cabin 2 a based on the head position and the key point position in the captured image (Step S 603 ). Further, the position detection unit 401 detects the front-rear-direction seating position of the occupant in the cabin 2 a based on the size of the head and the presence or absence of the key point in the captured image (Step S 604 ).
- the position determination unit 402 detects a seat belt worn by the seating position-detected occupant from the captured image (Step S 605 ).
- Step S 605 the position determination unit 402 determines that the occupant's seating position is likely to change and returns to Step S 601 without determining the occupant's seating position.
- Step S 606 the position determination unit 402 determines whether or not the occupant has alighted by using the captured image.
- Step S 606 determines the detected occupant seating position as the occupant's seating position (Step S 607 ) and returns to Step S 606 .
- the position determination unit 402 maintains the determination of the occupant's seating position.
- Step S 606 the position determination unit 402 resets the occupant seating position detection result and ends the occupant seating position detection processing.
- the seating position of an occupant in the cabin 2 a can be detected with high accuracy, even in a case where the occupant's seating position is detected by means of a captured image obtained by the inside of the cabin 2 a being imaged by the in-vehicle camera 16 that does not have a distance measurement function for measuring the distance to an object, since the positions of the occupant in the cabin 2 a in both of the front-rear and left-right directions can also be detected.
- the vehicle 1 of the present embodiment there is no need to install in the cabin 2 a a plurality of cameras such as a camera for imaging an occupant seated in the front seat (seats 2 b and 2 h ) and a camera for imaging an occupant seated in the rear seat ( 2 i and 2 j ) and there is no need to provide a camera having a distance measurement function such as a time of flight (TOF) camera in addition to the in-vehicle camera 16 , and thus the system for detecting the seating position of an occupant in the cabin 2 a can be reduced in size and cost.
- TOF time of flight
- the position detection unit may set, with respect to the captured image, a determination region corresponding to candidates for the seating position, determine whether or not the head is present in the determination region, and in a case where the head is present in the determination region, select a candidate for the seating position of the occupant having the head from the candidates for the seating position corresponding to the determination region based on the size of the head, the head position, and the key point position and detect the selected candidate for the seating position as the seating position.
- the position detection unit may detect the seating position of the occupant in the cabin in a left-right direction based on the head position and the key point position. Accordingly, as an example, the left-right-direction seating position of the occupant in the cabin can be detected with high accuracy.
- the position detection unit may detect the seating position of the occupant in the cabin in a front-rear direction based on the size of the head and the presence or absence of the key point. Accordingly, as an example, the front-rear-direction seating position of the occupant in the cabin can be detected with high accuracy.
- the indoor monitoring device of the embodiment may further include a determination unit determining, in a case where a seat belt worn by the occupant whose seating position has been detected is detected from the captured image, the detected seating position as the occupant's seating position until the occupant alights. Accordingly, as an example, it is possible to prevent the occupant's seating position from being erroneously detected due to a temporary change in the occupant's posture until the occupant alights in a case where the seat belt worn by the seating position-detected occupant is detected from the captured image.
- the position detection unit may further detect a front-rear-direction seat position of a seat on which the occupant is seated based on the size of the head of the occupant whose seating position has been detected. Accordingly, as an example, control processing for the vehicle related to the occupant's position can be executed with high accuracy.
- the position detection unit may further detect an occupant's posture based on the size of the head of the occupant whose seating position has been detected. Accordingly, as an example, control processing for the vehicle related to the occupant's position in the cabin can be executed with high accuracy.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Geometry (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Oral & Maxillofacial Surgery (AREA)
- Human Computer Interaction (AREA)
- Fittings On The Vehicle Exterior For Carrying Loads, And Devices For Holding Or Mounting Articles (AREA)
- Image Analysis (AREA)
Abstract
Description
- This application is based on and claims priority under 35 U.S.C. § 119 to Japanese Patent Application 2018-180433, filed on Sep. 26, 2018, the entire contents of which are incorporated herein by reference.
- An embodiment of this disclosure relates to an indoor monitoring device.
- Techniques for detecting the number of occupants who have entered a cabin and the number of occupants going out of the cabin or detecting an occupant in the cabin based on a captured image obtained by an imaging unit imaging the inside of the cabin of a vehicle have been developed.
- However, it is difficult to accurately estimate the position of an object in a three-dimensional space such as the cabin by using the imaging unit that does not have a distance measurement function for measuring the distance to an object. Although the above-described techniques are techniques for obtaining information on an occupant in a cabin by using the single imaging unit having no distance measurement function, it is difficult to detect the occupant's seating position in both front-rear and left-right directions.
- Thus, a need exists for an indoor monitoring device which is not susceptible to the drawback mentioned above.
- An indoor monitoring device of an embodiment includes, as an example, an image obtainment unit obtaining a captured image obtained by imaging an inside of a cabin of a vehicle by an imaging unit, and a position detection unit determining whether or not an occupant's head is present in the captured image, detecting a size of the head, a head position of the head, and a key point position of a key point other than the head of the occupant in the captured image in a case where the head is present in the captured image, and detecting a seating position of the occupant in the cabin based on the size of the head, the head position, and the key point position. Accordingly, as an example, it is possible to detect the seating position of the occupant in the cabin with high accuracy by using the single imaging unit that has no distance measurement function.
- The foregoing and additional features and characteristics of this disclosure will become more apparent from the following detailed description considered with the reference to the accompanying drawings, wherein:
-
FIG. 1 is a perspective view illustrating an example of a state where a part of the cabin of a vehicle equipped with an indoor monitoring device according to the present embodiment is seen through; -
FIG. 2 is a plan view of an example of the vehicle according to the present embodiment; -
FIG. 3 is a block diagram illustrating an example of the functional configuration of the vehicle according to the present embodiment; -
FIG. 4 is a block diagram illustrating an example of the functional configuration of an ECU of the vehicle according to the present embodiment; -
FIG. 5 is a diagram illustrating an example of the processing that the vehicle according to the present embodiment executes in order to detect an occupant's seating position; and -
FIG. 6 is a flowchart illustrating an example of the flow of the processing that the vehicle according to the present embodiment executes in order to detect an occupant's seating position. - An exemplary embodiment disclosed here will be disclosed below. The configuration of the following embodiment and the actions, results, and effects provided by the configuration are an example. The embodiment disclosed here can be realized by a configuration other than the configuration disclosed in the following embodiment, and at least one of various effects based on the basic configuration and derivative effects can be obtained.
- A vehicle equipped with an indoor monitoring device according to the present embodiment may be an automobile (internal combustion engine automobile) having an internal combustion engine (engine) as a drive source, an automobile (such as an electric vehicle and a fuel cell vehicle) having an electric motor (motor) as a drive source, or an automobile (hybrid car) having both an internal combustion engine and an electric motor as drive sources. In addition, the vehicle can be equipped with various transmission devices and various devices (such as systems and components) necessary for driving an internal combustion engine or an electric motor. In addition, a device related to driving a vehicle wheel of the vehicle can be variously set in terms of method, number, layout, and the like.
-
FIG. 1 is a perspective view illustrating an example of a state where a part of the cabin of the vehicle equipped with the indoor monitoring device according to the present embodiment is seen through. As illustrated inFIG. 1 , a vehicle 1 is provided with avehicle body 2, asteering unit 4, anacceleration operation unit 5, abraking operation unit 6, ashift operation unit 7, and amonitor device 11. Thevehicle body 2 has acabin 2 a that an occupant enters. -
Seats cabin 2 a sits are provided in thecabin 2 a. Theseat 2 b is a seat on which a driver sits as an occupant. Theseat 2 h is a passenger seat provided next to theseat 2 b, which is a driver's seat. Theseats 2 i and 2 j are rear seats provided behind theseats - In addition, the
steering unit 4, theacceleration operation unit 5, thebraking operation unit 6, theshift operation unit 7, and the like are provided in thecabin 2 a in a state where a driver as an occupant faces theseat 2 b. Thesteering unit 4 is, for example, a steering wheel projecting from adashboard 24. Theacceleration operation unit 5 is, for example, an accelerator pedal positioned under a driver's foot. Thebraking operation unit 6 is, for example, a brake pedal positioned under a driver's foot. Theshift operation unit 7 is, for example, a shift lever projecting from a center console. - The
monitor device 11 is provided in, for example, the middle portion of thedashboard 24 in the width direction (that is, the left-right direction) of the vehicle. Themonitor device 11 may have a function of a navigation system, an audio system, or the like. Themonitor device 11 has adisplay device 8, avoice output device 9, and anoperation input unit 10. In addition, themonitor device 11 may have various operation input units such as a switch, a dial, a joystick, and a push button. - A liquid crystal display (LCD), an organic electroluminescent display (OELD), or the like constitutes the
display device 8 capable of displaying various images based on image data. A speaker or the like constitutes thevoice output device 9 outputting various voices based on voice data. Thevoice output device 9 may be provided at a different position other than themonitor device 11 in thecabin 2 a. - A touch panel or the like constitutes the
operation input unit 10 allowing an occupant to input various types of information. In addition, theoperation input unit 10 is provided on the display screen of thedisplay device 8 and is capable of transmitting an image displayed on thedisplay device 8. As a result, theoperation input unit 10 allows an occupant to visually recognize an image displayed on the display screen of thedisplay device 8. Theoperation input unit 10 receives the input of various types of information by an occupant by detecting a touch operation that the occupant performs on the display screen of thedisplay device 8. -
FIG. 2 is a plan view of an example of the vehicle according to the present embodiment. As illustrated inFIGS. 1 and 2 , the vehicle 1 is a four-wheeled vehicle or the like and has two (left and right)front wheels 3F and two (left and right)rear wheels 3R. All or some of the fourvehicle wheels 3 are steerable. - The vehicle 1 is equipped with a plurality of imaging units 15 (in-vehicle cameras). In the present embodiment, the vehicle 1 is equipped with, for example, four
imaging units 15 a to 15 d. Theimaging unit 15 is a digital camera having an imaging element such as a charge coupled device (CCD) and a CMOS image sensor (CIS). Theimaging unit 15 is capable of imaging the surroundings of the vehicle 1 at a predetermined frame rate. Theimaging unit 15 outputs a captured image obtained by the surroundings of the vehicle 1 being imaged. Each of theimaging units 15 has a wide-angle lens or a fisheye lens and is capable of imaging, for example, a range of 140° to 220° in the horizontal direction. In addition, the optical axis of theimaging unit 15 may be set obliquely downward. - Specifically, the
imaging unit 15 a is positioned in, for example, arear end portion 2 e of thevehicle body 2 and is provided in a wall portion below the rear window of arear hatch door 2 k. Theimaging unit 15 a is capable of imaging the region behind the vehicle 1 among the surroundings of the vehicle 1. Theimaging unit 15 b is positioned in, for example, aright end portion 2 f of thevehicle body 2 and is provided on adoor mirror 2 g on the right side. Theimaging unit 15 b is capable of imaging the region beside the vehicle among the surroundings of the vehicle 1. Theimaging unit 15 c is positioned on the front side of thevehicle body 2 or the like, that is, in afront end portion 2 c in the front-rear direction of the vehicle 1 or the like. Theimaging unit 15 c is provided on a front bumper, a front grill, or the like. Theimaging unit 15 c is capable of imaging the region in front of the vehicle 1 among the surroundings of the vehicle 1. Theimaging unit 15 d is positioned on the left side of thevehicle body 2 or the like, that is, in aleft end portion 2 d in the vehicle width direction or the like. Theimaging unit 15 d is provided on thedoor mirror 2 g on the left side. Theimaging unit 15 d is capable of imaging the region beside the vehicle 1 among the surroundings of the vehicle 1. - In addition, as illustrated in
FIGS. 1 and 2 , an in-vehicle camera 16 (imaging unit) is provided in thecabin 2 a. The in-vehicle camera 16 is provided so as to be capable of imaging the inside of thecabin 2 a. In the present embodiment, the in-vehicle camera 16 is provided so as to be capable of imaging the entire inside of thecabin 2 a (at least an object on theseats seats vehicle camera 16 is an infrared camera and is capable of imaging the inside of thecabin 2 a at a predetermined frame rate. The in-vehicle camera 16 outputs an IR image (hereinafter, referred to as a captured image) obtained by the inside of thecabin 2 a being imaged. Although an IR camera is used as the in-vehicle camera 16 in the present embodiment, the embodiment disclosed here is not limited thereto insofar as the in-vehicle camera 16 is an imaging unit capable of imaging the entire inside of thecabin 2 a. For example, the in-vehicle camera 16 may be a visible light camera. -
FIG. 3 is a block diagram illustrating an example of the functional configuration of the vehicle according to the present embodiment. As illustrated inFIG. 3 , the vehicle 1 is provided with asteering system 13, abrake system 18, a steering angle sensor 19, anaccelerator sensor 20, ashift sensor 21, avehicle wheel sensor 22, an in-vehicle network 23, and an electronic control unit (ECU) 14. Themonitor device 11, thesteering system 13, thebrake system 18, the steering angle sensor 19, theaccelerator sensor 20, theshift sensor 21, thevehicle wheel sensor 22, and theECU 14 are electrically connected via the in-vehicle network 23, which is a telecommunication line. A controller area network (CAN) or the like constitutes the in-vehicle network 23. - The
steering system 13 is an electric power steering system, a steer by wire (SBW) system, or the like. Thesteering system 13 has an actuator 13 a and atorque sensor 13 b. Thesteering system 13 steers thevehicle wheel 3 by being electrically controlled by theECU 14 or the like, operating the actuator 13 a, and performing steering force compensation by applying torque to thesteering unit 4. Thetorque sensor 13 b detects the torque that a driver gives to thesteering unit 4 and transmits the result of the detection to theECU 14. - The
brake system 18 includes an anti-lock brake system (ABS) controlling the lock of the brake of the vehicle 1, an anti-slip device (electronic stability control: ESC) suppressing a side slip of the vehicle 1 during cornering, an electric brake system assisting the brake by brake force enhancement, and a brake by wire (BBW). Thebrake system 18 has an actuator 18 a and abrake sensor 18 b. Thebrake system 18 is electrically controlled by theECU 14 or the like and applies a braking force to thevehicle wheel 3 via theactuator 18 a. Thebrake system 18 detects, for example, the lock of the brake, the idle rotation of thevehicle wheel 3, and the indication of a side slip from the rotational difference between the left andright vehicle wheels 3 or the like and executes control to suppress the lock of the brake, the idle rotation of thevehicle wheel 3, and a side slip. Thebrake sensor 18 b is a displacement sensor detecting the position of the brake pedal as a movable portion of thebraking operation unit 6 and transmits the result of the brake pedal position detection to theECU 14. - The steering angle sensor 19 is a sensor detecting the amount of steering of the
steering unit 4 such as the steering wheel. In the present embodiment, a Hall element or the like constitutes the steering angle sensor 19 detecting the rotation angle of the rotating part of thesteering unit 4 as the steering amount and transmitting the result of the detection to theECU 14. Theaccelerator sensor 20 is a displacement sensor detecting the position of the accelerator pedal as a movable portion of theacceleration operation unit 5 and transmits the result of the detection to theECU 14. - The
shift sensor 21 is a sensor detecting the position of a movable portion (such as a bar, an arm, and a button) of theshift operation unit 7 and transmits the result of the detection to theECU 14. Thevehicle wheel sensor 22, which has a Hall element or the like, is a sensor detecting the amount of rotation of thevehicle wheel 3 and the number of rotations of thevehicle wheel 3 per unit time and transmits the result of the detection to theECU 14. - A computer or the like constitutes the
ECU 14 responsible for overall control of the vehicle 1 by means of hardware-software cooperation. Specifically, theECU 14 is provided with a central processing unit (CPU) 14 a, a read only memory (ROM) 14 b, a random access memory (RAM) 14 c, adisplay control unit 14 d, avoice control unit 14 e, and a solid state drive (SSD) 14 f. TheCPU 14 a, theROM 14 b, and theRAM 14 c may be provided in the same circuit board. - The
CPU 14 a reads a program stored in a non-volatile storage device such as theROM 14 b and executes various types of arithmetic processing in accordance with the program. For example, theCPU 14 a executes image processing with respect to image data to be displayed on thedisplay device 8, processing for detecting an occupant's seating position based on a captured image obtained by the in-vehicle camera 16, and the like. - The
ROM 14 b stores, for example, various programs and parameters necessary for executing the programs. TheRAM 14 c temporarily stores various data used in the operation at theCPU 14 a. Thedisplay control unit 14 d mainly executes, for example, image processing with respect to image data obtained from theimaging unit 15 and output to theCPU 14 a and conversion of image data obtained from theCPU 14 a into display image data to be displayed on thedisplay device 8 among the arithmetic processing at theECU 14. Thevoice control unit 14 e mainly executes processing of a voice obtained from theCPU 14 a and output to thevoice output device 9 among the arithmetic processing at theECU 14. TheSSD 14 f, which is a rewritable non-volatile storage unit, keeps storing data obtained from theCPU 14 a even in a case where theECU 14 is turned off. -
FIG. 4 is a block diagram illustrating an example of the functional configuration of the ECU of the vehicle according to the present embodiment. As illustrated inFIG. 4 , theECU 14 is provided with animage obtainment unit 400, aposition detection unit 401, a position determination unit 402, and acontrol unit 403. TheECU 14 realizes the functions of theimage obtainment unit 400, theposition detection unit 401, the position determination unit 402, and thecontrol unit 403 by, for example, a processor such as theCPU 14 a mounted on a circuit board executing a periphery monitoring program stored in a storage medium such as theROM 14 b and theSSD 14 f. Hardware such as a circuit may partially or wholly constitute theimage obtainment unit 400, theposition detection unit 401, the position determination unit 402, and thecontrol unit 403. In the present embodiment, theECU 14 functions as an example of the indoor monitoring device. - The
image obtainment unit 400 obtains a captured image obtained by the in-vehicle camera 16 imaging the entire inside of thecabin 2 a. - The
position detection unit 401 determines whether or not an occupant's head is present in the captured image obtained by theimage obtainment unit 400. In a case where it is determined that an occupant's head is present in the captured image, theposition detection unit 401 detects the size of the occupant's head, the position of the occupant's head (hereinafter, referred to as a head position), and the position of a key point other than the occupant's head (hereinafter, referred to as a key point position) in the captured image. Here, the key point is a preset non-head point in the occupant's body. For example, the key points are the occupant's joint position (such as neck, shoulder, elbow, wrist, hip joint, and knee) and the occupant's crown or chest. - Next, the
position detection unit 401 detects the occupant's seating position in thecabin 2 a based on the size of the occupant's head, the occupant's head position, and the key point position. As a result, the seating position of an occupant in thecabin 2 a can be detected with high accuracy, even in a case where the occupant's seating position is detected by means of a captured image obtained by the inside of thecabin 2 a being imaged by the in-vehicle camera 16 that does not have a distance measurement function for measuring the distance to an object, since the positions of the occupant in thecabin 2 a in both of the front-rear and left-right directions can also be detected. - In addition, there is no need to install in the
cabin 2 a a plurality of cameras such as a camera for imaging an occupant seated in the front seat (seats vehicle camera 16, and thus the system for detecting the seating position of an occupant in thecabin 2 a can be reduced in size and cost. - Specifically, the
position detection unit 401 detects the seating position of the occupant in thecabin 2 a in the left-right direction based on the head position and the key point position. As a result, the seating position of the occupant in thecabin 2 a in the left-right direction can be detected with high accuracy. In the present embodiment, in a case where theposition detection unit 401 detects the seating position of the occupant seated in the front seat (seats position detection unit 401 detects whether the occupant is seated on the left side or the right side as the seating position in the left-right direction. In addition, in a case where theposition detection unit 401 detects the seating position of the occupant seated in the rear seat (seats 2 i and 2 j), theposition detection unit 401 detects whether the occupant is seated on the left side, in the middle, or on the right side as the seating position in the left-right direction. - Although an example in which the
position detection unit 401 detects two seating positions (such as the left side and the right side) or three seating positions (such as the left side, the middle, and the right side) as the left-right-direction seating positions of an occupant in thecabin 2 a is described in the present embodiment, the embodiment disclosed here is not limited thereto and four or more seating positions can be detected as the left-right-direction seating positions in thecabin 2 a. - In addition, specifically, the
position detection unit 401 detects the front-rear-direction seating position of an occupant in thecabin 2 a based on the presence or absence of the key point and the size of his or her head in a captured image. As a result, the front-rear-direction seating position of an occupant in thecabin 2 a can be detected with high accuracy. In the present embodiment, theposition detection unit 401 detects whether an occupant is seated in the front seat (seats seats 2 i and 2 j) in thecabin 2 a as the front-rear-direction seating position. - Although an example in which the
position detection unit 401 detects two seating positions (such as the front seat and the rear seat) as the front-rear-direction seating positions of an occupant in thecabin 2 a is described in the present embodiment, the embodiment disclosed here is not limited thereto and three or more seating positions can be detected as the front-rear-direction seating positions in thecabin 2 a. - In addition, the
position detection unit 401 detects the front-rear-direction position of theseat position detection unit 401 pre-stores a seat position table in a storage medium such as theRAM 14 c. Associated in the seat position table for each of theseats seats - The
position detection unit 401 detects, as the seat position of the seat in which an occupant is seated, the seat position stored in association with the reference size corresponding to the occupant's head size among the reference sizes of the seat detected as the occupant's seating position in the seat position table. As a result, a more detailed position of an occupant in thecabin 2 a can be detected, and thus control processing for the vehicle 1 related to the position of the occupant can be executed with high accuracy. - Further, the
position detection unit 401 detects an occupant's posture based on the size of the head of the occupant (seating position-detected occupant) in a captured image. In the present embodiment, theposition detection unit 401 detects that an occupant is bent forward in a case where the occupant's head size exceeds the upper limit of the reference size of the seat detected as the occupant's seating position in the seat position table. - The
position detection unit 401 detects that an occupant is in a supine state in a case where the occupant's head size is exceeded by the lower limit of the reference size of the seat detected as the occupant's seating position in the seat position table. As a result, a more detailed position of an occupant in thecabin 2 a can be detected, and thus control processing for the vehicle 1 related to the position of the occupant in thecabin 2 a can be executed with high accuracy. - In a case where a seat belt worn by a seating position-detected occupant is detected from a captured image, the position determination unit 402 determines the detected seating position as the occupant's seating position until the occupant alights. After an occupant sits on a seat and wears a seat belt, the occupant is unlikely to move to another seat and the seating position is unlikely to change. Even if the occupant's seating position detected by means of a captured image temporarily changes due to a change in the posture of the occupant or the like after the seat belt worn by the occupant is detected from the captured image, the possibility of returning to the original seating position is high.
- Accordingly, in a case where a seat belt worn by a seating position-detected occupant is detected from a captured image, the position determination unit 402 determines the detected seating position as the occupant's seating position until the occupant alights. As a result, in a case where a seat belt worn by a seating position-detected occupant is detected from a captured image, it is possible to prevent the occupant's seating position from being erroneously detected due to a temporary change in the occupant's posture until the occupant alights.
- The
control unit 403 executes control processing for the vehicle 1 related to an occupant's position based on the occupant's seating position detected by theposition detection unit 401. In the present embodiment, thecontrol unit 403 executes airbag control processing, seat belt reminder control processing, and the like based on an occupant's seating position detected by theposition detection unit 401. - Next, an example of the processing that the vehicle 1 according to the present embodiment executes in order to detect an occupant's seating position will be described with reference to
FIG. 5 .FIG. 5 is a diagram illustrating an example of the processing that the vehicle according to the present embodiment executes in order to detect an occupant's seating position. In a captured image 500 illustrated inFIG. 5 , the vertical axis is the Y axis and the horizontal axis is the X axis. - In the present embodiment, the captured image 500 is obtained by the
image obtainment unit 400 and then theposition detection unit 401 sets, with respect to the obtained captured image 500,determination regions 501 to 503 corresponding to seating position candidates in thecabin 2 a as illustrated inFIG. 5 . Although threedetermination regions 501 are set with respect to the captured image 500 in the present embodiment, the embodiment disclosed here is not limited thereto and one determination region, two determination regions, or four or more determination regions may be set with respect to the captured image 500. - Here, the
determination regions 501 to 503 are regions for determining whether or not an occupant's head is present. Thedetermination region 501 is a region corresponding to the driver's seat (seat 2 b) and the rear seat (right side and middle) as seating position candidates. Thedetermination region 502 is a region corresponding to the rear seat (right side, middle, and left side) as a seating position candidate. Thedetermination region 503 is a region corresponding to the passenger seat (seat 2 h) and the rear seat (left side and middle) as seating position candidates. - Next, the
position detection unit 401 detects occupant heads H1 to H5 from each of thedetermination regions 501 to 503 by image processing. In other words, theposition detection unit 401 determines whether or not occupant heads H1 to H5 are present in each of thedetermination regions 501 to 503. Then, theposition detection unit 401 determines whether or not an occupant has entered thecabin 2 a based on the presence or absence of the heads H1 to H5 in each of thedetermination regions 501 to 503. - In a case where the heads H1 to H5 are present in the
determination regions 501 to 503, theposition detection unit 401 detects head positions HP1 to HP5 and the sizes of the heads H1 to H5 and the occupant's key point positions KP1 to KP12 in the captured image 500 by image processing. - In the present embodiment, the
position detection unit 401 detects head coordinates as the head positions HP1 to HP5 of the heads H1 to H5. The head coordinates are the coordinates of the heads H1 to H5 in an XY coordinate system in which the vertical axis of the captured image 500 is the X axis and the horizontal axis of the captured image 500 is the X axis. In addition, in the present embodiment, theposition detection unit 401 detects key point position coordinates, which are the coordinates of key points in the XY coordinate system, as the key point positions KP1 to KP12. - In a case where the plurality of heads H1 and H2 are detected from the
determination region 501 as illustrated inFIG. 5 , theposition detection unit 401 selects a candidate for the occupant's seating position for each of the plurality of heads H1 and H2 from the seating position candidates corresponding to the determination region 501 (driver's seat, rear seat right side, and rear seat middle) based on the sizes of the heads H1 and H2, the head positions HP1 and HP2, and the key point positions KP1 to KP5 and detects the selected seating position candidate as the occupant's seating position. - Specifically, the
position detection unit 401 detects the driver's seat as the occupant's seating position corresponding to the head H1 among the driver's seat and the rear seat (right side and middle), which are seating position candidates, since the head H1 is larger in size than the head H2 and the key point positions KP1 to KP3 of the occupant to which the head H1 belongs are not hidden by theseat 2 b as illustrated inFIG. 5 . In the present embodiment, theposition detection unit 401 determines that the head H1 is larger in size than the head H2 in a case where the size of the head H1 exceeds the size of the head H2 by a predetermined threshold value or more. Here, the predetermined threshold value is a head size difference at which it is determined that front-rear-direction seating positions differ from each other. - The
position detection unit 401 detects the right side of the rear seat as the occupant's seating position corresponding to the head H2 among the driver's seat and the rear seat (right side and middle), which are seating position candidates, since the head H1 is larger in size than the head H2 and only the key point positions KP4 and KP5 on the left side of the occupant to which the head H2 belongs are detected (that is, the key point position on the right side of the occupant to which the head H2 belongs is hidden by theseat 2 b) as illustrated inFIG. 5 . In the present embodiment, theposition detection unit 401 determines that the head H2 is smaller in size than the head H1 in a case where the size of the head H2 is exceeded by the size of the head H1 by a predetermined threshold value or more. - Although not illustrated, even in a case where the plurality of heads H1 and H5 of the same size are detected from the
determination region 501, theposition detection unit 401 selects the occupant seating position candidates of the respective heads from the driver's seat and the rear seat (middle and right side), which are seating position candidates corresponding to thedetermination region 501, based on the respective head positions HP1 and HP5 and key point positions KP1 to KP3 and KP9 to KP11 of the plurality of heads H1 and H5 and detects the selected seating position candidates as occupant seating positions. - In this case, the key point positions KP1 to KP3 and KP9 to KP11 of the occupants respectively having the plurality of heads H1 and H5 are hidden by no seat, and thus it is impossible to detect the left-right-direction seating positions of the occupants having the heads H1 and H5 based on a key point position hidden by the seat. Accordingly, the
position detection unit 401 detects the left-right-direction seating positions of the occupants having the heads H1 and H5 based on the positional relationship between the head position HP1 and the key point positions KP1 to KP3 of the occupant having the head H1 and the head position HP5 and the key point positions KP9 to KP11 of the occupant having the head H5. - Specifically, the occupant having the head H1 is likely to be positioned to the right of the occupant having the head H5 in a case where the head position HP1 and the key point positions KP1 to KP3 of the occupant having the head H1 are present to the left of the head position HP5 and the key point positions KP9 to KP11 of the occupant having the head H5 in the captured image 500. In this case, the
position detection unit 401 detects the driver's seat as the seating position of the occupant having the head H1. As for the seating position of the occupant having the head H5, the passenger seat present to the left of the driver's seat is detected as the seating position. - In addition, in a case where the head H3 is detected from the
determination region 502 as illustrated inFIG. 5 , theposition detection unit 401 selects a candidate for the seating position of the occupant having the head H3 from the seating position candidates corresponding to the determination region 502 (rear seat right side, middle, and left side) based on the head position HP3 and the key point positions KP6, KP7, and KP12 and detects the selected seating position candidate as an occupant seating position. - Specifically, the
position detection unit 401 detects the middle of the rear seat as the occupant's seating position corresponding to the head H3 in the rear seat (right side, middle, and left side), which is a seating position candidate, since the key point positions KP6, KP7, and KP12 of the occupant having the head H3 are hidden neither by theseat 2 b nor by theseat 2 h as illustrated inFIG. 5 . - Although not illustrated, in a case where a plurality of heads of the same size are detected from the
determination region 502, theposition detection unit 401 selects the occupant seating position candidates of the respective heads from the seating position candidates corresponding to the determination region 502 (rear seat right side, middle, and left side) based on the respective head positions and key point positions of the plurality of heads and detects the selected seating position candidates as occupant seating positions. - For example, in a case where the head position and the key point position of one head are present to the left of the head position and the key point position of the other head and the key point position on the right side of the occupant having the one head is hidden by the
seat 2 b in the captured image 500 (that is, in a case where the key point position on the right side of the occupant having the one head is not detected), theposition detection unit 401 detects the right side of the rear seat as the seating position of the occupant having the one head. - In addition, in a case where, for example, the head position and the key point position of one head are present to the right of the head position and the key point position of the other head and the key point position on the left side of the occupant having the one head is hidden by the
seat 2 h in the captured image 500 (that is, in a case where the key point position on the left side of the occupant having the one head is not detected), theposition detection unit 401 detects the left side of the rear seat as the seating position of the occupant having the one head. - Further, in a case where the plurality of heads H4 and H5 are detected from the
determination region 503 as illustrated inFIG. 5 , theposition detection unit 401 selects a candidate for the occupant's seating position for each of the plurality of heads H4 and H5 from the seating position candidates corresponding to the determination region 503 (passenger seat, rear seat middle, and rear seat left side) based on the sizes of the heads H4 and H5, the head positions HP4 and HP5, and the key point positions KP8 to KP11 and detects the selected seating position candidate as the occupant's seating position. - Specifically, the
position detection unit 401 detects the passenger seat as the occupant's seating position corresponding to the head H5 among the passenger seat and the rear seat (middle and left side), which are seating position candidates, since the head H5 is larger in size than the head H4 and the key point positions KP9 to KP11 of the occupant to which the head H5 belongs are not hidden by theseat 2 h as illustrated inFIG. 5 . In the present embodiment, theposition detection unit 401 determines that the head H5 is larger in size than the head H4 in a case where the size of the head H5 exceeds the size of the head H4 by a predetermined threshold value or more. - The
position detection unit 401 detects the left side of the rear seat as the occupant's seating position corresponding to the head H4 among the passenger seat and the rear seat (left side and middle), which are seating position candidates, since the head H5 is larger in size than the head H4 and only the key point position KP8 on the right side of the occupant to which the head H4 belongs is detected (that is, the key point position on the left side of the occupant to which the head H4 belongs is hidden by theseat 2 h) as illustrated inFIG. 5 . In the present embodiment, theposition detection unit 401 determines that the head H2 is smaller in size than the head H1 in a case where the size of the head H2 is exceeded by the size of the head H1 by a predetermined threshold value or more. - Although not illustrated, even in a case where the plurality of heads H1 and H5 of the same size are detected from the
determination region 503, theposition detection unit 401 selects the occupant seating position candidates of the respective heads from the passenger seat and the rear seat (middle and left side), which are seating position candidates corresponding to thedetermination region 503, based on the respective head positions HP1 and HP5 and key point positions KP1 to KP3 and KP9 to KP11 of the plurality of heads H1 and H5 and detects the selected seating position candidates as occupant seating positions. - In this case, the key point positions KP1 to KP3 and KP9 to KP11 of the occupants respectively having the plurality of heads H1 and H5 are hidden by no seat, and thus it is impossible to detect the left-right-direction seating positions of the occupants having the heads H1 and H5 based on a key point position hidden by the seat. Accordingly, the
position detection unit 401 detects the left-right-direction seating positions of the occupants having the heads H1 and H5 based on the positional relationship between the head position HP1 and the key point positions KP1 to KP3 of the occupant having the head H1 and the head position HP5 and the key point positions KP9 to KP11 of the occupant having the head H5. - Specifically, the occupant having the head H5 is likely to be positioned to the left of the occupant having the head H1 in a case where the head position HP5 and the key point positions KP9 to KP11 of the occupant having the head H5 are present to the right of the head position HP1 and the key point positions KP1 to KP3 of the occupant having the head H1 in the captured image 500. In this case, the
position detection unit 401 detects the passenger seat as the seating position of the occupant having the head H5. As for the seating position of the occupant having the head H5, the passenger seat present to the right of the passenger seat is detected as the seating position as described above. - Next, an example of the flow of the processing that the vehicle 1 according to the present embodiment executes in order to detect an occupant's seating position will be described with reference to
FIG. 6 .FIG. 6 is a flowchart illustrating an example of the flow of the processing that the vehicle according to the present embodiment executes in order to detect an occupant's seating position. - When a captured image is obtained by the
image obtainment unit 400, theposition detection unit 401 determines whether or not an occupant's head is present in the obtained captured image (Step S601). In a case where an occupant's head is not present in the obtained captured image (Step S601: No), theposition detection unit 401 waits until theimage obtainment unit 400 obtains a new IR image. - In a case where an occupant's head is present in the captured image (Step S601: Yes), the
position detection unit 401 detects the size and the position of the occupant's head and the key point position in the captured image (Step S602). - Then, the
position detection unit 401 detects the left-right-direction seating position of the occupant in thecabin 2 a based on the head position and the key point position in the captured image (Step S603). Further, theposition detection unit 401 detects the front-rear-direction seating position of the occupant in thecabin 2 a based on the size of the head and the presence or absence of the key point in the captured image (Step S604). - When the occupant's seating position is detected by the
position detection unit 401, the position determination unit 402 detects a seat belt worn by the seating position-detected occupant from the captured image (Step S605). - In a case where the seat belt worn by the seating position-detected occupant is not detected (Step S605: No), the position determination unit 402 determines that the occupant's seating position is likely to change and returns to Step S601 without determining the occupant's seating position.
- In a case where the seat belt worn by the seating position-detected occupant is detected (Step S605: Yes), the position determination unit 402 determines whether or not the occupant has alighted by using the captured image (Step S606).
- In a case where it is determined that the occupant has not alighted (Step S606: No), the position determination unit 402 determines the detected occupant seating position as the occupant's seating position (Step S607) and returns to Step S606. Here, in a case where the occupant's seating position is already determined, the position determination unit 402 maintains the determination of the occupant's seating position.
- In a case where it is determined that the occupant has alighted (Step S606: Yes), the position determination unit 402 resets the occupant seating position detection result and ends the occupant seating position detection processing.
- As described above, according to the vehicle 1 of the present embodiment, the seating position of an occupant in the
cabin 2 a can be detected with high accuracy, even in a case where the occupant's seating position is detected by means of a captured image obtained by the inside of thecabin 2 a being imaged by the in-vehicle camera 16 that does not have a distance measurement function for measuring the distance to an object, since the positions of the occupant in thecabin 2 a in both of the front-rear and left-right directions can also be detected. - In addition, according to the vehicle 1 of the present embodiment, there is no need to install in the
cabin 2 a a plurality of cameras such as a camera for imaging an occupant seated in the front seat (seats vehicle camera 16, and thus the system for detecting the seating position of an occupant in thecabin 2 a can be reduced in size and cost. - In the indoor monitoring device of the embodiment, as an example, the position detection unit may set, with respect to the captured image, a determination region corresponding to candidates for the seating position, determine whether or not the head is present in the determination region, and in a case where the head is present in the determination region, select a candidate for the seating position of the occupant having the head from the candidates for the seating position corresponding to the determination region based on the size of the head, the head position, and the key point position and detect the selected candidate for the seating position as the seating position.
- Accordingly, as an example, it is possible to detect the seating position of the occupant in the cabin with high accuracy by using the single imaging unit that has no distance measurement function.
- In the indoor monitoring device of the embodiment, as an example, the position detection unit may detect the seating position of the occupant in the cabin in a left-right direction based on the head position and the key point position. Accordingly, as an example, the left-right-direction seating position of the occupant in the cabin can be detected with high accuracy.
- In the indoor monitoring device of the embodiment, as an example, the position detection unit may detect the seating position of the occupant in the cabin in a front-rear direction based on the size of the head and the presence or absence of the key point. Accordingly, as an example, the front-rear-direction seating position of the occupant in the cabin can be detected with high accuracy.
- As an example, the indoor monitoring device of the embodiment may further include a determination unit determining, in a case where a seat belt worn by the occupant whose seating position has been detected is detected from the captured image, the detected seating position as the occupant's seating position until the occupant alights. Accordingly, as an example, it is possible to prevent the occupant's seating position from being erroneously detected due to a temporary change in the occupant's posture until the occupant alights in a case where the seat belt worn by the seating position-detected occupant is detected from the captured image.
- In the indoor monitoring device of the embodiment, as an example, the position detection unit may further detect a front-rear-direction seat position of a seat on which the occupant is seated based on the size of the head of the occupant whose seating position has been detected. Accordingly, as an example, control processing for the vehicle related to the occupant's position can be executed with high accuracy.
- In the indoor monitoring device of the embodiment, as an example, the position detection unit may further detect an occupant's posture based on the size of the head of the occupant whose seating position has been detected. Accordingly, as an example, control processing for the vehicle related to the occupant's position in the cabin can be executed with high accuracy.
- The principles, preferred embodiment and mode of operation of the present invention have been described in the foregoing specification. However, the invention which is intended to be protected is not to be construed as limited to the particular embodiments disclosed. Further, the embodiments described herein are to be regarded as illustrative rather than restrictive.
- Variations and changes may be made by others, and equivalents employed, without departing from the spirit of the present invention. Accordingly, it is expressly intended that all such variations, changes and equivalents which fall within the spirit and scope of the present invention as defined in the claims, be embraced thereby.
Claims (16)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JPJP2018-180433 | 2018-09-26 | ||
JP2018180433A JP7210965B2 (en) | 2018-09-26 | 2018-09-26 | indoor monitoring device |
JP2018-180433 | 2018-09-26 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200098128A1 true US20200098128A1 (en) | 2020-03-26 |
US11127152B2 US11127152B2 (en) | 2021-09-21 |
Family
ID=69725332
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/578,574 Active 2039-10-02 US11127152B2 (en) | 2018-09-26 | 2019-09-23 | Indoor monitoring device |
Country Status (3)
Country | Link |
---|---|
US (1) | US11127152B2 (en) |
JP (1) | JP7210965B2 (en) |
DE (1) | DE102019125627B4 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2022042203A1 (en) * | 2020-08-31 | 2022-03-03 | 魔门塔(苏州)科技有限公司 | Human body key point detection method and apparatus |
US11365579B2 (en) * | 2018-11-01 | 2022-06-21 | Mitsui Ktnzoku Act Corporation | Automatic door opening and closing system |
US11485383B2 (en) * | 2019-12-06 | 2022-11-01 | Robert Bosch Gmbh | System and method for detecting and mitigating an unsafe condition in a vehicle |
US20230196798A1 (en) * | 2021-12-21 | 2023-06-22 | Veoneer Us, Inc. | Method and system for seatbelt detection using adaptive histogram normalization |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP7401338B2 (en) * | 2020-02-20 | 2023-12-19 | フォルシアクラリオン・エレクトロニクス株式会社 | Information processing device, program and information processing method |
JP7446492B2 (en) | 2021-02-12 | 2024-03-08 | 三菱電機株式会社 | Vehicle monitoring device, vehicle monitoring system, and vehicle monitoring method |
JP7537366B2 (en) | 2021-05-24 | 2024-08-21 | 株式会社デンソー | Passenger imaging device |
Family Cites Families (16)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2003252127A (en) * | 2001-12-28 | 2003-09-10 | Pioneer Electronic Corp | Drive control device and drive control method |
DE10246055A1 (en) | 2002-10-02 | 2004-04-15 | Robert Bosch Gmbh | Protection device for a vehicle occupant |
DE10308415B3 (en) | 2003-02-27 | 2004-06-03 | Bayerische Motoren Werke Ag | Seat setting control process for vehicles involves filming and storing person's seated position and using control unit to set seat accordingly |
US7372996B2 (en) | 2004-12-27 | 2008-05-13 | Trw Automotive U.S. Llc | Method and apparatus for determining the position of a vehicle seat |
JP2007022401A (en) * | 2005-07-19 | 2007-02-01 | Takata Corp | Occupant information detection system, occupant restraint device and vehicle |
JP2007253821A (en) * | 2006-03-23 | 2007-10-04 | Toyota Motor Corp | Occupant imaging device |
JP2008146449A (en) * | 2006-12-12 | 2008-06-26 | Konica Minolta Holdings Inc | Authentication system, authentication method and program |
JP2010226391A (en) * | 2009-03-23 | 2010-10-07 | Nikon Corp | Image processing unit, program, and method of processing image |
DE102010044449B4 (en) | 2009-12-31 | 2014-05-08 | Volkswagen Ag | Recognizing the degree of driving ability of the driver of a motor vehicle |
JP5453230B2 (en) * | 2010-12-15 | 2014-03-26 | 本田技研工業株式会社 | Occupant detection device |
JP2014092965A (en) | 2012-11-05 | 2014-05-19 | Denso Corp | Occupant monitoring device |
JP6131692B2 (en) | 2013-04-18 | 2017-05-24 | 日産自動車株式会社 | Occupant detection device |
JP6187399B2 (en) | 2014-06-24 | 2017-08-30 | 株式会社デンソー | Vehicle seat belt warning device |
DE102016215766A1 (en) | 2016-08-23 | 2018-03-01 | Robert Bosch Gmbh | Method and device for operating an interior camera |
JP6458792B2 (en) * | 2016-11-04 | 2019-01-30 | 本田技研工業株式会社 | Vehicle control system, vehicle control method, and vehicle control program |
US10220806B2 (en) * | 2017-08-02 | 2019-03-05 | Ford Global Technologies, Llc | Monitoring and alerting vehicle occupants for ignition systems |
-
2018
- 2018-09-26 JP JP2018180433A patent/JP7210965B2/en active Active
-
2019
- 2019-09-23 US US16/578,574 patent/US11127152B2/en active Active
- 2019-09-24 DE DE102019125627.5A patent/DE102019125627B4/en active Active
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11365579B2 (en) * | 2018-11-01 | 2022-06-21 | Mitsui Ktnzoku Act Corporation | Automatic door opening and closing system |
US11485383B2 (en) * | 2019-12-06 | 2022-11-01 | Robert Bosch Gmbh | System and method for detecting and mitigating an unsafe condition in a vehicle |
WO2022042203A1 (en) * | 2020-08-31 | 2022-03-03 | 魔门塔(苏州)科技有限公司 | Human body key point detection method and apparatus |
US20230196798A1 (en) * | 2021-12-21 | 2023-06-22 | Veoneer Us, Inc. | Method and system for seatbelt detection using adaptive histogram normalization |
US11798296B2 (en) * | 2021-12-21 | 2023-10-24 | Veoneer Us, Llc | Method and system for seatbelt detection using adaptive histogram normalization |
Also Published As
Publication number | Publication date |
---|---|
US11127152B2 (en) | 2021-09-21 |
DE102019125627B4 (en) | 2022-12-22 |
JP7210965B2 (en) | 2023-01-24 |
DE102019125627A1 (en) | 2020-03-26 |
JP2020050090A (en) | 2020-04-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11127152B2 (en) | Indoor monitoring device | |
US10131277B2 (en) | Surroundings monitoring apparatus | |
JP7259324B2 (en) | indoor monitoring device | |
US10315569B2 (en) | Surroundings monitoring apparatus and program thereof | |
US20180253106A1 (en) | Periphery monitoring device | |
US20200139888A1 (en) | Vehicle periphery display device | |
WO2020116352A1 (en) | Road surface detection device and road surface detection program | |
US10970812B2 (en) | Image processing device | |
US10540807B2 (en) | Image processing device | |
US11475676B2 (en) | Periphery monitoring device | |
US10977506B2 (en) | Apparatus for determining visual confirmation target | |
US20200098134A1 (en) | Imaging system, imaging device and signal processor | |
US20200010076A1 (en) | Driving Support Device | |
US10846884B2 (en) | Camera calibration device | |
WO2018025441A1 (en) | Periphery monitoring device | |
US11091096B2 (en) | Periphery monitoring device | |
JP6965563B2 (en) | Peripheral monitoring device | |
US20210016711A1 (en) | Vehicle periphery display device | |
JP2018191061A (en) | Periphery monitoring device | |
WO2019035228A1 (en) | Peripheral monitoring device | |
US20210016712A1 (en) | Vehicle periphery display device | |
WO2023188927A1 (en) | Own position error estimation device and own position error estimation method | |
JP2022009331A (en) | Periphery monitoring device | |
JP2017069846A (en) | Display control device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: AISIN SEIKI KABUSHIKI KAISHA, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:FUJII, HIROYUKI;TANAKA, GODAI;TOMATSU, YOSHIAKI;SIGNING DATES FROM 20190826 TO 20190828;REEL/FRAME:050456/0826 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |