JP2019205032A - On-vehicle failure detection device and failure detection method - Google Patents

Abstract

To safely support traveling of a vehicle by captured images of imaging sections.SOLUTION: An on-vehicle failure detection device includes: cameras 50 mounted on a vehicle 10 so as to image an outside of the vehicle; a detection section 81 for detecting a movement direction of a subject in imaging ranges of the cameras 50 on the basis of first captured images captured by the cameras 50 and second captured images captured by the cameras 50 after the first captured images; a traveling direction determination section 82 for determining a traveling direction of the vehicle 10 on the basis of vehicle information acquired from the vehicle 10; and a failure determination section 83 for determining a failure in the cameras 50 by determining whether the movement direction of the subject in the imaging ranges to be imaged by the cameras 50 coincides with a movement direction of imaging ranges detected by the detection section 81 when the vehicle 10 is traveling in a traveling direction determined by the traveling direction determination section 82.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a vehicle-mounted failure detection device and a failure detection method.

  2. Description of the Related Art Conventionally, various techniques for mounting a camera on a vehicle and performing various types of vehicle control based on captured images around the vehicle captured by the camera have been proposed and put into practical use.

  The imaging apparatus disclosed in Patent Document 1 detects pan (left / right) movement and tilt (up / down) movement of a housing that houses an image sensor based on the output of a vibrating gyroscope, and moves the subject image in the moving direction based on the output of the image sensor. A certain image moving direction is detected. Then, when the pan movement direction of the housing matches the image movement direction, the imaging device reverses the image signal obtained from the output of the imaging element from side to side. In addition, the imaging device flips the image signal upside down when the tilt movement direction of the housing matches the image movement direction.

JP 2011-130014 A

  An imaging unit mounted on a vehicle needs to take safety measures. The imaging device of Patent Literature 1 detects pan movement and tilt movement of a housing that houses an imaging element based on the output of a vibrating gyroscope. For this reason, the detection accuracy of the moving direction of the housing depends on the accuracy of the output of the vibration gyro, and there is a problem that the detection accuracy is not stable.

  The present invention has been made in view of the above circumstances, and an object thereof is to safely support traveling of a vehicle by a captured image of an imaging unit.

  In order to solve the above-described problems, the present invention provides an imaging unit that is mounted on a vehicle and captures a predetermined imaging range including the outside of the vehicle, and an acquisition unit that acquires vehicle information indicating a running state of the vehicle from the vehicle. And the moving direction of the subject in the imaging range of the imaging unit based on the first captured image captured by the imaging unit and the second captured image captured by the imaging unit after the first captured image. When the vehicle travels in the traveling direction determined by the traveling direction determining unit, the traveling direction determining unit that determines the traveling direction of the vehicle based on the vehicle information acquired by the acquiring unit, and the traveling direction determining unit A failure determination unit that determines whether or not the moving direction of the subject in the imaging range of the imaging unit matches the moving direction of the subject detected by the detection unit, and determines a failure of the imaging unit; With features That.

  Further, according to the present invention, the detection unit changes the imaging position in the first captured image and the second captured image of a stationary object captured in the first captured image and the second captured image. The moving direction of the subject in the imaging range of the imaging unit is detected based on the above.

  In addition, the present invention is characterized by including a processing unit that inverts the left and right of the captured image data output from the imaging unit when the failure determination unit determines a failure of the imaging unit.

  In addition, the present invention is characterized by including a notifying unit for notifying the failure of the imaging unit when the failure determining unit determines a failure of the imaging unit.

  In the present invention, it is preferable that the traveling direction determination unit is based on at least one of the vehicle speed, the steering angle of the steering wheel, and the shift position of the shift lever acquired by the acquisition unit as the vehicle information. It is characterized by determining.

  In order to solve the above problems, the present invention includes an acquisition step of acquiring vehicle information indicating a running state of the vehicle from a vehicle, and an imaging unit that is mounted on the vehicle and captures a predetermined imaging range including the outside of the vehicle. A detection step of detecting a moving direction of a subject in an imaging range of the imaging unit based on a first captured image that has been captured and a second captured image captured by the imaging unit after the first captured image; A traveling direction determining step for determining a traveling direction of the vehicle based on the vehicle information acquired in the acquiring step; and when the vehicle travels in the traveling direction determined in the traveling direction determining step, A failure that determines whether or not the moving direction of the subject in the imaging range matches the moving direction of the subject detected in the detection step, and determines a failure of the imaging unit And having a constant step.

  According to the present invention, in the detection step, a change in imaging position in the first captured image and the second captured image of a stationary object captured in the first captured image and the second captured image. The moving direction of the subject in the imaging range of the imaging unit is detected based on the above.

  In addition, the present invention is characterized by having an inversion step of inverting the left and right of the captured image data output by the imaging unit when the failure of the imaging unit is determined by the failure determination step.

  In addition, the present invention is characterized by comprising a reporting step of notifying the failure of the imaging unit when the failure of the imaging unit is determined in the failure determination step.

  In the present invention, it is preferable that the traveling direction determination step is based on at least one of the vehicle speed, the steering angle of the steering, and the shift position of the shift lever acquired by the acquiring step as the vehicle information. It is characterized by determining.

  ADVANTAGE OF THE INVENTION According to this invention, driving | running | working of a vehicle can be safely supported by the captured image of an imaging part.

It is a block diagram which shows schematic structure of a vehicle. It is a figure which shows a moving direction registration table. It is a flowchart which shows operation | movement of display control ECU. It is a figure which shows the captured image data of a rear camera. It is a figure which expands and shows a part of captured image data of a rear camera. It is a figure which expands and shows a part of captured image data of a rear camera. It is a figure which shows the captured image data of the rear camera which reversed right and left.

  Embodiments to which the present invention is applied will be described below with reference to the accompanying drawings.

FIG. 1 is a configuration diagram illustrating a schematic configuration of the vehicle 10.
A vehicle speed sensor 11, an accelerator opening sensor 12, a brake sensor 13, a steering angle sensor 14, a shift position sensor 15, a vehicle control ECU (Electronic Control Unit) 20, and the like are mounted on the vehicle 10.

  The vehicle speed sensor 11 detects the traveling speed of the vehicle 10 (hereinafter referred to as vehicle speed) and outputs a sensor signal indicating the detected vehicle speed to the vehicle control ECU 20. The accelerator opening sensor 12 detects the opening of the accelerator pedal 2 and outputs a sensor signal indicating the detected opening to the vehicle control ECU 20. The brake sensor 13 detects the depression amount of the brake pedal 3 and outputs a sensor signal indicating the detected depression amount to the vehicle control ECU 20. The steering angle sensor 14 detects the steering angle of the steering wheel 4 and outputs a sensor signal indicating the detected steering angle to the vehicle control ECU 20. The shift position sensor 15 detects the shift position of the shift lever 5 and outputs a sensor signal indicating the detected shift position to the vehicle control ECU 20.

  The vehicle control ECU 20 is a computer device that includes a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and the like. The ROM stores various control programs and maps executed by the CPU. The CPU executes various control processes by executing a control program. The RAM temporarily stores data such as CPU calculation results.

  The vehicle control ECU 20 controls the travel of the vehicle 10 based on sensor signals input from the vehicle speed sensor 11, the accelerator opening sensor 12, the brake sensor 13, the steering angle sensor 14, and the shift position sensor 15. Specifically, the vehicle control ECU 20 controls the output of an engine (not shown) based on the opening degree of the accelerator pedal 2 and a sensor signal indicating the vehicle speed. Further, the vehicle control ECU 20 controls an actuator provided in a steering device (not shown) based on a sensor signal indicating the steering angle, and changes the direction of the front wheels that are the steering wheels. Furthermore, vehicle control ECU20 controls the actuator with which a brake device not shown is provided based on the sensor signal which shows the depression amount of the brake pedal 3, and controls the braking force given to a wheel.

The vehicle 10 is equipped with a vehicle-mounted failure detection device 100 having a plurality of cameras 50, a display unit 60, and a display control ECU 70.
The display control ECU 70 processes the captured image data captured by the plurality of cameras 50, for example, captured image data around the vehicle viewed from a virtual viewpoint above the vehicle 10 (hereinafter referred to as vehicle ambient image data). Generate. The display control ECU 70 causes the display unit 60 to display an image based on the generated vehicle surrounding image data. The driver of the vehicle 10 confirms the situation around the vehicle 10 based on the image displayed on the display unit 60, for example, whether there is a following vehicle or an obstacle around the vehicle 10. Further, the display control ECU 70 diagnoses a failure of the plurality of cameras 50 mounted on the vehicle 10.

  In the vehicle 10, a front camera 51, a rear camera 52, a left side camera 53, and a right side camera 54 are mounted as a plurality of cameras 50. Hereinafter, when the front camera 51, the rear camera 52, the left side camera 53, and the right side camera 54 are collectively referred to, they are simply referred to as “camera 50”. The front camera 51, the rear camera 52, the left side camera 53, and the right side camera 54 correspond to the “imaging unit” of the present invention.

  The front camera 51 is installed at the front portion of the vehicle 10 and captures images at predetermined time intervals with a predetermined range outside the vehicle 10 and including the front as an imaging range. The rear camera 52 is installed at the rear portion of the vehicle 10 and takes an image at a predetermined time interval with a predetermined range outside the vehicle 10 including the rear as an imaging range. The left side camera 53 is installed on a left door mirror (not shown) of the vehicle 10 and performs imaging at predetermined time intervals with a predetermined range outside the vehicle 10 including the left side as an imaging range. The right side camera 54 is installed on a right door mirror (not shown) of the vehicle 10 and performs imaging at predetermined time intervals with a predetermined range including the right side outside the vehicle 10 as an imaging range. Here, “left” refers to the left hand side of the occupant who has boarded the vehicle 10 and turned forward (in the direction of the front camera 51), and “right” refers to the right hand side of the occupant.

  The camera 50 is a digital camera using a solid-state image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 50 is equipped with, for example, an ultra-wide angle lens having an angle of view of 180 degrees or more, and can capture a relatively wide range of images. The captured image data generated by the camera 50 is added with identification information for identifying the camera 50 and additional information such as an imaging time, is output to the display control ECU 70, and is temporarily stored in the memory 73.

  The display unit 60 is configured by a display device such as an LCD (Liquid Crystal Display) or an organic EL panel, for example, and displays various data under the control of the display control ECU 70.

  The display control ECU 70 is an electronic circuit unit that includes a CPU 80, a memory 73 such as a ROM and a RAM, various interface circuits, and the like. The CPU 80 executes various control processes by executing a control program stored in the memory 73. Thus, the CPU 80 functions as the detection unit 81, the traveling direction determination unit 82, and the failure determination unit 83. Further, the CPU 80 executes the control program, thereby executing the “acquisition step”, “detection step”, “traveling direction determination step”, and “failure determination step” that constitute the failure detection method of the present invention. Details of operations of the detection unit 81, the traveling direction determination unit 82, and the failure determination unit 83 will be described later.

  The memory 73 is a non-volatile storage device, and stores various control programs executed by the CPU 80 and captured image data of the camera 50. The memory 73 stores a movement direction registration table 65, which will be described later, and a first threshold value and a second threshold value. Details of these will be described later.

  The display control ECU 70 includes a CAN (Controller Area Network) communication unit 71 and an image processing unit 72. The display control ECU 70 is connected to a plurality of ECUs (a plurality of ECUs include the vehicle control ECU 20) that controls each part of the vehicle 10 by a CAN cable. The CAN communication unit 71 performs data communication conforming to the CAN communication standard with other ECUs. For example, the CAN communication unit 71 performs data communication with the vehicle control ECU 20 and acquires information such as a vehicle speed, a steering angle, and a shift position (hereinafter, these information are collectively referred to as vehicle information). The CAN communication unit 71 operates as an “acquisition unit” of the present invention.

  The image processing unit 72 is an arithmetic device for image processing such as a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). The image processing unit 72 reads captured image data from the memory 73 and performs image processing to generate vehicle surrounding image data. In addition, the image processing unit 72 executes processing for detecting a subject shown in the captured image data. The image processing unit 72 detects an object imaged in the captured image data, and specifies the moving direction of the subject in the imaging range of the camera 50. Examples of the object include a person, a vehicle, a tunnel, a bridge, a house, characters indicating traffic information displayed on a road surface, a road traffic sign, and the like.

A process for detecting an object from captured image data will be described by taking as an example a case where captured image data of the rear camera 52 is processed. Note that the image processing unit 72 performs the same processing on the captured image data of the front camera 51, the left side camera 53, and the right side camera 54.
The image processing unit 72 reads the captured image data of the rear camera 52 from the memory 73, performs generally known processing such as feature amount matching on the read captured image data, and converts the captured image data into captured image data. Detect captured objects. Further, when the object cannot be detected from the captured image data, the image processing unit 72 reads the next captured image data from the memory 73 and performs processing. The object detected by the image processing unit 72 includes, for example, a vehicle traveling in the same lane as the vehicle 10, a vehicle traveling in the opposite lane, a traffic sign provided on the side of the road, a traffic light, a house, a roadside tree, and vending Machine. When the image processing unit 72 detects an object from the captured image data, the coordinate information indicating the position of the detected object on the captured image data and the shape, color, size, etc. of the object as information for specifying the detected object are displayed. The CPU 80 is notified of the information shown. When the processing on the captured image data is completed, the image processing unit 72 reads out the next captured image data from the memory 73 and detects an object shown in the captured image data.

The CPU 80 detects a failure of the camera 50. Here, the failure of the camera 50 detected by the CPU 80 will be described.
When the captured image data is generated, the camera 50 generates captured image data of a mirror image obtained by inverting the left and right of the generated captured image data. By using the captured image data as a mirror image, the left-right positional relationship of the vehicle 10 in the real space and the left-right positional relationship of the vehicle 10 in the image displayed on the display unit 60 can be made the same. In addition, by using the captured image data as a mirror image, the user can easily intuitively understand the positional relationship of objects existing around the vehicle 10.

  When a failure occurs in the camera 50, the captured image data may be output as a normal image instead of a mirror image obtained by inverting the left and right sides. The camera 50 includes a memory that temporarily stores captured image data, a computing device such as a DSP (Digital Signal Processor) that reads the captured image data from the memory and generates a mirror image, and the computing device and the DSP communicate with each other. It is provided with a configuration connected via When a failure occurs in the memory or when a communication failure occurs between the memory and the arithmetic device, the captured image data of the mirror image may not be output from the camera 50 in some cases.

  The detection unit 81 moves the imaging range of the front camera 51, the rear camera 52, the left side camera 53, and the right side camera 54 based on the captured image data and the captured image data captured after the captured image data. Is detected. Specifically, the detection unit 81 detects the moving direction of the imaging range of each camera 50 by detecting the moving direction of the object detected by the image processing unit 72.

  The operation of the detection unit 81 will be described using captured image data of the rear camera 52 as an example. The detecting unit 81 obtains the moving direction and moving speed of each object detected by the image processing unit 72. Here, the moving direction and moving speed detected by the detecting unit 81 are the moving direction and moving speed of the object viewed from the rear camera 52, that is, the moving direction and moving speed of the object viewed from the viewpoint of moving together with the vehicle 10. .

  The detection unit 81 obtains the moving direction and moving speed of the object based on the change in the position (coordinates) on the captured image data of the object captured in the captured image data of the rear camera 52 that is continuously captured. At this time, the detection unit 81 determines whether or not the objects shown in the plurality of captured image data are the same object based on information such as the shape, color, and size of the object notified from the image processing unit 72. May be. Also, the detection unit 81 determines whether or not the objects shown in the plurality of captured image data are the same object based on the imaging time of the plurality of captured image data and the position on the captured image data. Good.

  When the detection unit 81 identifies the same object that is captured in the captured image data of the rear camera 52 that is continuously captured, the detection unit 81 obtains the moving direction and the moving speed of the identified object in the imaging range. For example, the detection unit 81 captures captured image data captured at a certain time (referred to as a first captured image) and captured image data captured after the captured image data (referred to as a second captured image). The moving direction and moving speed of the object are obtained based on the change in the position of the object. Specifically, the detection unit 81 obtains the moving speed of the object based on the difference in imaging time between the first captured image and the second captured image and the moving distance of the object obtained from the coordinate information. The detection unit 81 detects the moving direction based on the coordinates on the plurality of captured image data from which the object is detected.

When the detecting unit 81 obtains the moving direction and moving speed of each object detected by the image processing unit 72, the first threshold value and the second threshold value corresponding to the vehicle speed acquired from the vehicle control ECU 20 are obtained from the memory 73. get. The first threshold value is a threshold value for specifying the vehicle 10 traveling in the oncoming lane from the object detected by the image processing unit 72. The second threshold value is a threshold value for identifying the vehicle 10 traveling in the same lane from the object detected by the image processing unit 72. The second threshold value is smaller than the first threshold value.
A plurality of first threshold values and second threshold values are prepared according to the vehicle speed of the vehicle 10 and stored in the memory 73 in advance. For example, assuming that the vehicle speed is V, vehicle speeds at preset intervals such as 0 km / h <V ≦ 10 km / h, 11 km / h ≦ V ≦ 20 km / h, 21 km / h ≦ V ≦ 30 km / h. A first threshold value and a second threshold value having different values are prepared for each.

The detection unit 81 excludes an object estimated as a vehicle from the objects detected by the image processing unit 72. Since the speeds of the following vehicles and the vehicles traveling in the oncoming lane are not known, when the camera 50 is detected to be faulty based on the moving direction or speed of these vehicles, an erroneous fault may be detected. For this reason, the detection unit 81 excludes the moving speed and moving direction of the vehicle from the determination materials for failure detection.
The detection unit 81 determines an object having a moving speed greater than the first threshold value as a vehicle traveling on the oncoming lane, and excludes the moving speed and moving direction of the object from the determination material for failure detection. In addition, the detection unit 81 determines an object whose moving speed is smaller than the second threshold value as a succeeding vehicle traveling in the same lane, and excludes the moving speed and moving direction of the object from determination materials for failure detection.

  Next, the detection unit 81 detects an object that is detected by the image processing unit 72 and is stationary (hereinafter referred to as a stationary object) from among objects that are determined as vehicles. When a stationary object is imaged by the rear camera 52 of the traveling vehicle 10, the captured image data continuously captured appears to move in a certain direction at a certain speed.

The detection unit 81 compares the detected moving direction and moving speed of each object, and determines that at least two or more objects moving at the same moving speed in the same moving direction are stationary objects. Note that there are cases where an error is included in the moving speed and moving direction obtained from the captured image data. For this reason, the moving speed and the moving direction do not have to be completely coincident with each other as long as they are within an error range.
In the present embodiment, two or more objects moving at the same movement speed in the same movement direction are determined as stationary objects. However, the stationary object is based on the vehicle speed input as vehicle information from the vehicle control ECU 20. May be detected. That is, an object whose moving speed obtained from captured image data substantially matches the vehicle speed may be determined as a stationary object.

  When the detection unit 81 identifies two or more stationary objects from the captured image data, the detection unit 81 detects the movement direction of the identified two or more restrained objects as the movement direction of the subject in the imaging range of the rear camera 52. In addition, the detection unit 81 detects the moving speeds of the two or more specified restraining objects as the moving speed of the subject in the imaging range. Here, the imaging range is a range captured by the camera 50 that moves with the traveling vehicle 10. As the vehicle 10 moves, the imaging range of the camera 50 also moves. The detection unit 81 detects the moving direction of the subject based on a change in the imaging position in each captured image of the subject captured in a plurality of captured images.

The traveling direction determination unit 82 determines the traveling direction of the vehicle 10 based on vehicle information such as a vehicle speed, a steering angle, and a shift position input from the vehicle control ECU 20.
The traveling direction determination unit 82 determines whether the vehicle 10 is in a traveling state or a stopped state based on the vehicle information. The traveling state refers to a state where the vehicle speed of the vehicle 10 is not 0 km / h, and the stopped state refers to a state where the vehicle speed is 0 km / h. The traveling direction determination unit 82 determines whether the vehicle 10 is in a traveling state or a stopped state based on the vehicle speed acquired from the vehicle control ECU 20.

  The traveling direction determination unit 82 determines whether the vehicle 10 is moving forward or backward when the vehicle 10 is in a traveling state. The CPU 80 determines whether the vehicle 10 is moving forward or backward based on the shift position acquired from the vehicle control ECU 20.

  Furthermore, when the vehicle 10 is in a traveling state, the traveling direction determination unit 82 determines whether the vehicle 10 is in a straight traveling state or a state of turning rightward or leftward. The traveling direction determination unit 82 determines whether the vehicle 10 is in a straight traveling state or a turning traveling state based on the steering angle acquired from the vehicle control ECU 20. For example, the traveling direction determination unit 82 determines that the vehicle 10 is in a straight traveling state when the steering angle acquired from the vehicle control ECU 20 is within a preset reference range.

  When the steering angle acquired from the vehicle control ECU 20 is outside the preset reference range and the steering wheel 4 is determined to be rotating in the right direction by the steering angle, the traveling direction determination unit 82 It is determined that a turn in the direction (hereinafter referred to as a right turn) is being performed. Further, when the steering angle acquired from the vehicle control ECU 20 is outside the reference range and it is determined that the steering wheel 4 is rotating leftward by the steering angle, the traveling direction determination unit 82 moves leftward. It is determined that a turn (hereinafter referred to as a left turn) is being performed. The forward, backward, right turn, and left turn determined by the traveling direction determination unit 82 are collectively referred to as a vehicle running state hereinafter.

The failure determination unit 83 determines whether the moving direction of the stationary object detected by the detection unit 81 matches the moving direction of the stationary object imaged by the camera 50 of the vehicle 10 traveling in the traveling direction determined by the traveling direction determination unit 82. Determine whether or not.
First, the failure determination unit 83 refers to the movement direction registration table 65 based on the traveling direction of the vehicle 10 determined by the traveling direction determination unit 82. When the vehicle 10 travels in the traveling direction determined by the traveling direction determination unit 82, the failure determination unit 83 stores information on the moving direction of the stationary object captured by each camera 50 mounted on the vehicle 10 as a movement direction registration table. Get from 65. Information on the movement direction acquired from the movement direction registration table 65 is referred to as a reference movement direction.

  FIG. 2 shows an example of the movement direction registration table 65. In the movement direction registration table 65, information on the movement direction of the stationary object in the captured image captured by each camera 50 when the camera 50 is operating normally is registered for each vehicle traveling state.

Of the registration information in the movement direction registration table 65, information on the reference movement direction of the stationary object in the captured image captured by the rear camera 52 will be described first.
For example, when the vehicle running state is forward and right turn, the movement direction registration table 65 includes information on the moving direction of the stationary object from left to right and gradually moving away from the vehicle 10. Is registered.
Further, when the vehicle traveling state is forward and left turn, the movement direction registration table 65 has information on the moving direction of the stationary object from the right direction to the left direction and gradually moving away from the vehicle 10. It is registered.
Further, when the vehicle traveling state is reverse and right turn, the movement direction registration table 65 includes information on the moving direction of the stationary object from the right direction to the left direction and gradually approaching the vehicle 10. Is registered.
Further, when the vehicle traveling state is backward and left turn, the movement direction registration table 65 includes information on the moving direction of the stationary object from the left to the right and gradually approaching the vehicle 10. It is registered.

Next, information on the reference moving direction of the stationary object in the captured image captured by the front camera 51 will be described.
When the vehicle traveling state is forward and right turn, the movement direction registration table 65 registers information on the moving direction of the stationary object from the left direction to the right direction and gradually approaching the vehicle 10. Has been.
Further, when the vehicle traveling state is forward and left turn, the movement direction registration table 65 includes information on the moving direction of the stationary object from the right direction to the left direction and the direction gradually approaching the vehicle 10. It is registered.
Further, when the vehicle traveling state is reverse and right turn, the movement direction registration table 65 includes information on the moving direction of the stationary object from the right direction to the left direction and gradually moving away from the vehicle 10. Is registered.
Further, when the vehicle traveling state is backward and left turn, the movement direction registration table 65 has information on the moving direction of the stationary object from left to right and gradually moving away from the vehicle 10. It is registered.

Next, information on the reference moving direction of the stationary object in the captured image captured by the left side camera 53 will be described.
When the vehicle traveling state is straight and forward, information on the left direction to the right direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.
Further, when the vehicle traveling state is straight and reverse, information on the right direction to the left direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.
Further, when the vehicle running state is forward and right turn, information from the left direction to the right direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.
Further, when the vehicle traveling state is forward and left turn, information from the left direction to the right direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.
Further, when the vehicle traveling state is reverse and right turn, information from the right direction to the left direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.
Further, when the vehicle traveling state is backward and left turning, information on the right direction to the left direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.

Next, information on the reference moving direction of the stationary object in the captured image captured by the right side camera 54 will be described.
When the vehicle traveling state is straight and forward, information on the right direction to the left direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.
Further, when the vehicle traveling state is straight and reverse, information on the left direction to the right direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.
Further, when the vehicle traveling state is forward and right turn, information on the right direction to the left direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.
Further, when the vehicle traveling state is forward and left turn, information on the right direction to the left direction is registered in the movement direction registration table 65 as the movement direction of the stationary object.
Further, when the vehicle traveling state is reverse and right turn, information from the left direction to the right direction is registered in the movement direction registration table 65 as the stationary object movement direction.
In addition, when the vehicle traveling state is reverse and turning left, the movement direction registration table 65 registers information from the left direction to the right direction as the moving direction of the stationary object.

  When the failure determination unit 83 acquires information on the reference movement direction corresponding to the traveling direction of the vehicle 10 determined by the traveling direction determination unit 82, whether the movement direction of the stationary object detected by the detection unit 81 matches the reference movement direction. Determine whether or not. The failure determination unit 83 determines that the camera 50 has failed when the moving direction of the stationary object detected by the detection unit 81 does not match the reference movement direction. The failure determination unit 83 determines that the camera 50 has not failed when the movement direction of the stationary object detected by the detection unit 81 matches the reference movement direction.

  If the failure determination unit 83 determines that the camera 50 has failed, it performs a notification operation. For example, the failure determination unit 83 causes the display unit 60 to display information indicating the camera 50 in which the failure has occurred and a message indicating that the camera 50 has failed. Further, the failure determination unit 83 may output information of the camera 50 in which a failure has occurred due to a speaker (not shown) mounted on the vehicle 10. The failure determination unit 83 (CPU 80) and the display unit 60 operate as the “notification unit” of the present invention.

  Further, the failure determination unit 83 may cause the image processing unit 72 to generate captured image data of a mirror image when a failure occurs in the camera 50. The failure determination unit 83 outputs the identification information of the camera 50 determined to have a failure and the mirror image generation instruction to the image processing unit 72. When the mirror image generation instruction and the identification information of the camera 50 are input from the failure determination unit 83, the image processing unit 72 has the left and right sides when reading the captured image data of the camera 50 corresponding to the identification information from the memory 73. The captured image data is read from the memory 73 so as to obtain an inverted image. The image processing unit 72 operates as a “processing unit” of the present invention.

FIG. 3 is a flowchart showing the operation of the display control ECU 70.
The operation of the display control ECU 70 will be described with reference to the flowchart shown in FIG.
First, the display control ECU 70 acquires vehicle information from the vehicle control ECU 20 by the CAN communication unit 71 (step S1). This step S1 corresponds to the “acquisition step” of the present invention. When the display control ECU 70 acquires the vehicle information, the display control ECU 70 determines whether or not the shift position included in the acquired vehicle information is a parking range (step S2). When the shift position is in the parking range (step S2 / YES), the display control ECU 70 determines that the vehicle 10 is in a stopped state. In this case, the display control ECU 70 acquires the vehicle information from the vehicle control ECU 20 again, and waits for the start of the process until the shift position included in the acquired vehicle information becomes a range other than the parking range. When the display control ECU 70 determines that the shift position is not in the parking range (step S2 / NO), the display control ECU 70 determines whether the vehicle speed of the vehicle 10 is 0 km / h (step S3).

When the vehicle speed of the vehicle 10 is 0 km / h (step S3 / YES), the display control ECU 70 determines that the vehicle 10 is in a stopped state. In this case, the display control ECU 70 acquires the vehicle information from the vehicle control ECU 20 again, and waits for the start of the process until the vehicle speed included in the acquired vehicle information is not 0 km / h.
Further, when the vehicle speed is not 0 km / h (step S3 / NO), the display control ECU 70 determines the vehicle running state (step S4). The display control ECU 70 determines whether the vehicle 10 is moving forward or backward as the vehicle running state. Further, the display control ECU 70 determines whether the vehicle 10 is moving straight, turning right, or turning left as the vehicle running state.

  When the display control ECU 70 determines the vehicle travel state (step S4), the display control ECU 70 acquires information on the reference travel direction corresponding to the determined vehicle travel state with reference to the travel direction registration table 65 (step S5). Next, the display control ECU 70 determines whether or not an object has been detected by the image processing unit 72 (step S6). When the object is not detected (step S6 / NO), the display control ECU 70 returns to the determination of step S1 and repeats the processes of steps S1 to S6.

  Further, when the image processing unit 72 detects an object (step S6 / YES), the display control ECU 70 obtains the moving direction and moving speed of the detected object based on the captured image data continuously captured. The display control ECU 70 compares the obtained moving speed of the object with the first threshold value, and excludes an object having a moving speed higher than the first threshold value as a vehicle traveling in the oncoming lane (step S7). Further, the display control ECU 70 compares the obtained moving speed of the object with the second threshold value, and excludes an object having a moving speed lower than the second threshold value as a vehicle traveling in the same lane (step S8). ).

  Next, the display control ECU 70 determines whether there are a plurality of objects detected from the captured image data, excluding the objects excluded in steps S7 and S8 (step S9). If there are not a plurality of objects (step S9 / NO), the display control ECU 70 returns to step S6 and performs the process of detecting the object from the captured image data again. Further, when there are a plurality of objects (step S9 / YES), the display control ECU 70 determines whether or not there are two or more objects moving at the same speed in the same direction among the plurality of objects ( Step S10). When there are no two or more objects that move at the same speed in the same direction (step S10 / NO), the display control ECU 70 returns to the process of step S6 and performs a process of detecting the object from the captured image data.

  Further, when there are two or more objects moving at the same speed in the same direction (step S10 / YES), the display control ECU 70 determines these objects as stationary objects, and the determined moving direction of the stationary object is a reference. It is determined whether or not the movement direction matches (step S11). When the moving direction of the stationary object matches the reference moving direction (step S11 / YES), the display control ECU 70 returns to step S1 and repeats the processing from step S1 again. Further, when the moving direction of the stationary object does not coincide with the reference moving direction (step S11 / NO), the display control ECU 70 notifies the camera 50 of failure detection (step S12). Step S12 corresponds to the “notification step” of the present invention. For example, the display control ECU 70 causes the display unit 60 to display the camera 50 that has detected the failure and a message indicating that the failure has occurred. Further, the display control ECU 70 causes the image processing unit 72 to invert the left and right of the captured image data to generate mirror image captured image data (step S13), and causes the display unit 60 to display the generated mirror image captured image data. Step S13 corresponds to an “inversion step” of the present invention.

FIG. 4 is a diagram showing captured image data 200 of the rear camera 52.
The captured image data 200 of the rear camera 52 includes a vehicle 201 traveling in the same lane, a vehicle 202 traveling in the opposite lane, a house 203, a street tree 204, a traffic sign 205, and the like. The image processing unit 72 of the display control ECU 70 detects these objects from the captured image data 200 of the rear camera 52.

  Next, the display control ECU 70 obtains the detected moving direction and moving speed of the object. The display control ECU 70 detects an object from each of the captured image data 200 continuously captured by the rear camera 52, and moves the object based on the position on the captured image data, the imaging time of the captured image data, and the like. And find the moving speed. When the display control ECU 70 obtains the moving direction and moving speed of the object, the display control ECU 70 detects the vehicles 201 and 202 by comparing the obtained moving speed of the object with the first threshold value and the second threshold value. The vehicles 201 and 202 are excluded from the target.

5 and 6 are diagrams showing captured image data 200 of the rear camera 52. FIG. In particular, FIGS. 5 and 6 are enlarged views of the house 203, the roadside tree 204, and the traffic sign 205, which are objects other than the vehicles 201 and 202 among the detected objects.
In the following description, it is assumed that the vehicle 10 is in a forward and right turn vehicle traveling state. The display control ECU 70 acquires the reference movement direction of the rear camera 52 corresponding to the forward and right turn vehicle running states with reference to the movement direction registration table 65. In the movement direction registration table 65, information that “moves away from the left direction to the right direction” is registered as the reference movement direction of the rear camera 52 corresponding to the forward movement and the right turn.

  5 and 6, the positions of the house 203, the street tree 204, and the traffic sign 205 captured in the captured image data 200 (corresponding to the first captured image) captured at a certain time A are indicated by broken lines. Further, the house 203, the street tree 204, and the traffic sign 205 captured at time A are referred to as a house 203A, a street tree 204A, and a traffic sign 205A, respectively. In addition, the positions of the house 203, the street tree 204, and the traffic sign 205 in the captured image data (second captured image) captured at time B after the lapse of a certain time from time A are indicated by solid lines. In addition, the house 203, the street tree 204, and the traffic sign 205 captured at time B are referred to as a house 203B, a road tree 204B, and a traffic sign 205B, respectively.

FIG. 5 shows a case where the moving direction of the stationary object obtained based on the captured image data captured continuously does not match the reference moving direction. FIG. 6 shows a case where the moving direction of the stationary object obtained based on the captured image data continuously captured matches the reference moving direction.
When the vehicle traveling state of the vehicle 10 is forward and right turn, the reference movement direction is information that “the vehicle moves away from the left direction to the right direction”. In the case of the captured image data 200 shown in FIG. 5, the positions of the house 203B, the street tree 204B, and the traffic sign 205B are to the left of the positions of the house 203A, the road tree 204A, and the traffic sign 205A. For this reason, it can be determined that the house 203, the roadside tree 204, and the traffic sign 205 have moved from right to left. Therefore, the display control ECU 70 determines that a failure has occurred in the rear camera 52.
In the case of the captured image data 200 shown in FIG. 6, the positions of the house 203B, the street tree 204B, and the traffic sign 205B are on the right side of the positions of the house 203A, the road tree 204A, and the traffic sign 205A. For this reason, it can be determined that the house 203, the roadside tree 204, and the traffic sign 205 have moved from left to right. Therefore, the display control ECU 70 determines that the rear camera 52 has not failed.

FIG. 7 is a diagram illustrating mirrored captured image data 300 obtained by inverting the left and right of the captured image data 200 of the rear camera 52 illustrated in FIG. 4.
When the display control ECU 70 determines that the rear camera 52 has failed, the display control ECU 70 causes the image processing unit 72 to invert the left and right of the captured image data 200 to generate the captured image data 300 of a mirror image. Thereby, the left-right positional relationship of the vehicle 10 in the real space and the left-right positional relationship of the vehicle 10 in the image displayed on the display unit 60 can be made the same.

As described above, the display control ECU 70 of this embodiment includes the detection unit 81, the traveling direction determination unit 82, and the failure determination unit 83.
The detection unit 81 detects the moving direction of the imaging range of the camera 50 based on the captured image data of the camera 50 and the captured image data captured after the captured image data.
The traveling direction determination unit 82 determines the traveling direction of the vehicle 10 based on the vehicle information acquired from the vehicle 10.
When the vehicle 10 travels in the traveling direction determined by the traveling direction determining unit 82, the failure determining unit 83 matches the moving direction of the imaging range captured by the camera 50 with the moving direction of the imaging range detected by the detecting unit 81. It is determined whether or not to do.
When the moving direction of the imaging range of the camera 50 mounted on the vehicle 10 traveling in the traveling direction matches the moving direction of the imaging range detected by the detecting unit 81, the failure determination unit 83 has not failed. Is determined. Further, the failure determination unit 83 determines that the camera 50 has failed when the movement direction of the imaging range of the camera 50 mounted on the vehicle 10 traveling in the traveling direction does not match the movement direction of the imaging range detected by the detection unit 81. judge.
For this reason, the display control ECU 70 can detect a failure of the camera 50 based on the vehicle information and the captured image data of the camera 50. Therefore, it is possible to detect a malfunction of the camera 50 at an early stage and to safely support the traveling of the vehicle by the captured image data of the camera 50.

  The detection unit 81 detects the moving direction of the imaging range of the camera 50 based on the change in the imaging position of the captured image data of the stationary object captured by the captured image data continuously captured by the camera 50. For this reason, it is possible to detect a failure of the camera 50 with the simplest possible configuration.

  When the failure determination unit 83 determines that the camera 50 has failed, the image processing unit 72 generates captured image data of a mirror image obtained by inverting the left and right of the captured image data output from the camera 50. The display control ECU 70 displays the captured image data of the generated mirror image on the display unit 60. Therefore, even if a failure occurs in the camera 50, the same captured image data as before the failure can be displayed.

  Further, when the failure determination unit 83 determines that the camera 50 has failed, the failure of the camera 50 is displayed on the display unit 60 and notified. Therefore, it is possible to notify the driver of the malfunction of the camera 50 and prevent an accident or the like from occurring.

  The traveling direction determination unit 82 determines the traveling direction of the vehicle based on at least one of the vehicle speed, the steering angle of the steering, and the shift position of the shift lever as vehicle information acquired from the vehicle 10. Therefore, the traveling direction of the vehicle can be specified with high accuracy.

The above-described embodiments are merely illustrative of one aspect of the present invention, and can be arbitrarily modified and applied without departing from the spirit of the present invention.
For example, the functional blocks shown in FIG. 1 are schematic diagrams showing the functions of the display control ECU 70 classified according to the main processing contents. The configuration of the display control ECU 70 can be divided into more blocks according to the processing content. Further, this functional block may be configured to execute more processing by one block shown in FIG. The processing of each block may be executed by one hardware or a plurality of hardware. The processing of each block may be realized by one program or a plurality of programs.

  Further, the processing unit of the flowchart shown in FIG. 3 is divided according to main processing contents in order to make the processing of the display control ECU 70 easy to understand. There is no limit. The processing of the display control ECU 70 can also be divided into more processing units according to the processing content. Moreover, it can also divide | segment so that one process unit may contain many processes. Further, the processing order of the above flowchart is not limited to the illustrated example.

2 Accelerator pedal 3 Brake pedal 4 Steering wheel 5 Shift lever 10 Vehicle 11 Vehicle speed sensor 12 Accelerator opening sensor 13 Brake sensor 14 Steering angle sensor 15 Shift position sensor 50 Camera 51 Front camera 52 Rear camera 53 Left side camera 54 Right side camera 65 Movement direction registration table 71 CAN communication unit 72 Image processing unit 73 Memory 80 CPU
DESCRIPTION OF SYMBOLS 81 Detection part 82 Travel direction determination part 83 Failure determination part 100 Vehicle-mounted fault detection apparatus

Claims (10)

An imaging unit mounted on a vehicle and imaging a predetermined imaging range including the outside of the vehicle;
An acquisition unit for acquiring vehicle information indicating a running state of the vehicle from the vehicle;
Based on the first captured image captured by the image capturing unit and the second captured image captured by the image capturing unit after the first captured image, the moving direction of the subject in the imaging range of the image capturing unit is detected. A detector to perform,
A traveling direction determination unit that determines a traveling direction of the vehicle based on the vehicle information acquired by the acquisition unit;
When the vehicle travels in the traveling direction determined by the traveling direction determination unit, it is determined whether or not the moving direction of the subject in the imaging range of the imaging unit matches the moving direction of the subject detected by the detection unit. A failure determination unit that determines a failure of the imaging unit;
A vehicle-mounted failure detection apparatus comprising:   The detection unit is based on a change in imaging position of the first captured image and the second captured image of a stationary object captured in the first captured image and the second captured image. The in-vehicle failure detection apparatus according to claim 1, wherein the moving direction of the subject in the imaging range is detected.   3. The vehicle-mounted failure according to claim 1, further comprising a processing unit that reverses left and right of the captured image data output by the imaging unit when the failure determination unit determines that the imaging unit has failed. Detection device.   The in-vehicle failure detection according to any one of claims 1 to 3, further comprising: a notification unit that notifies of a failure of the imaging unit when the failure determination unit determines a failure of the imaging unit. apparatus.   The advancing direction determination unit determines the advancing direction of the vehicle based on at least one of a vehicle speed, a steering angle, and a shift position of a shift lever acquired by the acquisition unit as the vehicle information. The vehicle-mounted failure detection device according to any one of claims 1 to 4. An acquisition step of acquiring vehicle information indicating a running state of the vehicle from a vehicle;
A first captured image captured by an imaging unit that is mounted on a vehicle and captures a predetermined imaging range including the outside of the vehicle, and a second captured image captured by the imaging unit after the first captured image Based on the detection step of detecting the moving direction of the subject in the imaging range of the imaging unit;
A traveling direction determination step of determining a traveling direction of the vehicle based on the vehicle information acquired by the acquisition step;
When the vehicle travels in the traveling direction determined in the traveling direction determination step, it is determined whether or not the moving direction of the subject in the imaging range of the imaging unit matches the moving direction of the subject detected in the detection step. A failure determination step of determining a failure of the imaging unit;
A failure detection method comprising:   The detecting step is based on a change in an imaging position of the first captured image and the second captured image of a stationary object captured in the first captured image and the second captured image. The failure detection method according to claim 6, wherein the moving direction of the subject in the imaging range is detected.   8. The failure detection method according to claim 6, further comprising: an inversion step of inverting the left and right of the captured image data output by the imaging unit when the failure of the imaging unit is determined in the failure determination step. .   9. The apparatus according to claim 6, further comprising a notification step of notifying the failure of the imaging unit when the failure of the imaging unit is determined in the failure determination step. The failure detection method according to the item. The advancing direction determination step determines the advancing direction of the vehicle based on at least one of a vehicle speed, a steering angle, and a shift position of a shift lever acquired as the vehicle information in the acquisition step. The failure detection method according to any one of claims 6 to 9.

Images