JP2020060550A - Abnormality detector, method for detecting abnormality, posture estimating device, and mobile control system - Google Patents

Abstract

To provide a technique which can detect abnormalities of a camera appropriately while prioritizing processing according to a purpose of equipping the camera into a mobile body.SOLUTION: The abnormality detector for detecting abnormalities of a camera equipped in a mobile body includes a controller which executes determination processing of determining the presence or absence of an abnormality in the camera on the basis of changes with time in the position of a feature point extracted from an image taken by the camera. The determination processing can select a first processing mode, in which the threshold value for extracting the feature point is a first threshold value, and a second processing mode, in which the threshold value is a second threshold value smaller than the first threshold value, the speed of the mobile body being higher in the first processing mode than in the second processing mode.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an abnormality detection device, an abnormality detection method, a posture estimation device, and a moving body control system.

  Conventionally, a vehicle such as an automobile is equipped with a camera used for parking assistance and the like. For example, an in-vehicle camera mounted on a vehicle is fixed to the vehicle before being shipped from the factory. However, the vehicle-mounted camera may deviate from the mounting state at the time of factory shipment due to, for example, an unexpected contact or aged deterioration. If the mounting position or the angle of the vehicle-mounted camera is deviated, for example, an error occurs in the steering amount of the steering wheel or the like determined by using the camera image. Therefore, it is important to detect the displacement of the vehicle-mounted camera.

  In Patent Document 1, a vehicle-mounted imaging unit that outputs a captured image obtained by capturing an image of the outside of the vehicle, and a predetermined feature point is extracted from the captured image output from the vehicle-mounted image capturing unit, and a change in time of movement of the feature point There is disclosed an imaging device for a vehicle, comprising: a calculating unit that calculates an optical flow from the above, and a correcting unit that corrects an optical axis of an in-vehicle image pickup unit based on the optical flow calculated by the calculating unit.

JP, 2011-55342, A

  By the way, when using a vehicle-mounted camera to perform driving assistance during low-speed traveling such as automatic parking, it is necessary to recognize a moving object or a stationary object existing around the vehicle by the vehicle-mounted camera. For this reason, when the vehicle is traveling at a low speed, it may be possible to prioritize the processing for recognizing the environment surrounding the vehicle over other processing. In this case, it is conceivable that the processing for detecting the camera shift of the vehicle-mounted camera is performed during high-speed traveling, in which it is not necessary to prioritize the processing for recognizing the vehicle surrounding environment. However, when the moving speed of the vehicle becomes high, it becomes difficult to track the feature points, and there is a concern that the detection accuracy of the camera shift is lowered.

  In addition, when the feature points with a high corner degree are used to accurately track the feature points from different viewpoints, for example, when the vehicle is leaving or when the vehicle is traveling on a narrow road, the corner degree is There is a possibility that the feature points cannot be extracted because there is no feature point with high. As a result, there is a concern that the chances of determining whether or not there is a camera shift decrease.

  It is an object of the present invention to provide a technique capable of appropriately detecting an abnormality in a camera while giving priority to processing according to the purpose of mounting the camera on a moving body as much as possible. Another object of the present invention is to provide a technique capable of increasing the chances of determining the presence / absence of a camera abnormality while improving the determination accuracy as much as possible. It is an object of the present invention to provide a technique capable of achieving at least one of the above two objects.

  In order to achieve the above object, an abnormality detecting device of the present invention is an abnormality detecting device for detecting an abnormality of a camera mounted on a moving body, and detects whether or not there is an abnormality in the camera, a captured image taken by the camera. A control unit that executes a determination process based on a temporal change in the position of the feature point extracted from the determination process, wherein the determination process includes a first processing mode in which a threshold value for extracting the feature point is a first threshold value; A second processing mode in which a threshold is a second threshold smaller than the first threshold is provided in a selectable manner. In the first processing mode, the speed of the moving body is higher than that in the second processing mode. The configuration (first configuration) is used in some cases.

  In the anomaly detection device having the first configuration, the first processing mode is a processing mode that is normally used, and the second processing mode is a case where a predetermined processing result is obtained by the first processing mode. It is preferable that the configuration is a processing mode used in (2nd configuration).

  In the abnormality detecting device having the second configuration, it is preferable that the predetermined processing result includes a processing result that the camera has an abnormality (third configuration).

  In the abnormality detecting device having the second or third configuration, the predetermined processing result includes a processing result indicating that the presence / absence of abnormality of the camera cannot be determined for a predetermined period (fourth configuration). It is preferable that it is a structure).

  In the abnormality detection device having any one of the second to fourth configurations, the control unit is provided to be capable of performing a recognition process of recognizing a surrounding environment of the moving body based on a captured image captured by the camera, In a normal state, the control unit prioritizes the recognition process over the determination process when the speed of the moving body is a speed corresponding to the second processing mode, and the control unit sets the second processing mode. It is preferable that when the execution of the processing mode is decided, the determination processing is prioritized over the recognition processing (fifth configuration).

  In the abnormality detection device having any one of the first to fifth configurations, the control unit, when switching from the first processing mode to the second processing mode, displays on the display device mounted on the moving body, The configuration (sixth configuration) may be configured to perform a process of displaying a message that prompts the driver to perform an operation suitable for the second processing mode.

  In the abnormality detection device having any one of the first to sixth configurations, the control unit controls automatic operation of the moving body when switching from the first processing mode to the second processing mode. The device may have a configuration (seventh configuration) that requests automatic operation.

  In the abnormality detection device having any one of the first to seventh configurations, the time of the position of the characteristic point is set in both the determination processing in the first processing mode and the determination processing in the second processing mode. An estimated value of the movement information of the moving body may be obtained from the change, and the presence or absence of abnormality of the camera may be determined based on the estimated value of the movement information (eighth configuration).

  In the abnormality detecting device having the first configuration, in both of the determination processing in the first processing mode and the determination processing in the second processing mode, the posture of the camera is determined from the time change of the position of the characteristic point. May be estimated, and the presence or absence of abnormality of the camera may be determined based on the estimated posture of the camera (ninth configuration).

  In the abnormality detection device having the first configuration, in the determination process in the first processing mode, the posture of the camera is estimated from the temporal change of the position of the feature point, and the posture of the camera is estimated based on the estimated posture of the camera. It is determined whether or not there is an abnormality in the camera, and in the determination processing in the second processing mode, the estimated value of the movement information of the moving body is obtained from the time change of the position of the feature point, and the estimated value of the movement information is obtained. The configuration (tenth configuration) may be configured to determine whether or not there is an abnormality in the camera based on the determination.

  In the abnormality detection device having the first or tenth configuration, the control unit is for a vehicle stop that determines whether there is an abnormality in the camera by a process different from the determination process when the moving body is stopped. It may be a configuration (11th configuration) that executes the determination process.

  In the abnormality detection device having the eleventh configuration, in the vehicle stop determination process, presence / absence of abnormality of the camera is determined based on an edge image of the body of the moving body obtained from a captured image captured by the camera. It is preferable that it has a configuration (twelfth configuration).

  In order to achieve the above object, the posture estimation device of the present invention is an estimation process for estimating the posture of a camera based on the temporal change in the position of a feature point extracted from a captured image captured by a camera mounted on a moving body. And a second processing mode in which the threshold value is a second threshold value smaller than the first threshold value, and the estimation process is a first processing mode in which the threshold value for extracting the feature points is a first threshold value. Are provided so as to be selectable, and the first processing mode has a configuration (thirteenth configuration) used when the speed of the moving body is higher than that of the second processing mode. .

  In order to achieve the above object, a mobile body control system of the present invention comprises an abnormality detection device having any one of the first to seventh configurations, and an automatic operation control device for controlling the automatic operation of the mobile body. At least when the determination process is performed in the second processing mode, the moving body is automatically driven (14th configuration).

  In order to achieve the above object, an abnormality detection method of the present invention is an abnormality detection method for detecting an abnormality of a camera mounted on a moving body by an apparatus, wherein whether or not there is an abnormality in the camera is captured by the camera. A processing step of making a determination based on a temporal change in the position of a feature point extracted from the captured image, the processing step comprising a first processing mode in which a threshold for extracting the feature point is a first threshold, and the threshold A second processing mode in which a second threshold value smaller than the first threshold value is provided, and the first processing mode is faster than the second processing mode in speed of the moving body. It has a configuration (15th configuration) used for.

  According to the present invention, it is possible to appropriately detect an abnormality of a camera while giving priority to processing according to the purpose of mounting the camera on a moving body as much as possible. Further, according to the present invention, it is possible to increase the chances of determining whether or not there is an abnormality in the camera while improving the determination accuracy as much as possible.

Block diagram showing the configuration of the mobile control system according to the first embodiment The flowchart which shows an example of the detection flow of the camera shift in 1st Embodiment. The flowchart which shows an example of the determination process of the presence or absence of a camera shift by the 1st process mode in the abnormality detection apparatus of 1st Embodiment. Diagram for explaining the method of extracting feature points Diagram for explaining the method for obtaining the optical flow Figure for explaining the coordinate conversion process The flowchart which shows an example of the shift | offset determination in 1st Embodiment. The flowchart which shows the detailed example of the process which judges whether the process result by 1st process mode is a predetermined process result. The flowchart which shows an example of the determination process of the presence or absence of a camera shift by the 2nd process mode in the abnormality detection apparatus of 1st Embodiment. The figure which shows an example of a 1st histogram The figure which shows an example of a 2nd histogram Schematic diagram illustrating a captured image captured by a camera with a large camera displacement The figure which illustrates the 1st histogram produced | generated based on the picked-up image image | photographed by the camera which produced the big camera shift. Block diagram showing the configuration of the abnormality detection system according to the second embodiment The flowchart which shows an example of the camera shift detection flow in 2nd Embodiment. Flowchart showing an example of processing mode selection processing The figure which illustrates the 1st feature point and the 2nd feature point extracted by the extraction part. FIG. 3 is a diagram for explaining a method of deriving an optical flow of a first feature point and an optical flow of a second feature point. Diagram showing a quadrangle formed virtually from the optical flow FIG. 6 is a diagram for explaining a method of identifying two sets of planes whose intersecting lines with a predetermined plane are parallel to each other. Diagram for explaining the method of finding the direction of the line of intersection between faces based on one of the identified face pairs Diagram for explaining the method of finding the direction of the line of intersection between the faces based on the other of the identified face pairs Block diagram showing the configuration of the abnormality detection system according to the third embodiment Flowchart showing an example of a camera shift detection flow in the third embodiment. The flowchart which shows an example of the detection flow of the camera shift which used the determination process for vehicle stop.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following description, the case where the moving body to which the present invention is applied is a vehicle will be described as an example, but the moving body to which the present invention is applied is not limited to the vehicle. The present invention may be applied to, for example, ships, aircrafts, robots, and the like. Vehicles include a wide range of vehicles having wheels, such as automobiles, trains, and automated guided vehicles.

  Further, in the following description, the direction in which the vehicle travels straight ahead and the direction from the driver's seat toward the steering wheel is referred to as “forward direction”. Further, the direction in which the vehicle travels straight ahead and the direction from the steering wheel to the driver's seat is referred to as “rearward direction”. In addition, a direction that is a direction perpendicular to the straight traveling direction of the vehicle and the vertical line and that goes from the right side to the left side of the driver facing the front direction is referred to as “left direction”. Further, a direction that is a direction perpendicular to the straight traveling direction of the vehicle and the vertical line and that is from the left side of the driver facing the front direction to the right side is referred to as “right direction”. The front, rear, left, and right directions are merely names used for description, and are not intended to limit the actual positional relationship and direction.

<< A. First Embodiment >>
<1. Mobile control system>
FIG. 1 is a block diagram showing the configuration of a mobile body control system SYS1 according to the first embodiment of the present invention. In this embodiment, the moving body is a vehicle, and the moving body control system SYS1 is a vehicle control system. As shown in FIG. 1, the mobile control system SYS1 includes an abnormality detection system SYS2, an automatic driving control device 5, and a display device 6. As described later, the abnormality detection system SYS2 includes the abnormality detection device 1. In other words, the mobile body control system SYS1 includes the abnormality detection device 1 and the automatic driving control device 5.

  The automatic driving control device 5 controls the automatic driving of the moving body. The automatic driving control device 5 is mounted on each moving body. Specifically, the automatic driving control device 5 is an ECU (Electronic Control Unit) that controls the drive unit, the braking unit, and the steering unit of the vehicle. The drive unit includes, for example, an engine and a motor. The braking unit includes a brake. The steering unit includes a steering wheel (handle). ON / OFF of the control operation by the automatic driving control device 5 is provided so as to be switchable according to an instruction from the abnormality detection device 1. When the control by the automatic driving control device 5 is started, the accelerator, the brake, and the steering wheel are automatically operated without the driver's operation. Note that it is preferable that the driver can switch ON / OFF of the control operation by the automatic driving control device 5.

  The display device 6 is mounted on each moving body. Specifically, the display device 6 is arranged inside the vehicle at a position where the display surface can be seen by the driver. The display device 6 may be, for example, a liquid crystal display device, an organic EL display device, a plasma display device, or the like. The display device 6 may be a device that is fixed to the vehicle, or may be a device that can be taken out of the vehicle.

<2. Anomaly detection system>
As shown in FIG. 1, the abnormality detection system SYS2 includes an abnormality detection device 1, an imaging unit 2, and a sensor unit 3. In the present embodiment, the abnormality means a state in which the mounting of the camera mounted on the vehicle is displaced (hereinafter referred to as “camera displacement”). That is, the abnormality detection system SYS2 is a system that detects a camera shift of a camera mounted on a vehicle. More specifically, the abnormality detection system SYS2 is a system that detects a camera shift from a mounting state that is a reference such as a mounting state of the camera on the vehicle at the time of factory shipment. The camera misalignment includes a wide range of misalignment and misalignment due to rotation around the axis. The misalignment includes misalignment of the mounting position and misalignment of the mounting angle.

  The imaging unit 2 is provided in the vehicle for the purpose of recognizing the situation around the vehicle. The photographing unit 2 includes a camera 21. The camera 21 is a vehicle-mounted camera. The camera 21 is configured using, for example, a fisheye lens. The camera 21 is connected to the abnormality detection device 1 by wire or wirelessly and outputs a captured image to the abnormality detection device 1.

  In addition, in the present embodiment, the imaging unit 2 includes a plurality of cameras 21. The plurality of cameras 21 are four cameras of a front camera, a back camera, a left side camera, and a right side camera. The front camera is a camera that photographs the front of the vehicle. The back camera is a camera that captures the rear of the vehicle. The left side camera is a camera that shoots the left side of the vehicle. The right side camera is a camera that shoots the right side of the vehicle. The four cameras 21 can photograph the entire circumference of the vehicle in the horizontal direction. As a result, it is possible to safely support the parking of the vehicle. However, the number of cameras 21 included in the image capturing unit 2 may be singular or plural other than four.

  The abnormality detection device 1 is a device that detects an abnormality of the camera 21 mounted on the moving body. Specifically, the abnormality detection device 1 detects the camera shift of the camera 21 itself based on the information from the camera 21 mounted on the vehicle. That is, the abnormality detection device 1 is a camera shift detection device. By using the abnormality detection device 1, it is possible to quickly detect the camera shift while the vehicle is traveling. The abnormality detection device 1 can prevent driving assistance such as parking assistance from being performed, for example, in a state where a camera shift has occurred.

  In the present embodiment, the abnormality detection device 1 is mounted on a vehicle equipped with a camera that is a detection target of a camera shift. Hereinafter, a vehicle equipped with the abnormality detection device 1 may be referred to as a host vehicle. However, the abnormality detection device 1 may be arranged in a place other than the vehicle in which the camera that is the detection target of the camera shift is mounted. For example, the abnormality detection device 1 may be arranged in a data center or the like that can communicate with a vehicle having the camera 21.

  In the present embodiment, since the imaging unit 2 has the plurality of cameras 21, the abnormality detection device 1 detects the camera shift for each camera 21. The abnormality detection device 1 is connected to the automatic operation control device 5 and the display device 6 by wire or wirelessly, and exchanges information with these devices 5, 6. Details of the abnormality detection device 1 will be described later.

  The sensor unit 3 has a plurality of sensors that detect information regarding the vehicle in which the camera 21 is mounted. In this embodiment, the sensor unit 3 includes a speed sensor 31 and a steering angle sensor 32. The speed sensor 31 detects the speed of the vehicle and outputs an electric signal according to the detected value. The steering angle sensor 32 detects the rotation angle of the steering wheel (handle) of the vehicle and outputs an electric signal according to the detected value. The speed sensor 31 and the steering angle sensor 32 are connected to the abnormality detection device 1 via the communication bus 4. That is, the vehicle speed information acquired by the speed sensor 31 is input to the abnormality detection device 1 via the communication bus 4. The rotation angle information of the steering wheel of the vehicle acquired by the steering angle sensor 32 is input to the abnormality detection device 1 via the communication bus 4. The communication bus 4 may be, for example, a CAN (Controller Area Network) bus.

<3. Abnormality detection device>
As shown in FIG. 1, the abnormality detection device 1 includes an image acquisition unit 11, a control unit 12, and a storage unit 13.

  The image acquisition unit 11 periodically acquires an analog or digital captured image (frame image) from the camera 21 of the vehicle at a predetermined cycle (for example, 1/30 second cycle). In the present embodiment, since the image capturing unit 2 has the plurality of cameras 21, the image acquisition unit 11 acquires the frame image from each camera 21. When the acquired frame image is analog, the analog frame image is converted into a digital frame image (A / D conversion). The image acquisition unit 11 performs a predetermined image process on the acquired frame image and outputs the processed frame image to the control unit 12.

  The control unit 12 is, for example, a microcomputer, and integrally controls the entire abnormality detection device 1. The control unit 12 includes a CPU, a RAM, a ROM and the like. The storage unit 13 is, for example, a non-volatile memory such as a flash memory, and stores various kinds of information. The storage unit 13 stores various computer programs 131 and data (not shown). Various functions of the control unit 12 are realized by the CPU performing arithmetic processing according to the computer program 131 stored in the storage unit 13. The image acquisition unit 11 may have a configuration realized by the CPU of the control unit 12 performing arithmetic processing according to the computer program 131.

  In the present embodiment, the control unit 12 executes a determination process of determining whether or not there is an abnormality in the camera 21 based on the temporal change in the position of the feature point extracted from the captured image captured by the camera 21. The determination process is provided so that the first processing mode and the second processing mode can be selected. That is, the control unit 12 may determine whether there is a camera shift according to the first processing mode or may determine whether there is a camera shift according to the second processing mode. In addition, in the present embodiment, since the imaging unit 2 includes the plurality of cameras 21, a determination process for determining whether or not there is an abnormality in the camera 21 is performed for each camera 21.

  In the first processing mode, the threshold for extracting the feature points is the first threshold. On the other hand, in the second processing mode, the threshold value for extracting the feature points is the second threshold value that is smaller than the first threshold value. A difference occurs in the extraction result of the feature points between the first processing mode and the second processing mode.

  A feature point is a point that can be prominently detected in a frame image, such as an intersection of edges in the frame image. The characteristic points are extracted from, for example, a corner of the road marking drawn with a white line or the like, a crack on the road surface, a stain on the road surface, gravel on the road surface, or the like. The feature points can be extracted using a known method such as Harris operator.

  In the present embodiment, the control unit 12 calculates the feature amount for each of the pixels forming the frame image, and extracts the pixels having the feature amount exceeding a predetermined threshold value as the feature points. The feature amount is an index of how characteristic a pixel is, and is, for example, a corner degree indicating what is called a corner. The predetermined threshold differs between the first processing mode and the second processing mode.

なお、Ｇ１１は、ｘ方向の微分結果の２乗値である。Ｇ１２は、ｘ方向の微分結果とｙ方向の微分結果の積である。Ｇ２２は、ｙ方向の微分結果の２乗値である。 In the present embodiment, the KLT method (Kanade-Lucus-Tomasi tracker) is used to calculate the feature amount. In calculating the feature amount, an xy coordinate system is defined on the frame image, and the following three parameters G11, G12, and G22 are obtained for each pixel using a Sobel filter.

Note that G11 is the squared value of the differential result in the x direction. G12 is the product of the differential result in the x direction and the differential result in the y direction. G22 is a square value of the differentiation result in the y direction.

By the processing using the Sobel filter, the following matrix M is obtained for each pixel.

The eigenvalue λ of the matrix M (equation (4)) is obtained from the following equation (5) with I as a unit matrix.

The solution of the equation (5) is obtained as a solution of the quadratic equation as λ1 and λ2 shown in the following equations (6) and (7).

In this embodiment, pixels that satisfy the following expression (8) are extracted as feature points. The eigenvalue λ1 having the minimum value is used as the feature amount. In Expression (8), T is a predetermined threshold value for detecting the feature point. Specifically, the predetermined threshold value T is the first threshold value T1 in the first processing mode and the second threshold value T2 in the second processing mode. The second threshold T2 is smaller than the first threshold T1. The first threshold T1 is set so that only feature points having a very large feature amount, such as white corners, are extracted. The second threshold T2 is set so that a large number of feature points derived from the fine unevenness of the road surface are extracted.

  The first processing mode is used when the speed of the moving body (vehicle) is higher than that in the second processing mode. In the first processing mode, feature points can be tracked by using feature points having a very large feature amount such as white line corners. For this reason, even when the speed of the vehicle is high to some extent, it is possible to accurately obtain the temporal change in the position of the characteristic point, and it is possible to suppress a decrease in the determination accuracy of the presence or absence of the camera shift. On the other hand, the second processing mode has a large number of feature points, but includes many feature points with a small feature amount. As the vehicle speed increases, it becomes difficult to track feature points with small feature amounts. For this reason, the second processing mode is a mode suitable when the vehicle is at a low speed. That is, the first processing mode is a processing mode for high speed traveling, and the second processing mode is a processing mode for low speed traveling.

  According to the configuration of the present embodiment, when the vehicle runs at high speed, the camera shift determination process is performed in the first processing mode, and when the vehicle runs at low speed, the camera shift determination process is performed in the second processing mode. be able to. That is, the camera shift can be determined by a method suitable for the traveling speed, and erroneous detection of the camera shift can be reduced.

  In the present embodiment, in detail, the first processing mode is a processing mode that is normally used. The second processing mode is a processing mode used when a predetermined processing result is obtained by the first processing mode. That is, in principle, the abnormality detection device 1 uses information obtained from a vehicle traveling at high speed to determine whether or not there is a camera shift in the first processing mode. Then, the abnormality detection device 1 exceptionally switches from the first processing mode to the second processing mode only under specific conditions, and determines whether or not there is a camera shift using information obtained from a vehicle traveling at a low speed. To do. According to the present embodiment, it is possible to suppress the processing relating to the detection of the camera shift during low speed traveling, and it is possible to execute the processing unrelated to the detection of the camera shift during the low speed traveling.

  In the present embodiment, the control unit 12 is provided so as to be able to perform a recognition process of recognizing the surrounding environment of the moving body (vehicle) based on a captured image captured by the camera 21. The recognition process is, for example, a process of extracting an edge from a frame image and recognizing a moving object or a stationary object around the vehicle. For example, a moving object or a stationary object is recognized by a known pattern matching process or a calculation process using a neural network. In other words, the control unit 12 has a function other than the function as a device for detecting an abnormality of the camera 21. In the present embodiment, the control unit 12 also has a function as a device that performs parking assistance.

  When the speed of the moving body (vehicle) is normally the speed corresponding to the second processing mode, the control unit 12 performs recognition processing for recognizing the surrounding environment of the vehicle and determines whether or not there is an abnormality in the camera 21. Priority is given to the determination process. In the present embodiment, during normal times, the control unit 12 gives priority to the peripheral environment recognition processing at low speed traveling, and gives priority to the determination processing for judging the presence or absence of camera displacement at high speed traveling. According to this, it is possible to prevent the processing of the control unit 12 from being excessively concentrated at one time, and it is possible to reduce the processing load of the control unit 12. Since the recognition processing of the surrounding environment for parking assistance of the vehicle is performed at low speed traveling, the configuration of the present embodiment is suitable when the control unit 12 also has a function as a device for performing parking assistance.

  FIG. 2 is a flowchart showing an example of a camera shift detection flow by the abnormality detection device 1 of the first embodiment. The process of detecting the camera shift is performed, for example, at a predetermined time period (every week, etc.), at a predetermined traveling distance (every 100 km, etc.), every time the engine is started (every time the ignition (IG) is turned on), and when a predetermined number of engine starts are reached. It may be executed every time. In the present embodiment, since the image capturing unit 2 includes the four cameras 21, the camera shift detection flow shown in FIG. 2 is performed for each camera 21. In order to avoid duplicated description, here, the detection flow of the camera shift will be described using the case where the camera 21 is a front camera as a typical example.

  As shown in FIG. 2, first, the control unit 12 monitors whether or not the vehicle equipped with the camera 21 is traveling straight (step S1). Whether the vehicle is traveling straight can be determined based on, for example, the rotation angle information of the steering wheel obtained from the steering angle sensor 32. It should be noted that straight traveling includes both forward traveling and backward traveling.

  The control unit 12 repeats the monitoring of step S1 until it detects that the vehicle is going straight. That is, the control unit 12 advances the process regarding the camera shift on condition that the vehicle is traveling straight ahead. According to this, since the process relating to the detection of the camera shift can be performed without using the information when the traveling direction of the vehicle is bent, it is possible to prevent the information processing from becoming complicated.

  When it is determined that the vehicle is traveling straight (Yes in step S1), the control unit 12 confirms whether the speed of the vehicle is within the first speed range (step S2). The first speed range may be, for example, 15 km or more per hour and 30 km or less. It is preferable that the first speed range is set to a speed higher than a speed at which parking assistance of the vehicle is performed. If the speed of the vehicle becomes too fast, it will be difficult to track the characteristic points. Therefore, it is preferable that the speed of the vehicle is not too fast.

  When the speed of the vehicle is outside the first speed range (No in step S2), the control unit 12 returns to step S1. That is, the control unit 12 advances the process relating to the camera shift on the condition that the vehicle is traveling straight ahead and that the vehicle speed is within the first speed range. In the present embodiment, as a general rule, the processing relating to the camera shift is not started when the vehicle is traveling at a low speed, so that it is possible to prevent the processing for recognizing the surrounding environment of the vehicle and the processing relating to the camera shift from proceeding at the same time. It is possible to prevent the processing load of the above from being concentrated at one time.

  When it is determined that the vehicle is traveling within the first speed range (Yes in step S2), the control unit 12 performs a process of determining whether or not there is a camera shift in the first processing mode (step S3). The order of step S1 and step S2 may be exchanged.

  FIG. 3 is a flowchart showing an example of a process of determining the presence / absence of a camera shift in the first processing mode in the abnormality detection device 1 of the first embodiment. As shown in FIG. 3, the control unit 12 performs a process of extracting feature points from the frame image (step S31). In the first processing mode, the first threshold T1 for extracting the feature points is set to a high value. For this reason, the extracted feature points are feature points having a large feature amount indicating the likelihood of a corner.

  FIG. 4 is a diagram for explaining a method of extracting the feature point FP. FIG. 4 schematically shows a frame image P captured by the camera 21 (front camera). As shown in FIG. 4, the control unit 12 sets a predetermined extraction region ER in the frame image P. The control unit 12 extracts the feature point FP from the predetermined extraction area ER. The predetermined extraction region ER is set at a position where the road surface RS portion appears. It is preferable that the predetermined extraction region ER is set in a wide range including the central portion C of the frame image P. As a result, even if the locations where the feature points FP are generated are not uniform but are unevenly distributed in the uneven range, the feature points FP can be extracted.

  Although the number of feature points FP is two in FIG. 4, this number is for convenience and does not indicate the actual number. In some cases, a plurality of feature points FP may be extracted, and in some cases only one feature point FP may be extracted. Moreover, the feature point FP may not be extracted.

  Returning to FIG. 3, when the feature point FP extraction processing is performed, it is confirmed whether the number of feature points is equal to or more than a predetermined number (step S32). When the number of characteristic points is less than the predetermined number (No in step S32), the control unit 12 determines that the presence / absence of a camera shift cannot be determined (step S37), and ends the determination process in the first processing mode. On the other hand, when the number of feature points is equal to or greater than the predetermined number (Yes in step S32), the optical flow indicating the movement of the feature point FP between the two frame images input at the predetermined time interval is calculated. (Step S33). The predetermined number may be one or more, and the number may be determined by, for example, an experiment or a simulation.

  FIG. 5 is a diagram for explaining a method for obtaining the optical flow OF. FIG. 5 is a schematic view shown for convenience as in FIG. FIG. 5 shows a frame image (current frame image) P ′ captured by the camera 21 after a predetermined capturing time interval (corresponding to one cycle) has elapsed after capturing the frame image (previous frame image) P shown in FIG. Is. After the frame image P shown in FIG. 4 is captured, the host vehicle is moving backward until a predetermined time interval elapses. The dotted circles shown in FIG. 5 indicate the positions of the feature points FP extracted in the previous frame image P shown in FIG.

  As shown in FIG. 5, when the host vehicle moves backward, the feature point FP existing in front of the host vehicle separates from the host vehicle. That is, the feature point FP appears at a different position between the current frame image P ′ and the previous frame image P. The control unit 12 associates the feature point FP of the current frame image P ′ with the feature point FP of the previous frame image P in consideration of the pixel values in the vicinity thereof, and associates them with each of the associated feature points FP. Based on this, the optical flow OF is obtained. The control unit 12 does not perform the calculation process of the optical flow OF when there is no previous frame image (frame image of one cycle before). Also in this case, the presence or absence of camera shift is not determined. Further, when a plurality of feature points FP are extracted from the frame image P, the calculation process of the optical flow OF is performed on each feature point FP.

  Returning to FIG. 3, when the optical flow OF of the feature point FP is calculated, the control unit 12 performs coordinate conversion of the optical flow OF given in the camera coordinate system to calculate the movement vector V (step S34). The coordinate conversion is a process of converting the camera coordinate system into a coordinate system on the road surface. Note that the control unit 12 may convert the coordinates of the feature point FP extracted from the frame image P into coordinates on the road surface first, and may calculate the movement vector without calculating the optical flow OF.

  FIG. 6 is a diagram for explaining the coordinate conversion process. As shown in FIG. 6, the control unit 12 converts the optical flow OF viewed from the position of the camera 21 (viewpoint VP1) into a movement vector V viewed from a viewpoint VP2 above the road surface RS where the vehicle is present. The control unit 12 converts the optical flow OF on the frame image into the movement vector V by projecting the optical flow OF on the virtual plane RS_V corresponding to the road surface. The magnitude of the movement vector V indicates the movement amount (movement distance) of the vehicle on the road surface RS. In this embodiment, since the camera 21 is a fisheye lens, it is preferable that the coordinate conversion process includes distortion correction. When a plurality of optical flows OF are obtained, the movement vector V is calculated for each optical flow OF.

  Returning to FIG. 3, when the movement vector V indicating the movement on the road surface RS is calculated, the control unit 12 obtains an estimated value of the movement amount (movement distance) based on the movement vector V (step S35). In the present embodiment, the control unit 12 sets the length of the front-rear direction component of the movement vector V as the estimated value of the front-rear movement amount. Further, the control unit 12 sets the length of the horizontal component of the movement vector V as the estimated value of the horizontal movement amount.

  When the number of movement vectors V is plural, the estimated value may be the average value of the lengths obtained from the movement vectors V in the front-rear direction and the left-right direction. When there are a plurality of movement vectors V, the estimated values of the movement amounts in the front-rear direction and the left-right direction may be obtained using only the movement vector V obtained from the feature point FP having the largest feature amount. . When the number of movement vectors V is large, the estimated values of the movement amounts in the front-rear direction and the left-right direction may be obtained by statistical processing using a histogram described later.

  When the estimated value of the movement amount is obtained, the control unit 12 compares the estimated value with the comparison value obtained from the information from the sensor unit 3 to determine whether there is a camera shift (step S36). In this embodiment, the estimated value and the comparison value are compared with respect to the amount of movement in the front-rear direction. Further, the estimated value and the comparison value are compared with respect to the movement amount in the left-right direction. Using the comparison value as the correct answer value, the magnitude of the deviation of the estimated value from the correct answer value is determined. When the magnitude of the shift exceeds a predetermined threshold value, it is determined that the camera shift has occurred.

  The comparison value (movement amount) in the front-rear direction can be calculated from the shooting time intervals of the two frame images for deriving the optical flow OF and the speed of the host vehicle obtained from the speed sensor 31 in the time intervals. . Further, in the present embodiment, the presence / absence of a camera shift is determined based on the frame image obtained when the vehicle is traveling straight in the front-rear direction. Therefore, the comparison value (movement amount) in the left-right direction becomes zero.

  The comparison value may be obtained based on information from other than the sensor unit 3. For example, the comparison value may be obtained based on information obtained from a GPS sensor or a camera other than the camera shift determination target. Further, the estimated value and the comparison value may not be the movement amount, and may be the speed, for example.

  FIG. 7 is a flowchart showing an example of the deviation determination performed by the control unit 12 in the first embodiment. The process shown in FIG. 7 is a detailed process example of step S36 of FIG.

  First, the control unit 12 confirms whether or not the magnitude of the difference between the estimated value and the comparison value (the amount of deviation in the front-rear direction) of the movement amount of the vehicle in the front-rear direction is smaller than the first deviation threshold ( Step S361). When the shift amount in the front-rear direction is equal to or larger than the first shift threshold (No in step S361), the control unit 12 determines that there is a camera shift (step S364).

  On the other hand, when the shift amount in the front-rear direction is smaller than the first shift threshold (Yes in step S361), the control unit 12 determines the magnitude of the difference between the estimated value and the comparison value (left and right) with respect to the movement amount of the vehicle in the left-right direction. It is confirmed whether or not the (direction deviation amount) is smaller than the second deviation threshold value (step S362). When the shift amount in the left-right direction is equal to or larger than the second shift threshold (No in step S362), the control unit 12 determines that there is a camera shift (step S364).

  On the other hand, when the shift amount in the left-right direction is smaller than the second shift threshold (Yes in step S362), the control unit 12 determines that there is no camera shift (step S363).

  In the present embodiment, if any one of the front-rear direction movement amount and the left-right direction movement amount has a large displacement amount, it is determined that a camera displacement has occurred. According to this, it is possible to reduce the possibility that it is determined that the camera shift has not occurred even though the camera shift has occurred. However, this is an example. For example, the configuration may be such that it is determined that the camera shift occurs only when the shift amount becomes large in both the front-back direction movement amount and the left-right direction movement amount. Further, the determination of the camera shift is made not only by the amount of movement in the front-rear direction and the left-right direction, but also by, for example, an index combining these (for example, the sum of the square value of the amount of movement in the front-back direction and the square value of the amount of movement in the left-right direction). ). The camera misalignment may be determined using only one of the front-rear movement amount and the left-right movement amount.

  Further, in the present embodiment, the amount of movement in the front-rear direction and the amount of movement in the left-right direction are compared in order, but these comparisons may be performed at the same timing. Further, in the case of the configuration in which the amount of movement in the front-rear direction and the amount of movement in the left-right direction are compared in order, the order is not particularly limited, and the comparison may be performed in an order different from the order shown in FIG. 7. Further, in the above, the presence or absence of the camera shift is determined based on the processing result of one frame image, but this is merely an example. It may be configured to determine whether or not there is a camera shift based on the processing results of a plurality of frame images. As a result, it is possible to reduce the possibility of erroneously determining whether or not there is a camera shift.

  Returning to FIG. 2, when the process of determining the presence / absence of a camera shift in the first processing mode ends, the control unit 12 determines whether or not the processing result is a predetermined processing result (step S4). When the processing result is the predetermined processing result (Yes in step S4), it is determined that it is necessary to switch from the first processing mode to the second processing mode and perform the processing for determining whether or not there is a camera shift (step S5). On the other hand, when the processing result is not the predetermined processing result (No in step S4), the process returns to step S1 and the processes after step S1 are repeated.

  FIG. 8 is a flowchart showing a detailed example of processing for determining whether or not the processing result in the first processing mode is a predetermined processing result. FIG. 8 is a flowchart showing a detailed example of step S4 in FIG. As shown in FIG. 8, it is confirmed whether or not the camera shift is determined by the determination processing in the first processing mode (step S41).

  If it is determined that there is a camera shift (Yes in step S41), the process proceeds to step S5 shown in FIG. On the other hand, if it is not determined that there is a camera shift (No in step S41), it is confirmed whether or not the predetermined period determination is undetermined (step S42). In the first processing mode, when it is not continuously determined whether or not there is a camera shift over a plurality of predetermined frames, it is determined as undetermined for a predetermined period.

  When the determination is not made for the predetermined period (Yes in step S42), the process proceeds to step S5 shown in FIG. On the other hand, when the determination is not made for the predetermined period (No in step S42), the determination that there is no camera shift is confirmed (step S43), the process returns to step S1 and the processes in step S1 and subsequent steps are repeated.

  As described above, in the present embodiment, the predetermined processing result shown in step S4 of FIG. 2 includes the processing result indicating that the camera 21 has an abnormality (there is a camera shift). According to this, when a camera shift is recognized in the first processing mode performed during high-speed traveling, the camera shift determination processing in the second processing mode due to low-speed traveling is performed, so that the erroneous detection of the camera shift is prevented. It can be reduced.

  Further, in the present embodiment, the predetermined processing result shown in step S4 of FIG. 2 includes the processing result that it is not possible to determine the presence / absence of abnormality of the camera 21 (presence / absence of camera shift) for a predetermined period. . The situation in which the feature points FP are not detected by a predetermined number or more occurs not only when there is no feature point FP having a high feature amount such as a corner of a white line, but also when the camera 21 is largely attached. This difference cannot be clearly distinguished only by the first processing mode. When the state in which the presence or absence of the camera shift cannot be determined is repeated as in the present embodiment, the camera shift is appropriately performed by performing the determination process in the second processing mode that lowers the threshold value for extracting the feature point FP. Can be detected.

  Returning to FIG. 2, when the second processing mode after step S5 is executed, the control unit 12 determines the abnormality (camera shift) of the camera 21 rather than the recognition processing of the surrounding environment of the vehicle using the camera 21. Give priority to processing. According to this, only when there is a possibility that a camera shift has occurred when the vehicle is traveling at a low speed, the camera shift determination process has priority over the surrounding environment recognition process. Therefore, it is possible to appropriately detect the abnormality of the camera 21 while giving priority to the processing (the recognition processing of the surrounding environment) according to the purpose of mounting the camera 21 in the vehicle as much as possible.

  In the present embodiment, the control unit 12 requests the automatic driving control device 5 to perform automatic driving when switching from the first processing mode to the second processing mode. In response to the request, the moving body control system SYS1 automatically drives the moving body (vehicle) when performing the determination process of determining whether the camera 21 is abnormal (camera shift) in the second processing mode. In addition, it is preferable that the automatic operation is started at a timing at which safety can be secured after it is determined that the first processing mode needs to be switched to the second processing mode. This determination may be performed by the automatic driving control device 5, and the automatic driving control device 5 notifies the control unit 12 of the start of the automatic driving. For example, automatic driving is temporarily performed when the vehicle starts or stops.

  According to the present embodiment, it is possible to accurately perform straight traveling with the steering wheel kept constant while the vehicle travels at a low speed within a predetermined speed range, and it is possible to accurately determine whether or not there is a camera shift in the second processing mode. You can do it well.

  When the automatic operation for the determination processing in the second processing mode is started, it is confirmed whether or not the vehicle is traveling straight ahead, as in the case of the first processing mode (step S6). Then, when the vehicle is traveling straight ahead (Yes in step S6), the control unit 12 confirms whether or not the speed of the vehicle is within the second speed range (step S7). The second speed range may be, for example, 3 km or more per hour and 5 km or less.

  When the vehicle speed is outside the second speed range (No in step S7), the control unit 12 returns to step S6. That is, the control unit 12 advances the process relating to the camera shift on the condition that the vehicle is traveling straight ahead and that the vehicle speed is within the second speed range. When it is determined that the vehicle is traveling within the second speed range (Yes in step S7), the control unit 12 performs a process of determining whether or not there is a camera shift in the second processing mode (step S8).

  FIG. 9 is a flowchart showing an example of a process for determining whether or not there is a camera shift in the second processing mode in the abnormality detection device of the first embodiment. The determination process in the second processing mode is similar to the determination process in the first processing mode shown in FIG. Detailed description of the same processing contents as those in FIG. 3 will be omitted.

  As shown in FIG. 9, the control unit 12 performs a process of extracting the feature point FP from the frame image (step S81). In the second processing mode, the pixels having the characteristic amount exceeding the second threshold T2 are extracted as the characteristic points FP. The second threshold T2 is set to a value lower than the first threshold T1 (threshold of the first processing mode). For this reason, a large number of feature points FP derived from the fine unevenness of the road surface are extracted. For example, many feature points FP are extracted from a road surface having many irregularities such as an asphalt road surface. Note that the number of feature points FP tends to decrease on a smooth road surface such as a concrete road surface. That is, it may happen that the number of feature points to be extracted cannot be sufficiently obtained depending on the condition of the road surface. In consideration of such a point, similarly to the first processing mode, in the second processing mode, when the number of feature points is less than the predetermined number, the determination is not possible and the determination processing may be performed again.

  When the feature point FP extraction processing is performed, the control unit 12 calculates the optical flow OF (step S82). When the optical flow OF is calculated, the control unit 12 performs coordinate conversion of the optical flow OF to calculate the movement vector V (step S83). These processes are similar to those in the first processing mode.

  When the movement vector V is calculated, the control unit 12 generates a histogram based on the plurality of movement vectors V (step S84). In this embodiment, the control unit 12 divides each movement vector V into two components in the front-rear direction and the left-right direction to generate a first histogram and a second histogram. The control unit 12 calculates the estimated value of the movement amount using the first histogram and the second histogram (step S85).

  FIG. 10 is a diagram showing an example of the first histogram HG1 generated by the control unit 12. FIG. 11 is a diagram showing an example of the second histogram HG2 generated by the control unit 12. It should be noted that the control unit 12 may perform the process of excluding the movement vector V satisfying a predetermined condition from all the previously obtained movement vectors V before and after the generation of the histograms HG1 and HG2. Good. For example, the movement vector V that largely deviates from the expected size or direction from the speed, steering angle, shift lever position, etc. of the host vehicle may be excluded. Further, for example, in the histograms HG1 and HG2, the movement vector V belonging to a class having an extremely low frequency may be excluded.

  The first histogram HG1 shown in FIG. 10 is a histogram obtained based on the front-back direction component of the movement vector V. The first histogram HG1 is a histogram in which the number of movement vectors V is a frequency, and the amount of movement in the front-rear direction (length of the front-rear direction component of the movement vector) is a class. The second histogram HG2 shown in FIG. 11 is a histogram obtained based on the horizontal component of the movement vector V. The second histogram HG2 is a histogram in which the number of movement vectors V is a frequency and the amount of movement in the left-right direction (the length of the left-right component of the movement vector) is a class.

  10 and 11 are histograms obtained when the vehicle travels straight backward. FIG. 10 and FIG. 11 are examples of histograms when the camera shift has not occurred. The first histogram HG1 has a normal distribution shape in which the frequency increases toward a specific amount of movement (class) on the rear side. On the other hand, the second histogram HG2 has a normal distribution shape in which the frequency increases due to being biased toward the class near the movement amount of zero. When the camera shift occurs, the shape of the histogram changes.

  In the present embodiment, the control unit 12 sets the median value (median) of the first histogram HG1 as the estimated value of the movement amount in the front-rear direction. The control unit 12 sets the median value of the second histogram HG2 as the estimated value of the movement amount in the left-right direction. However, the method of determining the estimated value by the control unit 12 is not limited to this. The control unit 12 may use, for example, the movement amount (moderate value) of the class in which the frequency of each of the histograms HG1 and HG2 is maximum as the estimated value.

  When the estimated value of the movement amount is obtained, the control unit 12 compares the estimated value with the comparison value obtained from the information from the sensor unit 3 to determine whether there is a camera shift, as in the case of the first processing mode. Is determined (step S86). Since the determination of the presence / absence of the camera shift is the same as the processing shown in FIG. 7, detailed description thereof will be omitted. In the second processing mode, since the amount of movement can be estimated using a large number of feature points, the accuracy of the process of determining whether or not there is a camera shift can be improved compared to the first processing mode. Similarly to the first processing mode, the second processing mode may be configured to determine whether or not there is a camera shift based on the processing results of a plurality of frame images.

  When it is determined in the second processing mode that there is a camera shift, the abnormality detection device 1 detects the camera shift. When the camera shift is detected, it is preferable that the abnormality detection device 1 perform a process for displaying the occurrence of the camera shift on the display device 6 to notify the driver or the like of the abnormality of the camera 21. Further, the abnormality detection device 1 preferably performs a process for stopping (turning off) a function (for example, an automatic parking function) that provides driving assistance using information from the camera 21. At this time, it is preferable to display on the display device 6 that the driving support function is stopped. When a plurality of cameras 21 are mounted on the vehicle, if even one of the plurality of cameras 21 causes a camera shift, the above-described driver notification process and driver assisting device stop process are performed. Is preferably performed.

  When it is determined that there is no camera shift in the second processing mode, the abnormality detection device 1 determines that no camera shift is detected, and temporarily ends the camera shift detection process. Then, after that, the camera shift detection process is restarted at a predetermined timing.

  In the present embodiment, normally, the first processing mode for determining the presence / absence of a camera shift during high-speed traveling is used, and only when a result in which the occurrence of a camera shift is suspected is obtained in the first processing mode, the camera during low-speed traveling is used. The second processing mode for determining the presence / absence of deviation is used. Therefore, during low-speed traveling, in principle, the peripheral environment recognition process can be performed by the camera 21 without being disturbed by the camera shift detection process. Further, since the camera shift can be determined using the two processing modes, it is possible to reduce the possibility of erroneously detecting the camera shift.

  Further, in the present embodiment, it is possible to properly detect that the mounting state of the camera 21 is largely deviated. This will be described with reference to FIGS. 12 and 13. FIG. 12 is a schematic diagram illustrating a captured image P captured by the camera 21 in which a large camera shift has occurred. FIG. 13 is a diagram exemplifying the first histogram HG1 generated based on the captured image P captured by the camera 21 in which a large camera shift has occurred.

  In the example shown in FIG. 12, the camera 21 is largely displaced, and the sky and distant buildings (three-dimensional objects) are mainly reflected in the feature point extraction area ER. Normally, the feature point FP of a three-dimensional object in the sky or a distant object is not extracted as the feature point FP in the first processing mode in which a large threshold value is set because the feature amount is small. That is, when a large camera shift occurs, in the first processing mode, the feature points FP are not extracted by the predetermined number or more, and the state in which the camera shift cannot be determined continues for the predetermined period or more. As a result, the camera shift determination process is performed by switching from the first process mode to the second process mode. In the second processing mode, since the threshold value for extracting the feature point FP is reduced, the feature point FP derived from the sky or a distant three-dimensional object is extracted.

  As shown in FIG. 13, the moving vector V of the feature point FP derived from the sky and a distant three-dimensional object has a vector magnitude of zero or a value near zero even though the vehicle is moving. This is because the feature point FP derived from the sky and a distant three-dimensional object exists at a position very far from the own vehicle. As a result, when a large camera shift occurs, the estimated value and the comparison value greatly shift, and the camera shift can be detected.

  As described above, in the present embodiment, in both the determination processing in the first processing mode and the determination processing in the second processing mode, the movement information of the vehicle (moving body) is calculated from the temporal change of the position of the feature point. For example, an estimated value of the movement amount) is obtained. Then, in any of the determination processes, the presence / absence of abnormality of the camera 21 is determined based on the estimated value of the movement information. That is, according to this configuration, the estimated value of the movement information of the vehicle can be obtained when determining the presence or absence of the abnormality of the camera 21 in both the first processing mode and the second processing mode.

<4. Modification>
In the above, the configuration is such that the automatic operation is performed when the determination process of the presence or absence of the camera shift is performed in the second processing mode, but this is merely an example. When performing the determination process of the presence or absence of the camera shift in the second processing mode, driving by the driver (manual driving) may be performed. In this case, when switching from the first processing mode to the second processing mode, the control unit 12 causes the display device 6 mounted on the moving body (vehicle) to perform an operation suitable for the second processing mode. It is preferable to perform a process of displaying a message prompting the user to do so. The display content of the display screen may include, for example, that it is necessary to perform a camera shift determination process, and what kind of driving is required for the determination process. By the display processing by the display device 6, the driver can recognize that it is necessary to determine the camera shift, and can start the driving according to the recognition. In addition to the message displayed on the screen, or in place of the message displayed on the screen, for example, a configuration may be adopted that prompts a driving suitable for the second processing mode by voice. Further, even when the determination process is performed in the first processing mode, the display device 6 may be configured to display a message prompting the driver to perform an operation suitable for the first processing mode. Further, in some cases, the configuration may be such that automatic operation is performed when the process of determining whether or not there is a camera shift is performed in the first processing mode.

  Further, in the above, the first processing mode is normally used, and the second processing mode is used only when a predetermined processing result is obtained in the first processing mode. Is. The second processing mode may be used regardless of the processing result of the first processing mode. For example, when it is determined by the navigation device or the like that the host vehicle is traveling in a place other than the parking lot, it is assumed that the host vehicle is traveling at a low speed (for example, 3 km or more and 5 km or less per hour). The camera shift detection process using the second process mode instead of the one process mode may be performed.

  Further, in the above, the number of processing modes that can be switched and used for the determination processing is two, but the number of processing modes may be three or more.

  Further, in the above description, the data used for detecting the abnormality of the camera 21 is collected when the host vehicle is traveling straight ahead. However, this is an example, and the data used for detecting the abnormality of the camera 21 may be collected when the vehicle is not traveling straight ahead.

<< B. Second embodiment >>
<1. Anomaly detection system>
Next, a second embodiment will be described. In the description of the second embodiment, the contents overlapping with those of the first embodiment will be omitted unless particularly necessary. Further, the same reference numerals are given to the same parts as those in the first embodiment.

  FIG. 14 is a block diagram showing the configuration of the abnormality detection system SYS2A according to the second embodiment of the present invention. Similar to the abnormality detection system SYS2 of the first embodiment, the abnormality detection system SYS2A of the second embodiment includes an abnormality detection device 1A, a photographing unit 2, and a sensor unit 3. The abnormality detection system SYS2A is also a system for detecting a camera shift, as in the first embodiment. Since the configurations of the image capturing unit 2 and the sensor unit 3 are the same as those in the first embodiment, the description thereof will be omitted.

<2. Abnormality detection device>
The abnormality detection device 1A is provided for each vehicle equipped with a camera, as in the first embodiment. The abnormality detection device 1A is a device (camera shift detection device) that detects an abnormality based on a captured image captured by a camera.

  As shown in FIG. 14, the abnormality detection device 1A includes an image acquisition unit 111, a control unit 12A, and a storage unit 13A. The configuration of the image acquisition unit 111 is the same as that of the first embodiment, and therefore its description is omitted.

  As in the first embodiment, the control unit 12A is a computer including, for example, a CPU, a RAM, a ROM, and the like, and controls the entire abnormality detection device 1A as a whole. The storage unit 13A is a non-volatile memory, and stores various kinds of information including a program as firmware, as in the first embodiment. Various functions of the control unit 12A are realized by the CPU performing arithmetic processing according to a computer program stored in the storage unit 13A.

  In the second embodiment as well, similar to the first embodiment, the control unit 12A determines whether or not there is an abnormality in the camera 21 based on the temporal change in the position of the feature point extracted from the captured image captured by the camera 21. Execute the judgment process. The determination process is provided so that the first processing mode and the second processing mode can be selected. However, the determination processing method and the processing mode selection method are different from those in the first embodiment. Hereinafter, this difference will be mainly described. Also in the present embodiment, the number of cameras 21 is plural, and the determination process is performed for each camera 21.

  The control unit 12A includes a mode selection unit 120, an extraction unit 121, a specification unit 122, an estimation unit 123, and a determination unit 124. The mode selection unit 120, the extraction unit 121, the identification unit 122, the estimation unit 123, and the determination unit 124 are conceptual components. The functions executed by one component may be distributed to a plurality of components, or the functions possessed by a plurality of components may be integrated into one component.

  The mode selection unit 120 selects the first processing mode and the second processing mode. Specifically, the mode selection unit 120 selects the first processing mode and the second processing mode according to the speed of the vehicle. The first processing mode is used when the speed of the vehicle is higher than that in the second processing mode.

  The extraction unit 121 extracts a plurality of feature points including the first feature point and the second feature point from the image captured by the camera 21. The extraction unit 121 calculates a feature amount for each of the pixels forming the captured image, and extracts a pixel having a feature amount exceeding a predetermined threshold value as a feature point. The predetermined threshold is the first threshold in the first processing mode and the second threshold in the second processing mode. The second threshold is smaller than the first threshold. The first threshold value is set so that only feature points having a very large feature amount such as a white line corner are extracted. The second threshold value is set so that a large number of feature points derived from fine irregularities on the road surface are extracted. The extraction unit 121 selects the first feature point and the second feature point from the feature points on the road surface. In this embodiment, a road surface is assumed as the predetermined plane.

  The identifying unit 122 derives each optical flow of the first feature point and the second feature point extracted from the two captured images captured at different times. The optical flow is a motion vector indicating the motion of the feature point between two captured images captured at different times. The interval between different times may be the same as the frame period of the image acquisition unit 11, or may be a multiple of the frame period of the image acquisition unit 11.

  The identifying unit 122 uses the optical flows of the first feature point and the second feature point to determine the positional changes of the first feature point and the second feature point extracted from the two captured images captured at different times. calculate. The specifying unit 122 specifies two sets of planes whose intersecting lines with a predetermined plane are parallel to each other based on the position change of the first feature point and the position change of the second feature point. The processing of the identifying unit 122 may be the same in the first processing mode and the second processing mode.

  The estimation unit 123 estimates the posture of the camera (camera attachment angle) based on the set of surfaces identified by the identification unit 122. For example, when the extraction unit 121 extracts the first feature point and the second feature point from the front camera, the estimation unit 123 estimates the posture of the front camera (camera mounting angle). The processing of the estimation unit 123 may be the same in the first processing mode and the second processing mode.

  The determination unit 124 determines whether or not there is a camera shift based on the posture of the camera 21 estimated by the estimation unit 123. If it is determined that there is a camera shift, the abnormality detection device 1A has detected an abnormality of the camera 21. For example, the abnormality detection device 1A notifies the user of the vehicle of this abnormality, so that the user of the vehicle can take measures such as requesting a camera mounting adjustment by the dealer. The processing of the determination unit 124 may be the same in the first processing mode and the second processing mode.

  The image estimating unit 11, the mode selecting unit 120, the extracting unit 121, the specifying unit 122, and the estimating unit 123 configure the posture estimating device 14. From the viewpoint of the posture estimation device 14, the control unit 12A estimates the posture of the camera 21 based on the change in the position of the feature point extracted from the captured image captured by the camera 21 mounted on the vehicle (moving body). The estimation process is performed. The estimation processing is provided so that the first processing mode and the second processing mode can be selected. The abnormality detection device 1A including the posture estimation device 14 determines whether or not there is an abnormality in the camera 21 based on the estimation processing by the posture estimation device 14.

  FIG. 15 is a flowchart showing an example of a camera shift detection flow in the abnormality detection device 1A according to the second embodiment of the present invention. The camera shift detection process is repeatedly performed at a predetermined timing after the ignition is turned on (power on in the case of an electric vehicle). Hereinafter, the detection process of the camera shift will be described taking the case where the camera 21 is the front camera as an example.

  As shown in FIG. 15, first, the control unit 12A monitors whether or not the vehicle equipped with the camera 21 is traveling straight (step S101). The monitoring is the same as in the case of the first embodiment. The control unit 12A does not determine whether or not there is a camera shift until the vehicle starts traveling straight ahead. In the process of step S104 described later, if the optical flow of the first feature point and the optical flow of the second feature point that do not overlap each other can be derived, the posture of the camera can be estimated, so unlike the present embodiment, the vehicle The camera misalignment may be determined when the traveling direction of is curved.

  In this embodiment, when it is determined that the vehicle is traveling straight (Yes in step S101), the mode selection unit 120 selects the processing mode of the determination process (step S102). FIG. 16 is a flowchart showing an example of processing mode selection processing. FIG. 16 is a flowchart showing a detailed example of step S102 in FIG.

  First, the mode selection unit 120 confirms whether or not the speed of the vehicle is within the first speed range (step S1021). The speed of the vehicle is obtained from the speed sensor 31, for example. The first speed range is, for example, not less than 5 km / h and less than 30 km / h.

  When the speed of the vehicle is within the first speed range (Yes in step S1021), the mode selection unit 120 selects the first processing mode (step S1022). On the other hand, when the vehicle speed is outside the first speed range (No in step S1021), the mode selection unit 120 confirms whether the vehicle speed is within the second speed range (step S1023). The second speed range is, for example, 3 km or more per hour and less than 5 km per hour.

  When the speed of the vehicle is within the second speed range (Yes in step S1023), the mode selection unit 120 selects the second processing mode (step S1024). On the other hand, when the speed of the vehicle is outside the second speed range (No in step S1023), the processing mode is not selected and the process returns to step S101. For example, when the speed of the vehicle is 30 km / h or higher, or when the speed is less than 3 km / h, the process returns to step S101. Consider that if the vehicle speed is too fast, it may not be possible to follow the movement of the feature points, and if the vehicle speed is too slow, the length of the optical flow may be shortened and the accuracy of the determination may decrease. It is a thing.

  In the present embodiment, the configuration is made in which it is confirmed whether or not it is within the second speed range after it is confirmed whether or not it is within the first speed range, but this is merely an example. For example, the configuration may be such that it is confirmed whether or not it is within the first speed range after it is confirmed whether or not it is within the second speed range.

  Returning to FIG. 15, when the processing mode is selected, the extraction unit 121 extracts feature points (step S103). The feature point extraction method is the same as that in the first embodiment. When the first processing mode is selected, only feature points with a high degree of corner, such as white line corners, are extracted. On the other hand, when the second processing mode is selected, a large number of feature points derived from, for example, the unevenness of the road surface (eg, asphalt road surface) are obtained in addition to the feature points having a high degree of corner such as the corners of the white line.

  Note that preprocessing may be performed on the captured image before extracting the characteristic points. For example, when the first processing mode is selected, noise may be removed from the captured image before extracting the characteristic points so that the characteristic points with poor tracking accuracy are not erroneously extracted. In addition, the feature point extraction process performed by the extraction unit 121 may not provide the feature points required for the following processes. In such a case, it may be determined that the presence / absence of a camera shift cannot be determined, and the camera shift detection process may be temporarily terminated.

  In the present embodiment, the extraction unit 121 extracts the first feature point FP1 and the second feature point FP2. FIG. 17 is a diagram exemplifying the first feature points FP1 and the second feature points FP2 extracted by the extraction unit 121. Note that FIG. 17 is a diagram schematically showing a captured image P captured by the camera 21. The captured image P includes a region BO in which the body of the vehicle is reflected. The first feature point FP1 and the second feature point FP2 exist on the road surface RS. In FIG. 17, the first characteristic point FP1 and the second characteristic point FP2 are present in the portion of the numeral indicating the speed limit drawn on the road surface.

  Note that the number of feature points extracted by the extraction unit 121 may be more than two. In particular, a large number of feature points are extracted in the second processing mode. When more than two feature points are extracted, two feature points suitable for the following processing may be selected from the plurality of feature points. Alternatively, two or more sets of the first feature point FP1 and the second feature point FP2 may be extracted and the following process may be performed on each of them.

  Returning to FIG. 15, when the first feature point FP1 and the second feature point FP2 are extracted, the identifying unit 122 determines the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2. Is derived (step S104).

  FIG. 18 is a diagram for explaining a method of deriving the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2. FIG. 18 is a schematic diagram shown for convenience as in FIG. FIG. 18 is a photographed image (current frame P ′) photographed by the camera 21 after a predetermined time has elapsed after the photographed image (previous frame P) shown in FIG. 17 was photographed. After the captured image P shown in FIG. 17 is captured, the vehicle is traveling straight ahead until a predetermined time elapses. A circle FP1P shown in FIG. 18 indicates the position of the first feature point FP1 at the time of capturing the captured image P shown in FIG. A circle FP2P shown in FIG. 18 indicates the position of the second feature point FP2 at the time of shooting the shot image P shown in FIG.

  As shown in FIG. 18, when the vehicle travels straight ahead, the first feature point FP1 and the second feature point FP2 existing in front of the vehicle approach the vehicle. That is, the first feature point FP1 and the second feature point FP2 appear at different positions in the current frame P ′ and the previous frame P. The identifying unit 122 associates the first feature point FP1 of the current frame P ′ and the first feature point FP1 of the previous frame P on the basis of the pixel values in the vicinity of each of the first feature points FP1. The optical flow OF1 of the first feature point FP1 is derived based on the position. Similarly, the identifying unit 122 associates the second feature point FP2 of the current frame P ′ with the second feature point FP2 of the previous frame P based on the pixel values in the vicinity thereof, and the associated second feature point FP2. The optical flow OF2 of the second feature point FP2 is derived based on the respective positions of.

  When the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2 are derived, the identifying unit 122 uses the internal parameter for the camera 21 stored in the storage unit 13A for the feature point. Coordinate conversion. In the coordinate conversion, aberration correction and distortion correction of the camera 21 are performed. Aberration correction is performed to correct distortion due to aberration of the optical system of the camera 21. Specifically, it is correction of distortion aberration such as barrel distortion and pincushion distortion. The distortion correction is performed to correct the distortion of the optical system itself of the camera 21. Specifically, it is fisheye correction. By the coordinate conversion, the coordinates of the feature points are converted into the coordinates obtained on the two-dimensional image when the subject is photographed by perspective projection.

  When the coordinate conversion of the feature points is performed, the specifying unit 122 sets the start point of the optical flow OF1 of the first feature point FP1 to the vertex SP1 and the end point of the optical flow OF1 of the first feature point FP1 to the vertex as illustrated in FIG. A quadrangle QL is virtually formed in which EP1 is the start point of the optical flow OF2 of the second feature point FP2 as the vertex SP2, and the end point of the optical flow OF2 of the second feature point FP2 is the vertex EP (step S105, see FIG. 15). . Hereinafter, for the sake of explanation, a quadrangle and geometric elements such as sides of the quadrangle are virtually formed and used. However, in the actual processing, the processing may not be based on the geometrical elements having the same action, based on the vector calculation of the coordinates of the characteristic points and the direction of the straight line.

  When the quadrangle QL is virtually formed, as shown in FIG. 20, the identifying unit 122 uses the quadrangle QL and the internal parameters of the camera 21 stored in the storage unit 13A to determine the camera in the three-dimensional space. The quadrangle QL is moved onto the projection plane IMG 21 to virtually generate the quadrangle QL1 on the projection plane IMG (step S106).

  For the sake of explanation, edges are defined as follows. The first side SD1 of the quadrangle QL1 corresponds to the side connecting the vertices SP1 and SP2 in the quadrangle QL (see FIG. 19). That is, it corresponds to the side connecting the first feature point FP1 and the second feature point FP2 in the previous frame P. Similarly, the second side SD2 of the quadrangle QL1 corresponds to the side connecting the vertices SP2 and EP2 in the quadrangle QL. That is, it corresponds to the optical flow OF2 of the second feature point FP2. Similarly, the third side SD3 of the quadrangle QL1 corresponds to the side connecting the vertices EP1 and EP2 in the quadrangle QL. That is, it corresponds to the side connecting the first feature point FP1 and the second feature point FP2 in the current frame P '. Similarly, the fourth side SD4 of the quadrangle QL1 corresponds to the side connecting the vertices SP1 and EP1 in the quadrangle QL. That is, it corresponds to the optical flow OF1 of the first feature point FP1.

  Further, the surface is defined as follows (see FIG. 20). A surface including the first side SD1 of the quadrangle QL1 and the optical center OC of the camera 21 is referred to as a first surface F1. Similarly, a surface including the second side SD2 of the quadrangle QL1 and the optical center OC of the camera 21 is referred to as a second surface F2. Similarly, a surface including the third side SD3 of the quadrangle QL1 and the optical center OC of the camera 21 is referred to as a third surface F3. Similarly, a surface including the fourth side SD4 of the quadrangle QL1 and the optical center OC of the camera 21 is referred to as a fourth surface F4.

  Next, the specifying unit 122 specifies two sets of surfaces whose intersecting lines with a predetermined plane are parallel to each other (step S107). The predetermined plane is a plane whose normal is known in advance. Specifically, it is the plane on which the vehicle is moving, that is, the road surface. The predetermined plane need not be a strict plane, and can be regarded as a plane when the specifying unit 122 specifies two sets of planes whose intersecting lines with the predetermined plane are parallel to each other. Good.

  In the present embodiment, the abnormality detection device 1A extracts feature points from two images captured at different times and calculates the optical flow of the feature points when the vehicle is traveling straight. In addition, the feature points are extracted from a stationary object located on a predetermined plane such as a road surface. Therefore, the calculated optical flow represents the relative position change of the stationary object with respect to the vehicle in the real world. In other words, it is the movement vector of the vehicle whose direction is reversed.

  The second side SD2 and the fourth side SD4 of the quadrangle QL1 both correspond to the optical flow of the feature point, and thus both correspond to the movement vector of the vehicle in the real world. Therefore, it is assumed that they are parallel to each other on the road surface.

  Further, since the first side SD1 and the third side SD3 of the quadrangle QL1 are both in the positional relationship between the feature points, they correspond to the positional relationship between the stationary objects associated with the movement of the vehicle in the real world. The positional relationship before the movement corresponds to the first side SD1, and the positional relationship after the movement corresponds to the third side SD3. At this time, since the position of the stationary object does not change, the positional relationship does not change before and after the movement. Therefore, it is assumed that these are also parallel to each other on the road surface.

  Therefore, the specifying unit 122 sets two sets of a set of the second face F2 and the fourth face F4 and a set of the first face F1 and the third face F3 as the faces whose intersection lines with the road surface are parallel to each other. Specify. In other words, the specifying unit 122 specifies a total of two sets, with the faces including the optical flow as one set and the faces including the feature points captured at the same time as another set.

  In FIG. 20, a quadrangle QL2 indicates the position of the first feature point FP1 in the three-dimensional space (real world) at the time of shooting the current frame P ′, and the second feature point FP2 at the time of shooting the current frame P ′. The position in the three-dimensional space, the position of the first feature point FP1 in the three-dimensional space at the time of capturing the previous frame P, and the position of the second feature point FP2 in the three-dimensional space at the time of capturing the previous frame P are It is a quadrangle with the apex. The first surface F1 includes the first side SD11 of the quadrangle QL2. Similarly, the second surface F2 includes the second side SD12 of the quadrangle QL2, the third surface F3 includes the third side SD13 of the quadrangle QL2, and the fourth surface F4 includes the fourth side SD14 of the quadrangle QL2. At this time, the quadrangle QL2 is assumed to be a parallelogram formed on the road surface as described above.

  Next, the estimation unit 123 calculates the normal line of the road surface (step S108). First, the estimation unit 123 obtains the direction of the line of intersection between the faces based on the first face F1 and the third face F3, which are one of the sets of faces identified by the identifying unit 122. Specifically, the direction of the line of intersection CL1 between the first surface F1 and the third surface F3 is obtained (see FIG. 21). The direction vector V1 of the intersection line CL1 is a vector perpendicular to the normal vector of the first surface F1 and the normal vector of the third surface F3. Therefore, the estimation unit 123 obtains the direction vector V1 of the intersection line CL1 by the cross product of the normal vector of the first surface F1 and the normal vector of the third surface F3. The first surface F1 and the third surface F3 have parallel lines of intersection with the road surface, so that the direction vector V1 is parallel to the road surface.

  Similarly, the estimation unit 123 obtains the direction of the line of intersection between the second surface F2 and the fourth surface F4 that is the other of the pair of surfaces identified by the identifying unit 122. Specifically, the direction of the line of intersection CL2 between the second surface F2 and the fourth surface F4 is obtained (see FIG. 22). The direction vector V2 of the intersection line CL2 is a vector perpendicular to the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Therefore, the estimation unit 123 obtains the direction vector V2 of the intersection line CL2 by the outer product of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Similarly, in the second surface F2 and the fourth surface F4, the lines of intersection with the road surface are also parallel, so the direction vector V2 is parallel to the road surface.

  The estimation unit 123 calculates the normal of the surface of the quadrangle QL2, that is, the normal of the road surface by the cross product of the direction vector V1 and the direction vector V2. Since the normal line of the road surface calculated by the estimating unit 123 is calculated in the camera coordinate system of the camera 21, the coordinate system of the three-dimensional space is obtained from the difference from the vertical direction which is the normal line of the actual road surface, and the camera 21 with respect to the road surface is obtained. Can be estimated. The estimation unit 123 estimates the attitude of the camera 21 with respect to the vehicle from the estimation result (step S109). The calculation process of step S108 can be executed by using, for example, a known ARToolkit.

  After estimating the posture of the camera 21, the determination unit 124 determines whether or not there is a camera shift based on the posture of the camera 21 estimated by the estimation unit 123 (step S110). The attitude of the camera 21 at the time of attachment is stored in advance in the storage unit 13A, and the presence or absence of the camera shift can be determined by comparison with the data stored in the storage unit 13A. When the determination as to whether or not there is a camera shift in step S110 is completed, the flow shown in FIG. 15 ends.

  As can be seen from the above, according to the abnormality detection device 1A of the present embodiment, the determination process of determining whether or not there is a camera shift (abnormality of the camera 21) depending on whether the vehicle (moving body) is fast or slow. Change the processing mode. As a result, it is possible to perform appropriate determination processing suitable for the speed of the vehicle. For example, when traveling at a low speed such as when leaving the warehouse or traveling on a narrow road, the second processing mode is used. In such a scene, there is a high possibility that it will be difficult to find a feature point with a high degree of corners, and it can be said that the determination process using the second processing mode that lowers the feature point extraction threshold is appropriate. On the other hand, in a scene in which the vehicle travels at a certain speed, white lines are often displayed on the road, and the determination processing using the first processing mode that can easily track the feature points and improve the determination accuracy is appropriate. I can say. According to the present embodiment, it is possible to increase the chances of determining the presence or absence of abnormality of the camera 21, while improving the determination accuracy as much as possible.

  Further, according to the posture estimation device 14 of the present embodiment, the processing mode of the estimation process for estimating the posture of the camera 21 is changed depending on whether the speed of the vehicle (moving body) is fast or slow. This makes it possible to perform an appropriate estimation process that matches the speed of the vehicle. For example, in a low-speed scene such as when leaving a warehouse or traveling on a narrow road, there is a high possibility that it is difficult to find a feature point with a high degree of cornering, and the estimation process using the second processing mode that lowers the feature point extraction threshold is Can be said to be appropriate. On the other hand, in a scene in which the vehicle travels at a certain speed, white lines are often displayed on the road, and the estimation processing using the first processing mode that can easily track the feature points and improve the estimation accuracy is appropriate. I can say. According to the present embodiment, it is possible to increase the chances of posture estimation of the camera 21 while improving the accuracy of posture estimation as much as possible.

  Further, in the present embodiment, the posture of the camera 21 is estimated from the temporal change in the position of the feature point in both the determination processing in the first processing mode and the determination processing in the second processing mode. Then, in any of the determination processes, the presence or absence of abnormality of the camera 21 is determined based on the estimated posture of the camera 21. That is, according to this configuration, the estimated value of the posture of the camera 21 can be obtained when determining whether or not there is an abnormality in the camera 21 in both the first processing mode and the second processing mode.

<3. Modification>
In the above description, the plane including the optical center OC of the camera 21 and one side of the quadrangle QL1 is specified, but the present invention is not limited to this. The plane may be specified by specifying the normal direction of the plane. For example, the normal direction of the plane may be obtained by the outer product of the direction vectors from the optical center OC to each vertex, and the surface may be specified by the normal direction. In this case, the normal direction of the first surface F1 and the normal direction of the third surface F3 are set as one set, and the normal direction of the second surface F2 and the normal direction of the fourth surface F4 are set as another set. Two sets should be identified.

  Also, the planes may be translated. For example, instead of the first surface F1, a surface obtained by moving the first surface F1 in parallel may be combined with the third surface F3. This is because the direction of the line of intersection with the predetermined plane does not change even if it moves in parallel.

  Further, in both the determination processing in the first processing mode and the determination processing in the second processing mode, the estimated value of the movement information of the vehicle is obtained from the temporal change of the position of the feature point, and based on the estimated value of the movement information. The presence / absence of abnormality of the camera 21 may be determined.

<< C. Third Embodiment >>
<1. Anomaly detection system>
Next, a third embodiment will be described. In the description of the third embodiment, the contents overlapping with those of the first embodiment and the second embodiment will be omitted unless particularly necessary. Further, the same reference numerals are given to the portions that overlap with the first embodiment and the second embodiment.

  FIG. 23 is a block diagram showing the configuration of the abnormality detection system SYS2B according to the third embodiment of the present invention. The abnormality detection system SYS2B of the third embodiment includes an abnormality detection device 1B, an imaging unit 2, and a sensor unit 3 as in the abnormality detection system SYS2 of the first embodiment and the abnormality detection system SYS2A of the second embodiment. Prepare The abnormality detection system SYS2B is also a system that detects a camera shift, as in the first and second embodiments. Since the configurations of the image capturing unit 2 and the sensor unit 3 are the same as those in the first and second embodiments, the description thereof will be omitted.

<2. Abnormality detection device>
The abnormality detection device 1B is provided for each vehicle equipped with a camera, as in the first and second embodiments. The abnormality detection device 1B is a device (camera deviation detection device) that detects an abnormality based on a captured image captured by a camera.

  As shown in FIG. 23, the abnormality detection device 1B includes an image acquisition unit 111, a control unit 12B, and a storage unit 13B. The configuration of the image acquisition unit 111 is the same as that of the first and second embodiments, and thus the description thereof is omitted. The control unit 12B and the storage unit 13B have different parts from those of the first and second embodiments. The storage unit 13B stores a program for realizing the function of the control unit 12B, which is different from those in the first and second embodiments.

  In the third embodiment as well, similar to the first and second embodiments, the control unit 12B determines the presence / absence of abnormality of the camera 21 based on the position of the feature point extracted from the captured image captured by the camera 21. A determination process for determining based on the change over time is executed. The determination process is provided so that the first processing mode and the second processing mode can be selected. However, for example, the method of determination processing is different from that of the first and second embodiments. Further, in the third embodiment, it is possible to detect whether or not there is an abnormality in the camera 21 not only when the vehicle is moving but also when the vehicle is stopped. Also in this respect, the third embodiment is different from the first and second embodiments. Also in the present embodiment, the number of cameras 21 is plural, and the determination process is performed for each camera 21.

  FIG. 24 is a flowchart showing an example of a camera shift detection flow in the abnormality detection device 1B according to the third embodiment of the present invention. The camera shift detection process is repeatedly performed at a predetermined timing after the ignition is turned on (power on in the case of an electric vehicle). Hereinafter, the detection process of the camera shift will be described taking the case where the camera 21 is the front camera as an example.

  As shown in FIG. 24, first, the control unit 12B confirms whether or not the vehicle equipped with the camera 21 is stopped (step S201). Whether or not the vehicle is stopped can be confirmed by, for example, speed information obtained from the speed sensor 31.

  When the vehicle is stopped (Yes in step S201), the control unit 12B executes the stop time determination process (step S202). The stoppage determination process determines whether or not there is an abnormality in the camera 21 (whether or not there is a camera shift) by a process different from the determination process that determines whether or not there is an abnormality in the camera 21 based on the change in the position of the feature point with time.

  According to this, since it is possible to determine whether or not there is an abnormality in the camera 21 not only when the vehicle is moving but also when the vehicle is stopped, it is possible to increase the chances of the determination. That is, the abnormality of the camera 21 can be noticed quickly. The camera shift may occur when an object hits the camera 21 while the vehicle is parked or the like. Therefore, it is effective to determine the camera shift even when the vehicle is stopped.

  In the present embodiment, in the stop-time determination process, it is determined whether or not the camera 21 is abnormal based on the edge image of the body of the vehicle (moving body) obtained from the captured image captured by the camera 21. For example, as shown in FIG. 4 and FIG. 17 described above, the body of the vehicle is reflected in the captured image of the camera 21. When the camera shift occurs, the appearance of the body changes. Therefore, the camera shift can be detected using the edge image of the body.

  FIG. 25 is a flowchart showing an example of a camera shift detection flow using the vehicle stop determination process. FIG. 25 is a flowchart showing a detailed example of step S202 of FIG.

  The control unit 12B performs preprocessing (blurring) on the captured image acquired from the camera 21 while the vehicle is stopped (step S2021). The pre-process is a process for smoothing a captured image, and is performed for the purpose of reducing noise. The preprocessing may be, for example, Gaussian filter processing or median filter processing. Further, the preprocessing may be performed by changing the processing condition for each part of the captured image. For example, when the center side of the captured image is sharp and the edge side is blurred, the processing condition is different for each part of the captured image so that the blurring on the middle side of the image is emphasized from the edge side. May be changed.

  The control unit 12B performs edge detection on the preprocessed captured image (step S2022). The edge is a portion in which the pixel value of the pixel in the image is abruptly changed, and is a line representing the contour of the object in the image. The control unit 12B detects the edge in the captured image by differentiating the pixel value of the pixel in the captured image. When the pixel value of the pixel in the captured image is differentiated, the differential value of the portion in which the pixel value of the pixel in the captured image is rapidly changed becomes large. The control unit 12B detects a portion having a large differential value as an edge based on a predetermined reference. The Canny method or the like may be used as the edge detection method.

  The control unit 12B performs mask processing on the edge image obtained by performing edge detection (step S2023). The control unit 12B performs the masking process on the image excluding the vehicle edge search region that is preset according to the body shape of each vehicle type. As a result, unnecessary edges such as scenery can be appropriately removed, and the possibility of falsely detecting the edge of the body can be reduced. In addition, this makes it possible to quickly acquire an image for determining the camera shift. The mask processing may not be performed.

  The control unit 12B performs processing for removing unnecessary edges on the edge image obtained after the mask processing (step S2024). For example, the control unit 12B performs a process of removing, from the edge image obtained after the masking process, an edge determined to be different from the edge assumed from the design value of the vehicle body.

  The control unit 12B performs pattern matching processing between the acquired edge image of the body and the reference image (step S2025). The reference image is an image stored in the storage unit 13B in advance, and is an image for comparison obtained from the captured image in the state where the camera 21 is properly attached to the vehicle.

  When the pattern matching process ends, the control unit 12B determines the camera shift based on the result of the pattern matching process (step S2026). The control unit 12B calculates, for example, the matching rate of the acquired edge image with respect to the reference image, and determines whether or not there is a camera shift based on the matching rate.

  Returning to FIG. 24, when the vehicle is not stopped (No in step S201), the control unit 12B confirms whether the vehicle is traveling straight (step S203). The method of checking whether the vehicle is going straight may be the same as that of the first embodiment. When it is determined that the vehicle is not traveling straight (No in step S203), the process returns to step S201.

  When it is determined that the vehicle is traveling straight (Yes in step S203), the control unit 12B confirms whether or not the speed of the vehicle is within the first speed range (step S204). The speed of the vehicle is obtained from the speed sensor 31, for example. The first speed range is, for example, not less than 5 km / h and less than 30 km / h.

  When the speed of the vehicle is within the first speed range (Yes in step S204), the control unit 12B executes the determination process in the first processing mode (step S205). The determination process in the first processing mode is a process of determining the presence / absence of a camera shift based on the result of the posture estimation process that estimates the posture of the camera 21 from the captured image of the camera 21. Since the determination process is the same as the process described in the second embodiment, detailed description will be omitted. Since it is in the first processing mode, the feature points used for the posture estimation of the camera 21 have a high degree of corners, and the feature point extraction error can be reduced. That is, the accuracy of the posture estimation process of the camera 21 can be improved, and the accuracy of the camera shift determination can be improved.

  When the vehicle speed is outside the first speed range (No in step S204), the control unit 12B confirms whether the vehicle speed is within the second speed range (step S206). The second speed range is, for example, 3 km or more per hour and less than 5 km per hour. When the vehicle speed is outside the second speed range (No in step S206), the process returns to step S201 without determining the camera shift.

  When the speed of the vehicle is within the second speed range (Yes in step S206), the control unit 12B executes the determination process in the second processing mode (step S207). The determination process in the second processing mode is a process of determining the presence or absence of a camera shift based on the result of movement information estimation that estimates movement information (for example, the amount of movement) of the vehicle from the image captured by the camera 21. The determination process is the same as the process described in the first embodiment (see FIG. 9), and thus detailed description will be omitted. Since the second processing mode is used, even if a feature point having a high degree of corner, such as a white line corner, cannot be extracted, the feature point can be extracted to determine whether or not there is a camera shift. For example, the presence / absence of a camera shift can be determined even when the vehicle is leaving the vehicle or running on a narrow road, and the opportunity for determining the presence / absence of a camera shift can be increased to quickly detect the camera shift.

  According to the third embodiment, since it is possible to perform an appropriate camera shift determination process according to the speed state of the vehicle, it is possible to accurately determine whether or not there is a camera shift, and further The detection can be done quickly.

  Further, in the present embodiment, in the determination processing in the first processing mode, the orientation of the camera 21 is estimated from the time change of the position of the feature point, and it is determined whether there is an abnormality in the camera 21 based on the estimated orientation of the camera 21. To be done. In the determination processing in the second processing mode, the estimated value of the movement information (for example, the movement amount) of the vehicle (moving body) is obtained from the temporal change of the position of the feature point, and the abnormality of the camera 21 is determined based on the estimated value of the movement information. Presence / absence is determined. The configuration of the determination process for determining whether or not there is an abnormality in the camera 21 is properly used according to the speed of the vehicle, and it is possible to appropriately determine whether or not there is an abnormality in the camera 21 according to the speed.

  In some cases, in the determination process in the first processing mode, the estimated value of the vehicle movement information is obtained from the temporal change in the position of the feature point, and the presence or absence of abnormality of the camera 21 is determined based on the estimated value of the movement information. You may In the determination process in the second processing mode, the posture of the camera 21 may be estimated from the temporal change in the position of the feature point, and the presence or absence of abnormality of the camera 21 may be determined based on the estimated posture of the camera 21.

<< D. Notes >>
The configurations of the embodiments and modified examples in the present specification are merely examples of the present invention. The configurations of the embodiments and the modifications may be appropriately modified without departing from the technical idea of the present invention. Further, the plurality of embodiments and modified examples may be implemented in combination within a possible range.

  Further, although it has been described above that various functions are realized by software by the arithmetic processing of the CPU according to the computer program, at least a part of these functions is realized by an electrical hardware circuit. May be. On the contrary, at least a part of the functions realized by the hardware circuit may be realized by software.

  INDUSTRIAL APPLICABILITY The present invention can be used to detect an abnormality in a camera that assists driving of a moving body such as automatic parking and to estimate a posture. Further, the present invention can be used to detect an abnormality in a camera that records driving information such as a drive recorder and estimate the posture. Further, the present invention can be used in a correction device or the like that corrects a captured image by using abnormality information of a camera and estimated posture information. Further, the present invention can be used for an apparatus or the like that operates in cooperation with a center that is communicable with a plurality of mobile units via a network. For example, when transmitting the captured image to the center, the apparatus may be configured to transmit camera abnormality information and posture estimation information together with the captured image. Then, the center uses various kinds of image processing (processing for changing the viewpoint / viewing direction of the image in consideration of the camera attitude, for example, converting the viewpoint / viewing direction into an image in the front direction of the vehicle body of the vehicle, using the estimated information of the camera attitude. Generated image, etc.), correction processing for the camera posture in measurement processing using the image, statistical processing of secular change of camera posture (data of many vehicles), etc. to provide useful data to the user. Generate, etc.

1, 1A, 1B ... Abnormality detection device 5 ... Automatic operation control device 6 ... Display device 12, 12A, 12B ... Control unit 21 ... Camera FP ... Characteristic point SYS1 ... Mobile control system T1 ... first threshold T2 ... second threshold

Claims (15)

An abnormality detection device for detecting an abnormality of a camera mounted on a moving body,
A control unit that executes a determination process of determining whether or not there is an abnormality in the camera based on a temporal change in the position of a feature point extracted from a captured image captured by the camera,
The determination process is
A first processing mode in which a threshold for extracting the feature points is a first threshold;
A second processing mode in which the threshold is a second threshold smaller than the first threshold;
Is provided so that
The first processing mode is an abnormality detection device used when the speed of the moving body is higher than that in the second processing mode. The first processing mode is a processing mode that is normally used,
The abnormality detection device according to claim 1, wherein the second processing mode is a processing mode used when a predetermined processing result is obtained in the first processing mode.   The abnormality detection device according to claim 2, wherein the predetermined processing result includes a processing result indicating that the camera has an abnormality.   The abnormality detection device according to claim 2, wherein the predetermined processing result includes a processing result indicating that the presence or absence of abnormality of the camera cannot be determined for a predetermined period. The control unit is provided to be able to perform a recognition process for recognizing the surrounding environment of the moving body based on a captured image captured by the camera,
In a normal time, the control unit prioritizes the recognition process over the determination process when the speed of the moving body is a speed corresponding to the second processing mode,
The abnormality detection device according to claim 2, wherein the control unit prioritizes the determination process over the recognition process when execution of the second processing mode is determined.   When switching from the first processing mode to the second processing mode, the control unit causes a display device mounted on the moving body to display a message urging to perform an operation suitable for the second processing mode. The abnormality detection device according to any one of claims 1 to 5, which performs processing.   7. The control unit requests automatic operation to an automatic operation control device that controls automatic operation of the moving body when switching from the first processing mode to the second processing mode. The abnormality detection device according to any one of items.   In both the determination processing in the first processing mode and the determination processing in the second processing mode, the estimated value of the movement information of the moving body is obtained from the temporal change in the position of the characteristic point, and the movement is performed. The abnormality detection device according to claim 1, wherein presence or absence of abnormality of the camera is determined based on an estimated value of information.   In both the determination processing in the first processing mode and the determination processing in the second processing mode, the orientation of the camera is estimated from the temporal change in the position of the feature point, and the estimated orientation of the camera The abnormality detection device according to claim 1, wherein the presence or absence of abnormality of the camera is determined based on. In the determination processing in the first processing mode, the orientation of the camera is estimated from the temporal change in the position of the feature point, and the presence or absence of abnormality of the camera is determined based on the estimated orientation of the camera,
In the determination processing in the second processing mode, the estimated value of the movement information of the moving body is obtained from the time change of the position of the characteristic point, and it is determined whether or not there is an abnormality in the camera based on the estimated value of the movement information. The abnormality detection device according to claim 1, which is performed.   11. The control unit according to claim 1 or 10, wherein when the mobile body is stopped, a vehicle stop determination process for determining whether or not there is an abnormality in the camera is performed by a process different from the determination process. Anomaly detection device.   The abnormality detection device according to claim 11, wherein in the determination processing for when the vehicle is stopped, presence or absence of abnormality of the camera is determined based on an edge image of a body of the moving body obtained from a captured image captured by the camera. . A control unit that executes an estimation process for estimating the posture of the camera based on the temporal change in the position of the feature point extracted from the captured image captured by the camera mounted on the moving body,
The estimation process is
A first processing mode in which a threshold for extracting the feature points is a first threshold;
A second processing mode in which the threshold is a second threshold smaller than the first threshold;
Is provided so that
The posture estimation device used in the first processing mode when the speed of the moving body is higher than that in the second processing mode. An abnormality detection device according to any one of claims 1 to 7,
An automatic driving control device for controlling the automatic driving of the moving body,
Equipped with
A mobile control system that automatically drives the mobile when the determination process is performed at least in the second processing mode. An abnormality detection method for detecting an abnormality of a camera mounted on a moving body by a device,
A process step of determining whether or not there is an abnormality in the camera based on a temporal change in the position of a feature point extracted from a captured image captured by the camera,
The processing step is
A first processing mode in which a threshold for extracting the feature points is a first threshold;
A second processing mode in which the threshold is a second threshold smaller than the first threshold;
Is provided so that
The first processing mode is an abnormality detection method used when the speed of the moving body is higher than that of the second processing mode.

Images