JP2019121032A - Camera deviation detection device, and camera deviation detection method - Google Patents

Abstract

To provide a technique that can properly detect irregularity such as a deviation of an attachment position or angle of an on-vehicle camera.SOLUTION: A camera deviation detection device, which detects an attachment deviation of a camera to be boarded in a moving body includes:, a derivation unit that derives an optical flow serving a movement of a characteristic point between two images input by the camera at a prescribed time interval for each characteristic point; an amount-of-movement estimation unit that estimates an amount of movement of the moving body boarding the camera on the basis of a plurality of optical flows; and a determination unit that determines a deviation of the camera on the basis of estimation information obtained by the amount-of-movement estimation unit and actual measurement information relating to a movement of the moving body to be input from an external sensor other than the camera. The determination unit is configured to, when an irregularity state where it is estimated by the estimation information that the moving body stands still although it is recognized based on the external sensor that the moving body is moving occurs, halt determination processing of the deviation of the camera.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a camera shift detection device and a camera shift detection method.

  2. Description of the Related Art Conventionally, driving assistance such as parking assistance for a vehicle has been performed using a camera mounted on a movable body such as a vehicle (hereinafter referred to as an on-vehicle camera). The on-vehicle camera is fixedly attached to the vehicle, for example, before the vehicle is shipped from the factory. However, the on-vehicle camera may deviate from the factory-installed state due to, for example, unexpected contact or aging. If the mounting position or angle of the in-vehicle camera deviates, an error occurs in the steering amount of the steering wheel determined using the camera image, so it is important to detect the mounting deviation of the in-vehicle camera.

  Patent Document 1 discloses a technique for detecting an optical axis deviation between a plurality of imaging means. Specifically, the parallax offset caused by the relative axis deviation of the optical axes of the two cameras is a change in the distance to the appropriate stationary object detected by each camera in the straight ahead state of the host vehicle in a predetermined time period; It is disclosed that the value corresponds to the deviation from the movement amount of the host vehicle detected by the navigation device or the like at this predetermined time.

JP, 2006-153778, A

  The optical axis offset detection means in Patent Document 1 can detect an optical axis offset between two cameras whose optical axes are arranged in parallel to each other. However, the optical axis misalignment detecting means can not detect the deviation from the factory-installed state of a single on-vehicle camera.

  An object of the present invention is to provide a technology capable of appropriately detecting a mounting position or an angle deviation of a vehicle-mounted camera.

  In order to achieve the above object, according to the present invention, a camera shift detection device for detecting a shift in mounting of a camera mounted on a mobile object is a feature point between two images input with a predetermined time by the camera. Obtained by a derivation unit that derives an optical flow that is a motion for each feature point, a movement amount estimation unit that estimates the movement amount of a moving object equipped with the camera based on a plurality of optical flows, and the movement amount estimation unit A determination unit that determines the displacement of the camera based on the estimated information obtained and the measurement information related to the movement of the moving object input from an external sensor other than the camera, the determination unit including the external In the case where an abnormal state occurs in which it is estimated by the estimated information that the moving body is stopped although it is recognized that the moving body is moving based on a sensor. The shift determination process of the serial cameras have been configured to stop (first configuration).

  In the camera shift detection device of the first configuration, in the abnormal state, although it is recognized that the movable body is moving based on the external sensor, the plurality of the derived units are derived by the deriving unit. The configuration (second configuration) may include the case where there is a predetermined ratio or more of the optical flow determined to have no motion in the optical flow.

  In the camera shift detection device having the first or second configuration, the determination unit may stop the shift determination process until the traveling direction of the movable body changes when the abnormal state is detected at least once. It is preferable that the configuration is a continuous configuration (third configuration).

  In the camera shift detecting device of the third configuration, when the traveling direction of the movable body changes, the movement amount by the movement amount estimation unit based on the plurality of optical flows newly acquired after the change of the traveling direction. It is preferable that it is the structure (4th structure) in which estimation and the said deviation determination processing by the said determination part are performed.

  In the camera shift detection device having any one of the first to fourth configurations, when the position of the shadow of the moving body can be recognized when the abnormal state occurs, the shift determination process is not stopped, and A configuration in which movement amount estimation by the movement amount estimation unit and the deviation determination processing by the determination unit are performed based on a plurality of optical flows derived using the feature points positioned in a range deviated from the shadow ( Fifth configuration).

  In the camera shift detecting device according to any one of the first to fifth configurations, when there is a plurality of cameras and the abnormal state occurs in some of the plurality of cameras, the abnormal state is generated. It is preferable that it is the structure (6th structure) by which the said shift | offset | difference determination process is performed about the said remaining camera which has not produced.

  The camera shift detecting device according to any one of the first to sixth configurations further includes a shadow position estimating unit configured to estimate a shadow position of the moving body, and the deriving unit is configured to determine the shadow position based on the estimated shadow position. It is preferable that the configuration (the seventh configuration) is to set the extraction range of the feature point.

  In order to achieve the above object, according to a camera shift detection method for detecting a mount shift of a camera mounted on a moving body of the present invention, movement of a feature point between two images inputted by a predetermined time by the camera Optical flow for each feature point, an estimation step of estimating the amount of movement of a moving object equipped with the camera based on a plurality of optical flows, and estimation information obtained in the estimation step And a determination step of determining the shift of the camera based on the measurement information related to the movement of the mobile input from a sensor other than the camera, and the mobile is moving based on the sensor The camera in the determination step if an abnormal state occurs in which it is estimated that the moving object is stopped in the estimation step despite the fact that Deviation determination process has a configuration which is stopped (eighth configuration).

  According to the present invention, it is possible to properly detect the mounting position and the angle deviation of the on-vehicle camera.

Block diagram showing the configuration of the camera shift detection system A diagram exemplifying a position where an on-vehicle camera is disposed on a vehicle Flow chart showing an example of camera shift detection flow by the camera shift detection device Diagram for explaining the method of extracting feature points Diagram for explaining the method of deriving optical flow Diagram for explaining the coordinate conversion process A diagram showing an example of a first histogram generated by the movement amount estimation unit A diagram showing an example of a second histogram generated by the movement amount estimation unit Diagram showing histogram change when camera shift occurs Flow chart showing an example of camera shift determination processing performed by the determination unit Schematic diagram of the image taken by the on-board camera A flowchart showing a first embodiment different from the camera displacement detection flow shown in FIG. 3 A flowchart showing a second embodiment different from the camera shift detection flow shown in FIG. 3 A schematic diagram for explaining the process of changing the feature point extraction range Figure for explaining an example of re-determination processing of camera shift A flowchart showing a modified example of the camera misalignment detection flow by the camera misalignment detection device Block diagram showing the configuration of a camera shift detection device according to a modification Flow chart showing an example of camera shift detection flow by the camera shift detection device of the modified example

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following description, a vehicle will be described as an example of a moving body, but it is not limited to a vehicle as long as it is a moving body. Further, in the following description, the direction in which the vehicle travels in a straight line, and the direction from the driver's seat to the steering wheel is referred to as "forward direction". In addition, a direction in which the vehicle travels straight ahead and which is directed from the steering wheel to the driver's seat is referred to as “rearward direction”. In addition, a direction perpendicular to the straight traveling direction of the vehicle and the vertical line, that is, a direction from the right side to the left side of the driver facing the front direction is referred to as “left direction”. Further, a direction perpendicular to the straight traveling direction of the vehicle and the vertical line, and a direction from the left side to the right side of the driver facing the front direction is referred to as “right direction”.

<1. Camera offset detection system>
FIG. 1 is a block diagram showing the configuration of a camera shift detection system SYS according to an embodiment of the present invention. The camera shift detection system SYS is a system that detects a shift of the on-vehicle camera from a reference installation state, such as the installation state of the vehicle at the time of factory shipment. As shown in FIG. 1, the camera shift detection system SYS includes a camera shift detection device 1, a photographing unit 2, an input unit 3, and a sensor unit 4.

  The camera shift detection device 1 is a device that detects a shift in mounting of an on-vehicle camera. Mounting deviations include mounting position deviations and mounting angle deviations. The camera shift detection device 1 is provided in each vehicle equipped with an on-vehicle camera. The camera shift detection device 1 processes the captured images captured by the on-vehicle cameras 21 to 24 included in the imaging unit 2 and information from the sensor unit 4 provided outside the device 1 to obtain the on-vehicle cameras 21 to 21. Detects the mounting position and mounting angle of 24. Details of the camera shift detection device 1 will be described later.

  The camera offset detection device 1 may output processing information to a display device or a driving support device (not shown). The display device appropriately displays a warning or the like on the screen based on the information output from the camera shift detection device 1. The driving support device appropriately stops the driving support function based on the information output from the camera shift detection device 1 and corrects the imaging information to perform the driving support. The driving support device may be, for example, a device supporting automatic driving, a device supporting automatic parking, a device supporting emergency braking, and the like.

  The imaging unit 2 is provided in the vehicle for the purpose of monitoring the situation around the vehicle. In the present embodiment, the imaging unit 2 includes four on-vehicle cameras 21 to 24. Each of the on-vehicle cameras 21 to 24 is connected to the camera shift detection device 1 by wire or wirelessly. FIG. 2 is a view exemplifying the positions at which the on-vehicle cameras 21 to 24 are disposed in the vehicle 7. FIG. 2 is a top view of the vehicle 7.

  The on-vehicle camera 21 is provided at the front end of the vehicle 7. For this reason, the on-vehicle camera 21 is also referred to as a front camera 21. The optical axis 21 a of the front camera 21 is along the front-rear direction of the vehicle 7. The front camera 21 shoots the front of the vehicle 7. The on-vehicle camera 22 is provided at the rear end of the vehicle 7. For this reason, the on-vehicle camera 22 is also referred to as a back camera 22. The optical axis 22 a of the back camera 22 is along the front-rear direction of the vehicle 7. The back camera 22 shoots the rear of the vehicle 7. The mounting position of the front camera 21 and the back camera 22 is preferably at the left and right center of the vehicle 7, but may be slightly offset from the left and right center in the left and right direction.

  The on-vehicle camera 23 is provided on the left side door mirror 71 of the vehicle 7. For this reason, the on-vehicle camera 23 is also referred to as the left side camera 23. The optical axis 23 a of the left side camera 23 is along the left-right direction of the vehicle 7. The left side camera 23 captures the left direction of the vehicle 7. The on-vehicle camera 24 is provided on the right side door mirror 72 of the vehicle 7. For this reason, the on-vehicle camera 24 is also referred to as the right side camera 24. The optical axis 24 a of the right side camera 24 is in the left-right direction of the vehicle 7. The right side camera 24 shoots the right of the vehicle 7.

  Each of the on-vehicle cameras 21 to 24 is configured by a fisheye lens, and the horizontal angle of view θ is 180 ° or more. Therefore, the on-vehicle cameras 21 to 24 can capture the entire circumference of the vehicle 7 in the horizontal direction. In the present embodiment, the number of in-vehicle cameras is four, but the number may be changed as appropriate, and may be plural or singular. For example, in the case where a vehicle-mounted camera is mounted for the purpose of assisting the vehicle 7 to park in the back, the vehicle-mounted camera of the photographing unit 2 includes three of the back camera 22, the left side camera 23 and the right side camera 24. It may consist of one.

  Returning to FIG. 1, the input unit 3 can input an instruction to the camera shift detection device 1. The input unit 3 may be configured by, for example, a touch panel, a button, a lever, or the like. The input unit 3 is connected to the camera shift detection device 1 by wire or wirelessly.

  Sensor part 4 has a plurality of sensors which detect information about vehicles 7 in which in-vehicle cameras 21-24 are carried. In the present embodiment, the sensor unit 4 includes a vehicle speed sensor 41 and a steering angle sensor 42. The vehicle speed sensor 41 detects the speed of the vehicle 7. The steering angle sensor 42 detects the rotation angle of the steering wheel (handle) of the vehicle 7. The vehicle speed sensor 41 and the steering angle sensor 42 are connected to the camera shift detection device 1 via the communication bus 50. That is, the speed information of the vehicle 7 acquired by the vehicle speed sensor 41 is input to the camera shift detection device 1 via the communication bus 50. The rotation angle information of the steering wheel of the vehicle 7 acquired by the steering angle sensor 42 is input to the camera shift detection device 1 via the communication bus 50. The communication bus 50 may be, for example, a CAN (Controller Area Network) bus.

<2. Camera offset detection device>
As shown in FIG. 1, the camera shift detection device 1 includes an image acquisition unit 11, a control unit 12, and a storage unit 13.

  The image acquisition unit 11 acquires a photographed image from each of the four on-vehicle cameras 21 to 24. The image acquisition unit 11 has a basic image processing function such as an A / D conversion function that converts an analog captured image into a digital captured image. The image acquisition unit 11 performs predetermined image processing on the acquired captured image, and inputs the processed captured image to the control unit 12.

  The control unit 12 is, for example, a microcomputer, and controls the entire camera displacement detection device 1 in an integrated manner. The control unit 12 includes a CPU, a RAM, a ROM, and the like. The storage unit 13 is, for example, a non-volatile memory such as a flash memory, and stores various types of information. The storage unit 13 stores programs as firmware and various data.

  In detail, the control unit 12 includes a derivation unit 121, a movement amount estimation unit 122, and a determination unit 123. That is, the camera misalignment detection device 1 includes the derivation unit 121, the movement amount estimation unit 122, and the determination unit 123. The functions of the units 121 to 123 included in the control unit 12 are realized, for example, by the CPU performing arithmetic processing in accordance with a program stored in the storage unit 13.

  Note that at least one of the derivation unit 121, the movement amount estimation unit 122, and the determination unit 123 of the control unit 11 is configured by hardware such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA). It may be done. Further, the derivation unit 121, the movement amount estimation unit 122, and the determination unit 123 are conceptual components. The function performed by one component may be distributed to a plurality of components, or the function possessed by a plurality of components may be integrated into one component.

  The deriving unit 121 derives, for each feature point, an optical flow (motion vector) that is a motion of a feature point between two images input with a predetermined time by the on-vehicle cameras 21 to 24. In the present embodiment, the vehicle 7 has four on-vehicle cameras 21-24. For this purpose, the deriving unit 121 derives an optical flow for each of the feature points for each of the on-vehicle cameras 21 to 24. A feature point is a point that can be detected prominently in a captured image, such as the intersection of edges in the captured image. The characteristic points are, for example, an edge of a white line drawn on the road surface, a crack on the road surface, a stain on the road surface, a gravel on the road surface, and the like. Usually, a large number of feature points exist in one captured image. The deriving unit 121 derives feature points of the captured image using, for example, a known method such as a Harris operator.

  The movement amount estimation unit 122 estimates the movement amount of the vehicle 7 on which the in-vehicle cameras 21 to 24 are mounted based on the plurality of optical flows. In detail, the movement amount estimation unit 122 estimates the movement amount of the vehicle 7 using statistical processing. In the present embodiment, since the vehicle 7 includes the four on-vehicle cameras 21 to 24, the movement amount estimating unit 122 obtains an estimated value of the movement amount of the vehicle 7 for each of the on-vehicle cameras 21 to 24. In the present embodiment, the statistical processing performed by the movement amount estimation unit 122 is processing using a histogram. Details of the process of estimating the movement amount using a histogram will be described later.

  The determination unit 123 shifts the on-vehicle cameras 21 to 24 based on the estimated information obtained by the movement amount estimation unit 122 and the measurement information related to the movement of the vehicle 7 input from an external sensor other than the on-vehicle cameras 21 to 24. Determine In the present embodiment, since the vehicle 7 includes the four on-vehicle cameras 21 to 24, the determination unit 123 determines the camera shift for each on-vehicle camera 21 to 24.

  In the present embodiment, the determination unit 123 determines the deviation of the in-vehicle cameras 21 to 24 based on the estimation information obtained by the movement amount estimation unit 122 and the speed information of the vehicle 7 input from the vehicle speed sensor 41. That is, in the present embodiment, the actual measurement information includes the speed information of the vehicle 7 input to the determination unit 123 via the communication bus 50. According to this, the actual movement amount of the vehicle 7 can be acquired easily and accurately.

  The actual measurement information related to the movement of the vehicle 7 is not limited to the speed information of the vehicle 7 acquired from the vehicle speed sensor 41. For example, the measurement information related to the movement of the vehicle 7 may be movement distance information (movement amount information) of the vehicle 7 acquired from a GPS (Global Positioning System) receiver. Further, the actual measurement information related to the movement of the vehicle 7 may be a plurality of pieces of information instead of one piece of information. For example, the measurement information related to the movement of the vehicle 7 is two pieces of information of the speed information of the vehicle 7 acquired from the vehicle speed sensor 41 and the rotation angle information of the steering wheel in the vehicle 7 acquired from the steering angle sensor 42 May be

  According to the configuration of the present embodiment, the camera shift detection device 1 can detect the camera shift using the captured image of the on-vehicle cameras 21 to 24 and the sensor such as the vehicle speed sensor 41 that is already provided to the vehicle 7. The cost required for the configuration for detecting camera misalignment can be suppressed. Further, in the configuration of the present embodiment, camera displacement can be detected by statistical processing using the optical flow of the feature points of the road surface by assuming that most of the photographed image is the road surface.

  FIG. 3 is a flow chart showing an example of a camera shift detection flow by the camera shift detection device 1. In the present embodiment, the camera shift detection flow shown in FIG. 3 is performed for each of the four on-vehicle cameras 21 to 24. In order to avoid redundant description, here, the detection flow of the camera shift will be described with the case of the front camera 21 as a representative example.

  As shown in FIG. 3, first, the control unit 12 checks whether the vehicle 7 on which the front camera 21 is mounted is going straight (step S1). Whether or not the vehicle 7 is traveling straight can be determined based on, for example, rotation angle information of the steering wheel obtained from the steering angle sensor 42. For example, assuming that the vehicle 7 travels completely straight when the rotation angle of the steering wheel is zero, not only when the rotation angle is zero but also when the rotation angle is within a certain range of plus and minus directions. And the vehicle 7 may be determined to be going straight. In addition, straight running includes both straight running in the forward direction and straight running in the reverse direction.

  The control unit 12 repeats the confirmation in step S1 until it is determined that the vehicle 7 is traveling straight. As long as the vehicle 7 does not travel straight, information for determining a camera shift is not acquired. In other words, the shift determination of the on-vehicle camera (front camera) 21 by the determination unit 123 is acquired when it is determined that the vehicle 7 is traveling straight by the external sensor other than the on-vehicle camera (the steering angle sensor 42 here). It is done using information. According to this, since the determination of the camera shift is not performed using the information when the traveling direction of the vehicle 7 is curved, it is possible to prevent the information processing for determining the camera shift from being complicated.

  When it is determined that the vehicle 7 is traveling straight (Yes in step S1), the control unit 12 checks whether the speed of the vehicle 7 is within a predetermined speed range (step S2). The predetermined speed range is, for example, 3 km / hour or more and 5 km / hour or less. In the present embodiment, the speed of the vehicle 7 can be acquired by the vehicle speed sensor 41 which is an external sensor other than the on-vehicle cameras 21-24. The order of step S1 and step S2 may be reversed. Moreover, processing may be performed simultaneously with step S1 and step S2.

  When the speed of the vehicle 7 is out of the predetermined speed range (No in step S2), the control unit 12 returns to step S1 to determine whether the vehicle 7 goes straight. As long as the speed of the vehicle 7 is not within the predetermined speed range, information for determining the camera shift is not acquired. In other words, the shift determination of the on-vehicle camera (front camera) 21 by the determination unit 123 determines that the vehicle 7 is traveling within the predetermined speed range by an external sensor other than the on-vehicle cameras 21 to 24 (here, the vehicle speed sensor 41). If done, it is done using the information obtained. For example, when the speed of the vehicle 7 is too fast, the derivation of the optical flow tends to be uncertain. On the other hand, when the speed of the vehicle 7 is too slow, the reliability of the speed of the vehicle 7 acquired from the vehicle speed sensor 41 is reduced. In this respect, according to the configuration of the present embodiment, the camera shift can be determined except when the speed of the vehicle 7 is too fast or too slow, so that the reliability of the determination of the camera shift can be improved. Can.

  Preferably, the predetermined speed range can be changed. According to this, the predetermined speed range can be set to a value suitable for each vehicle, and the reliability of the determination of the camera shift can be improved. In the present embodiment, the setting of the predetermined speed range can be performed by the input unit 3.

  When it is determined that the vehicle 7 is traveling within the predetermined speed range (Yes in step S2), extraction of feature points is performed by the derivation unit 121 (step S3). In other words, when the vehicle 7 travels straight at a predetermined low speed, the derivation unit 121 extracts a feature point. Extraction of the feature points by the derivation unit 121 is performed, for example, when the stopped vehicle 7 starts moving, when the traveling vehicle 7 stops, or the like.

  FIG. 4 is a diagram for explaining a method of extracting the feature point FP. FIG. 4 schematically shows a photographed image P photographed by the front camera 21. The feature point FP exists on the road surface RS. Although the number of feature points FP is two in FIG. 4, this number is for convenience and does not indicate the actual number. Usually, a large number of feature points are obtained. As shown in FIG. 4, the derivation unit 121 extracts the feature point FP in the predetermined region RE. The predetermined area RE is set in a wide range including the central portion C of the captured image P. Thereby, the feature point FP can be extracted even when the occurrence point of the feature point FP is not uniform but in a range which is biased. In addition, the predetermined area | region RE avoids the area | region where the body BO of the vehicle 7 is reflected, and is set.

  When the feature points FP are extracted, the derivation unit 121 derives an optical flow for each of the extracted feature points FP (step S4). FIG. 5 is a diagram for explaining a method of deriving the optical flow OF. FIG. 5 is a schematic view shown for convenience as in FIG. FIG. 5 shows a photographed image (current frame P ′) photographed by the front camera 21 after a predetermined time has elapsed after photographing of the photographed image (the previous frame P) shown in FIG. After the photographing of the photographed image P shown in FIG. 4, the vehicle 7 is moving backward until the predetermined time elapses. The broken-line circle shown in FIG. 5 indicates the position of the feature point FP at the time of shooting of the shot image P shown in FIG.

  As shown in FIG. 5, when the vehicle 7 moves backward, the feature point FP present in front of the vehicle 7 leaves the vehicle 7. That is, the feature point FP appears at different positions in the current frame P ′ and the previous frame P. The deriving unit 11 associates the feature point FP of the current frame P ′ with the feature point FP of the previous frame P based on the pixel values in the vicinity, and the optical flow based on the respective positions of the associated feature points FP. Derive the OF.

  When the optical flow OF is derived, the movement amount estimation unit 122 converts each optical flow OF obtained in the camera coordinate system into an optical flow in the world coordinate system (step S5). FIG. 6 is a diagram for explaining coordinate conversion processing. As shown in FIG. 6, the movement amount estimation unit 122 converts the optical flow OF viewed from the position of the front camera 21 (viewpoint VP1) into an optical flow OF_C viewed from the viewpoint VP2 above the road surface on which the vehicle 7 is present. Do. The movement amount estimation unit 122 converts each optical flow OF in the captured image P into an optical flow OF_C in the world coordinate system by projecting the optical flow OF onto the virtual plane RS_V corresponding to the road surface.

  Next, the movement amount estimation unit 122 generates a histogram based on each optical flow OF_C of the world coordinate system (step S6). In the present embodiment, the movement amount estimation unit 122 divides each optical flow OF_C into two components in the front-rear direction and the left-right direction to generate a first histogram and a second histogram. FIG. 7 is a diagram showing an example of the first histogram HG1 generated by the movement amount estimation unit 122. As shown in FIG. FIG. 8 is a diagram illustrating an example of the second histogram HG2 generated by the movement amount estimation unit 122. 7 and 8 show histograms obtained when no camera displacement occurs.

  The first histogram HG1 shown in FIG. 7 is a histogram obtained based on the longitudinal components of each optical flow OF_C. The first histogram HG1 is a histogram in which the number of optical flows OF_C is a frequency, and the amount of movement in the front-rear direction (the length of the front-rear direction component of the optical flow OF_C) is a class. The second histogram HG2 shown in FIG. 8 is a histogram obtained based on the left-right direction component of the optical flow OF_C. The second histogram HG2 is a histogram in which the number of optical flows OF_C is a frequency, and the amount of movement in the left-right direction (the length of the left-right direction component of the optical flow OF_C) is a class.

  FIG. 7 and FIG. 8 are histograms obtained when the camera 7 does not go straight and the vehicle 7 goes straight in the backward direction. For this reason, the first histogram HG1 has a high frequency biased to a specific movement distance (class) on the backward side. On the other hand, the frequency of the second histogram HG2 is increased toward the class near the movement amount zero.

  FIG. 9 is a diagram illustrating the change of the histogram when a camera shift occurs. FIG. 9 illustrates the case where the front camera 21 is rotated and shifted in the tilt direction (vertical direction). In FIG. 9, the upper part (a) is the first histogram HG1 when no camera displacement occurs (normal time), and the lower part (b) is the first histogram HG1 when camera displacement occurs. The rotational deviation of the front camera 21 in the tilt direction mainly affects the longitudinal component of the optical flow OF_C. In the example shown in FIG. 9, due to the rotational deviation of the front camera 21 in the tilt direction, the class whose frequency increases is deviated to the front side compared to that in the normal state.

  The rotational deviation of the front camera 21 in the tilt direction has little influence on the left-right direction component of the optical flow OF_C. For this reason, although illustration is omitted, changes before and after camera shift of the second histogram HG2 are smaller than those in the case of the first histogram HG1. However, this is a story when the front camera 21 shifts in the tilt direction, and the front camera 21 shifts in, for example, the pan direction (horizontal direction) or the roll direction (rotation direction about the optical axis). In this case, different histogram changes occur.

  The movement amount estimation unit 122 estimates the movement amount of the vehicle 7 based on the generated histograms HG1 and HG2 (step S7). In the present embodiment, the movement amount estimation unit 122 estimates the movement amount of the vehicle 7 in the front-rear direction by the first histogram HG1. The movement amount estimation unit 122 estimates the movement amount of the vehicle 7 in the left-right direction by the second histogram HG2. That is, the estimation information obtained by the movement amount estimation unit 122 includes estimated values of the movement amount of the vehicle 7 in the front-rear direction and the left-right direction. According to this, since the camera shift can be detected using the estimated values of the amount of movement of the vehicle 7 in the front-rear direction and the left-right direction, the reliability of the detection result of the camera shift can be improved. In the present embodiment, the amount of movement of the vehicle 7 is estimated on the premise that feature points extracted under specific conditions exist on the road surface RS.

  In the present embodiment, the movement amount estimation unit 122 sets the median (median) of the first histogram HG1 as the estimated value of the movement amount in the front-rear direction. The movement amount estimation unit 122 sets the median of the second histogram HG2 as the estimated value of the movement amount in the left-right direction. However, the method of determining the estimated value by the movement amount estimation unit 122 is not limited to this. The movement amount estimation unit 122 may use, for example, a class in which the frequency of each of the histograms HG1 and HG2 is maximum as an estimated value of the movement amount. In addition, for example, in each of the histograms HG1 and HG2, the movement amount estimation unit 122 may use the average value as the estimated value of the movement amount.

  In the example shown in FIG. 9, the dashed-dotted line indicates the estimated value of the moving amount in the back and forth direction when the front camera 21 is normal, and the two-dot chain line indicates the moving amount in the back and forth direction when camera displacement occurs. Indicates a value. As shown in FIG. 9, it can be seen that a difference Δ occurs in the estimated values of the amount of movement in the front-rear direction due to the occurrence of the camera shift.

  When the estimated value of the moving amount of the vehicle 7 is obtained by the moving amount estimating unit 122, the determining unit 123 determines the shift of the front camera 21 (step S8). The determination unit 123 is at least one of the movement amount of the vehicle 7 in the front-rear direction, the movement amount of the vehicle 7 in the left-right direction, and the specific amount obtained based on the movement amount of the vehicle 7 in the front-rear direction and the left-right direction. The difference between the estimated value obtained by the movement amount estimation unit 122 and the value obtained by the measurement information related to the movement of the vehicle 7 input from an external sensor other than the on-vehicle cameras 21 to 24 is a predetermined threshold or more. When it becomes, it determines with the shift | offset | difference of the vehicle-mounted camera (front camera) 21 having generate | occur | produced. According to this, the estimated value of the amount of movement of the vehicle 7 obtained from the on-vehicle camera is compared with the amount of movement of the vehicle 7 obtained from the measurement information using the external sensor to appropriately detect the deviation of the on-vehicle camera be able to.

  In the present embodiment, the measurement information related to the movement of the vehicle 7 input from an external sensor other than the on-vehicle cameras 21 to 24 is speed information of the vehicle 7 input from the vehicle speed sensor 41. From this speed information, the amount of movement of the vehicle 7 to be compared with the estimated value is calculated. In detail, the movement amount is calculated by the photographing time interval of the two photographed images for deriving the optical flow OF and the speed of the vehicle 7 obtained by the vehicle speed sensor 41 during that time. In addition, corresponding to the fact that the estimated value of the movement amount obtained by the movement amount estimation unit 122 is two in the front-rear direction and the left-right direction, the movement amount has two values in the front-rear direction and the left-right direction. Do. However, in the present embodiment, when the vehicle 7 travels straight in the front-rear direction, the vehicle 7 moves in the left-right direction because the captured image for deriving the optical flow OF is captured. In addition, the amount of movement in the left and right direction calculated from the speed information is always zero.

  FIG. 10 is a flowchart illustrating an example of the camera shift determination process performed by the determination unit 123. First, the determination unit 123 determines the difference between the estimated value estimated by the movement amount estimation unit 122 and the value obtained by the speed information of the vehicle 7 obtained from the vehicle speed sensor 41 for the movement amount in the front-rear direction. It is checked whether it is smaller than 1 threshold (step S11). If the magnitude of the difference between the two is greater than or equal to the first threshold (No in step S11), the determination unit 123 determines that the mounting state of the front camera 21 is abnormal and camera misalignment occurs (step S15). . On the other hand, when the magnitude of the difference between the two is smaller than the first threshold (Yes in step S11), the determination unit 123 determines that an abnormality is not detected from the amount of movement in the front-rear direction.

  When no abnormality is detected from the amount of movement in the front-rear direction, the determination unit 123 obtains the amount of movement in the left-right direction from the estimated value estimated by the movement amount estimation unit 122 and the speed information of the vehicle 7 obtained from the vehicle speed sensor 41 It is checked whether the magnitude of the difference from the value to be calculated is smaller than the second threshold (step S12). If the magnitude of the difference between the two is greater than or equal to the second threshold (No in step S12), the determination unit 123 determines that the mounting state of the front camera 21 is abnormal and camera misalignment occurs (step S15). . On the other hand, when the magnitude of the difference between the two is smaller than the second threshold (Yes in step S12), the determination unit 123 determines that an abnormality is not detected from the movement amount in the left and right direction.

  When no abnormality is detected from the amount of movement in the left and right direction, the determination unit 123 determines the estimated value estimated by the movement amount estimation unit 122 for the specific amount obtained based on the amount of movement in the front and rear direction and the left and right direction It is checked whether the magnitude of the difference with the value obtained by the speed information of the vehicle 7 obtained from 41 is smaller than the third threshold (step S13). In the present embodiment, the specific amount is a square root value of a sum of a value obtained by squaring the amount of movement in the front-rear direction and a value obtained by squaring the amount of movement in the left-right direction. However, this is merely an example, and the specific amount may be, for example, a sum of a value obtained by squaring the amount of movement in the front-rear direction and a value obtained by squaring the amount of movement in the left-right direction.

  When the magnitude of the difference between the estimated value of the specific amount and the value determined by the speed information is equal to or greater than the third threshold (No in step S13), the determination unit 123 determines that the mounting state of the front camera 21 is abnormal. It is determined that camera shift has occurred (step S15). On the other hand, when the magnitude of the difference between the two is smaller than the third threshold (Yes in step S13), the determination unit 123 determines that the attachment state of the front camera 21 is normal (step S14).

  In the present embodiment, the estimated value obtained by the moving amount estimating unit 122 and the value obtained by the speed information are at least one of the moving amount in the front-rear direction, the moving amount in the left-right direction, and the specific amount. If an abnormality is found by the comparison, it is determined that a camera shift has occurred. According to this, it is possible to reduce the possibility of determining that the camera shift has not occurred although the camera shift has occurred. However, this is an example. For example, when abnormality is recognized by comparison between the estimated value obtained by the movement amount estimation unit 122 and the value obtained from the velocity information in all of the movement amount in the front-rear direction, the movement amount in the left-right direction, and the specific amount. The configuration may be limited to determine that a camera shift has occurred. It is preferable that the determination criterion of the camera shift can be appropriately changed by the input unit 3.

  Although the present embodiment is configured to sequentially compare the estimated value and the value obtained from the speed information for the amount of movement in the front-rear direction, the amount of movement in the left-right direction, and the specific amount, these comparisons have the same timing May be performed. In the case of a configuration in which the estimated value and the value obtained from the speed information are sequentially compared for the movement amount in the front-rear direction, the movement amount in the left-right direction, and the specific amount, the order is not particularly limited. The order may be different from the order shown in FIG. In the present embodiment, the camera shift is determined based on the movement amount, but the camera shift may be determined based on the speed. In this case, the longitudinal and lateral velocities are estimated based on the movement amount estimated from the optical flow and the time between frames used for calculating the optical flow, and compared with the velocity of the vehicle 7 obtained from the vehicle speed sensor 41 The camera deviation may be determined by The speed of the vehicle 7 obtained from the vehicle speed sensor 41 in this case is the speed in the front-rear direction, and the speed in the left-right direction is zero. That is, in the present embodiment, the movement amount and the speed are synonymous.

  In addition, when a camera shift is detected, it is preferable that the camera shift detection apparatus 1 perform a process for notifying the driver or the like of the fact. In addition, it is preferable that the camera shift detection device 1 perform a process of notifying the driving assistance device that performs driving assistance using information from the on-vehicle cameras 21 to 24 that a camera shift is occurring. In the present embodiment, there are four in-vehicle cameras 21-24, but it is preferable to perform the notification process and the notification process when camera displacement occurs even in one of the four in-vehicle cameras 21-24. .

<3. Processing related to abnormal condition>
FIG. 11 is a schematic view of a photographed image P photographed by the on-vehicle cameras 21-24. As shown in FIG. 11, a shadow SH of the vehicle 7 (the host vehicle) may appear in the captured image P captured by the on-vehicle cameras 21 to 24. For example, when the solar radiation direction is forward from the rear of the traveling direction of the vehicle 7, the shadow SH of the vehicle 7 may be reflected on the front camera 21. As shown in FIG. 11, when the shadow SH falls within the predetermined region RE for extracting the feature point FP, the derivation unit 121 may extract the feature point FP derived from the shadow SH. Since the feature point FP derived from the shadow SH moves with the vehicle 7, the size of the optical flow OF based on the feature point FP is zero, and the movement of the vehicle 7 is not reflected.

  That is, in the above-described method for detecting a camera shift using the optical flow OF, the movement amount estimation unit 122 obtains the movement despite the fact that the vehicle 7 is moving based on an external sensor (for example, the vehicle speed sensor 41). The estimated information may cause an abnormal state in which it is estimated that the vehicle 7 is at a stop. Since this abnormal state occurs even when no camera shift has occurred, there is a possibility that the camera shift may be erroneously detected. For this reason, it is preferable that the determination unit 123 stop the shift determination process of the in-vehicle cameras 21 to 24 when an abnormal state occurs. According to this, it is possible to suppress the occurrence of an erroneous determination that it is determined that a camera shift has occurred although the camera shift has not occurred.

  FIG. 12 is a flow chart showing a first embodiment different from the detection flow of camera misalignment shown in FIG. In addition, the detection flow of a camera offset shown in FIG. 12 is implemented about each of four vehicle-mounted cameras 21-24. In the example shown in FIG. 12, the same processing as the example shown in FIG. 3 is performed in steps S1 to S7. Since these processes are the same as the processes described above, the description will be omitted. In the example shown in FIG. 12, after the movement amount estimation process of step S7, an abnormal state confirmation process is performed (step S21). The abnormal state confirmation process is performed by, for example, the determination unit 123, but not limited to this, for example, may be performed by the movement amount estimation unit 122.

  For example, based on the speed information obtained from the vehicle speed sensor 41, the determination unit 123 confirms whether or not the vehicle 7 has moved in the front-rear direction at the shooting time interval of two shot images for deriving the optical flow OF. . However, in the present embodiment, since the captured image for deriving the optical flow OF is detected when the vehicle 7 is moving in the front-rear direction according to the information obtained from the sensor unit 4, the determination unit In 123, confirmation of whether the above-mentioned vehicle 7 moved to the front-back direction is not performed substantially. That is, the determination unit 123 only confirms whether or not it is estimated that the vehicle 7 is stopped by the movement amount estimation value in the front-rear direction obtained by the movement amount estimation unit 122. When it is estimated that the vehicle 7 has stopped, the determination unit 123 detects an abnormal state. On the other hand, when it is estimated that the vehicle 7 is not stopped, the determination unit 123 does not detect an abnormal state. In addition, not only when the movement amount estimated value in the front-rear direction is zero, but also when it is near zero, it may be estimated that the vehicle 7 is stopped.

  When an abnormal state is not detected (No in step S21), the determination unit 123 performs the shift determination process of step S8 described above. On the other hand, when an abnormal state is detected (Yes in step S21), the determination unit 123 stops the deviation determination process (step S22). That is, the determination unit 123 does not perform the deviation determination process. In the present embodiment, the in-vehicle cameras 21 to 24 are plural. When an abnormal state occurs in some of the plurality of on-vehicle cameras 21 to 24, the deviation determination process is performed on the remaining on-vehicle cameras in which the abnormal state does not occur. As a result, camera displacements of the plurality of on-vehicle cameras 21 to 24 can be efficiently detected.

  After the determination processing of the camera shift is stopped, it is checked whether the stop of the shift determination processing is continued (step S23). The confirmation process is performed by, for example, the determination unit 123. In detail, the determination unit 123 monitors the traveling direction of the vehicle 7 by the information obtained from the steering angle sensor 42. If the traveling direction of the vehicle 7 is constant, the determination unit 123 determines to continue stopping the deviation determination process (Yes in step S23). The determination unit 123 repeats the confirmation process of step S23 until the traveling direction of the vehicle 7 changes. When the traveling direction of the vehicle 7 changes, the determination unit 123 determines that the stop of the deviation determination process is not continued (No in step S23). In other words, the determination unit 123 continues stopping the deviation determination process until the traveling direction of the vehicle 7 changes. When it is determined that the traveling direction of the vehicle 7 has changed by a predetermined angle or more based on the information obtained from the steering angle sensor 42, for example, the determination unit 123 determines that the vehicle 7 has changed the traveling direction.

  When the abnormal state is caused by the shadow SH of the vehicle 7, if the traveling direction of the vehicle 7 is constant, the abnormal state caused by the shadow SH of the vehicle 7 may be detected again even if the determination is made again. Is high. Therefore, by continuing the stop of the deviation determination process until the traveling direction of the vehicle 7 changes, repeated detection of an abnormal state can be suppressed.

  In the above example, the deviation determination process is stopped when an abnormal state is detected once. However, the deviation determination process may be stopped when an abnormal state is detected a plurality of times in succession. That is, when an abnormal state is detected at least once, the determination unit 123 may continue stopping the deviation determination process until the traveling direction of the vehicle 7 changes.

  In addition, the traveling direction of the vehicle 7 may be monitored using information from a GPS sensor instead of the information from the steering angle sensor 42, for example. In addition to the information from the steering angle sensor 42, for example, the traveling direction of the vehicle 7 may be monitored using information from a GPS sensor.

  Also, in the abnormal state, although it is recognized that the vehicle 7 is moving based on the external sensor (for example, the vehicle speed sensor 41), the movement is made in the plurality of optical flows OF derived by the deriving unit 121. There may also be included a case where there is a predetermined ratio or more of optical flows OF determined to be absent. The predetermined ratio may be appropriately determined by experiment, simulation or the like. When there are a large number of optical flows OF judged to have no movement among the plurality of optical flows OF derived by the deriving unit 121 although the vehicle 7 is moving, the movement amount of the movement amount estimating unit 122 The reliability of the estimate is reduced and the possibility of falsely detecting camera misalignment is increased. According to this configuration, since it is possible to widen the range where the reliability of the movement amount estimated value of the vehicle 7 is expected to be low and stop the determination of the camera shift, there is a possibility that an erroneous determination regarding the camera shift may occur. It can be reduced.

  If it is determined by the determination unit 123 that the stop of the deviation determination process is not continued (No in step S23), the process returns to step S1 and a photographed image is acquired at a predetermined timing and the derivation process of the optical flow OF by the derivation unit 121 Is resumed. That is, when the traveling direction of the vehicle 7 changes, movement amount estimation by the movement amount estimation unit 122 and deviation determination processing by the determination unit 123 are performed based on the plurality of optical flows OF newly acquired after the change of the movement direction. It will be. According to this, there is a high possibility that the optical flow OF can be derived using a captured image in which the shadow SH of the vehicle 7 is not reflected, and the possibility of an erroneous determination regarding camera shift can be reduced. In the example shown in FIG. 12, when the abnormal state is detected again based on a plurality of newly acquired optical flows OF, the deviation determination process is stopped.

  FIG. 13 is a flow chart showing a second embodiment different from the detection flow of camera misalignment shown in FIG. In addition, the detection flow of a camera offset shown in FIG. 13 is implemented about each of four vehicle-mounted cameras 21-24. Also in the example shown in FIG. 13, the same processing as the example shown in FIG. 3 is performed from step S1 to step S7 as in the example shown in FIG. Also in the example shown in FIG. 13, as in the example shown in FIG. 12, after the movement amount estimation process of step S7, an abnormal state confirmation process is performed (step S21). The process of confirming the abnormal state is the same as the process described above, and therefore the description thereof is omitted. Further, the process (step S8) in the case where an abnormal state is not detected (No in step S21) is also the same as the above-described process, and thus the description is omitted.

  In the example shown in FIG. 13, unlike the example shown in FIG. 12, when an abnormal state is detected (Yes in step S21), the deviation determination processing by the determination unit 123 is not immediately stopped, but is determined It is confirmed whether the process is stopped (step S24). The confirmation process is performed by, for example, the determination unit 123.

  In detail, the determination unit 123 confirms whether or not the position of the shadow SH of the vehicle 7 (own vehicle) can be recognized. If the position of the shadow SH of the vehicle 7 can not be recognized, the determination unit 123 determines that the shift determination process is to be stopped (Yes in step S24). On the other hand, when the position of the shadow SH of the vehicle 7 can be recognized, the determination unit 123 determines that the shift determination process is not to be stopped (No in step S24). The position of the shadow SH of the vehicle 7 is determined based on, for example, the contrast of a captured image captured to obtain the optical flow OF. Besides, the position of the shadow SH of the vehicle 7 is, for example, the time of the photographed image taken to obtain the optical flow OF, the weather forecast at the time, vehicle information obtained via the communication bus 50 (for example It may be determined by one or more pieces of information such as direction).

  If it is determined that the deviation determination process is to be stopped (Yes in step S24), the above-described step S23 is performed. Since the said process is the same as the above-mentioned process, description is abbreviate | omitted. On the other hand, when it is determined that the shift determination process is not to be stopped (No in step S24), a process of changing the feature point extraction range is performed (step S25).

  FIG. 14 is a schematic diagram for explaining the process of changing the feature point extraction range RE. As described above, when it is determined that the deviation determination process is not to be stopped, the position of the shadow SH of the vehicle 7 can be recognized. Therefore, in deriving the optical flow OF, it is possible to specify a position which is not affected by the shadow SH of the vehicle 7. For this reason, as shown in FIG. 14, the feature point extraction range RE 'is set at a position shifted from the shadow SH of the vehicle 7 (a position not overlapping the shadow SH).

  When the feature point extraction position RE 'is set, the process returns to step S3 and the optical flow OF is derived in the newly set extraction range, and the movement amount estimation by the movement amount estimation unit 122 is performed based on the optical flow OF. A shift determination process by the determination unit 123 is performed. The captured image for deriving the optical flow OF is the same as the captured image previously used to detect an abnormal state.

  As described above, in the example illustrated in FIG. 13, when the position of the shadow SH of the vehicle 7 can be recognized when an abnormal state occurs, the shift determination processing by the determination unit 123 is not stopped, and The movement amount estimation by the movement amount estimation unit 122 and the deviation determination processing by the determination unit 123 are performed based on a plurality of optical flows OF derived using feature points FP located in the deviated range. According to this, it is possible to appropriately determine the camera shift without stopping the shift determination process. That is, camera shift can be detected quickly.

<4. Modified example etc>
<4-1. First Modified Example>
In the above, the determination unit 123 determines that the camera shift has occurred only once it has been determined that the camera shift has occurred, and the camera shift is detected. The present invention is not limited to this. When it is determined by the determination unit 123 that a camera shift has occurred, redetermination is performed at least once, and when it is determined that another camera shift has occurred by redetermination, a camera shift It may be determined that the occurrence of The re-determination may simply be configured to repeat steps S1 to S8 illustrated in FIG. 3 at least once, but may be another configuration. The re-determination is also applicable to the flow to which the procedure for detecting an abnormal state shown in FIG. 12 or 13 is added.

  FIG. 15 is a diagram for describing an example of the camera shift re-determination process. The description of the feature points is omitted in FIG. As illustrated in FIG. 15, the determination unit 123 determines that the on-vehicle camera is shifted based on the estimated information estimated using the optical flow of the feature point in the first region RE1 including the central portion C of the image. In this case, the displacement of the on-vehicle camera may be re-judged based on the estimated information obtained using the optical flow of the feature points in the second region RE2 not including the central portion C of the image. The image used in the redetermination may be the same as the image used in the previous determination of the camera shift, but in some cases it is an image captured again after the image used in the previous camera shift determination It is also good.

  The second region RE2 may be one or plural. In the example shown in FIG. 15, the second region RE2 is two, but may be three or more. When there are a plurality of second areas RE2, in each second area RE2, derivation of the optical flow OF by the derivation unit 121, acquisition of estimated movement amount information by the movement amount estimation unit 122, and camera displacement by the determination unit 123 A determination is made. The second region RE2 is preferably a region smaller than the first region RE1.

  When the camera shift of the on-vehicle cameras 21 to 24 occurs, the moving amount of the vehicle 7 obtained by the moving amount estimating unit 122, the vehicle speed sensor 41, and the like in the region farther from the central portion C than the central portion C of the image. The difference between the amount of movement of the vehicle 7 and the amount of movement of the vehicle 7 obtained by the actual measurement information obtained from the sensor For this reason, by performing redetermination using the second region RE2 located away from the central portion C of the image, it is possible to improve the detection accuracy of the camera shift.

<4-2. Second Modified Example>
FIG. 16 is a flow chart showing a modification of the camera shift detection flow by the camera shift detection device 1. The processes in steps S1 to S7 shown in FIG. 16 are the same as the processes shown in FIG. In the present modification, after the estimated value of the movement amount is obtained in step S7, a process of adding the estimated value of the movement amount obtained to the cumulative value of the movement amounts obtained in advance is performed (step S9). The process may be performed by, for example, the movement amount estimation unit 122 or the determination unit 123. The accumulated value calculation process is performed on the estimated value of the movement amount in the front-rear direction and the estimated value of the movement amount in the left-right direction. In addition, when the first estimated value is obtained, the accumulated value obtained in the accumulated value calculation process becomes the estimated value itself obtained in step S7 because there is no previously obtained accumulated value.

  When the cumulative value is calculated, it is checked whether the movement amount of the vehicle 7 has reached a predetermined movement distance (step S10). In this modification, the amount of movement of the vehicle 7 referred to here is obtained by integrating the amount of movement of the vehicle 7 in the front-rear direction at the photographing time interval of the two images for deriving the optical flow OF. The amount of movement of the vehicle 7 in the front-rear direction is calculated from the speed information obtained by the vehicle speed sensor 41. The start of integration of the movement amount is performed at the start of the accumulation processing of the estimated value. If the amount of movement of the vehicle 7 in the front-rear direction does not reach the predetermined movement distance (No in step S10), the process returns to step S1, and steps S1 to S7 and step S9 are sequentially performed.

  On the other hand, when the movement amount of the vehicle 7 in the front-rear direction reaches a predetermined movement distance (Yes in step S10), the determination unit 123 determines the camera shift (step S8A). The determination unit 123 uses an accumulated value as the estimated value of the movement amount. In the present embodiment, the accumulated value includes an accumulated value in the front-rear direction and an accumulated value in the left-right direction. The value to be compared is a value obtained by integrating the amount of movement of the vehicle 7 at the photographing time interval of the two images for deriving the optical flow OF, and is calculated from the speed information obtained by the vehicle speed sensor 41. The values include the amount of movement in the front-rear direction and the amount of movement in the left-right direction. However, in this modification, since the photographed image for deriving the optical flow OF is photographed when the vehicle 7 travels straight in the front-rear direction, the vehicle 7 moves in the left-right direction during this period. The movement amount in the left and right direction (a comparison value to be compared with the accumulated value in the left and right direction) is zero.

  In this modification, the process of adding the estimated values obtained by the movement amount estimation unit 122 and obtaining the cumulative value is performed until the movement amount of the vehicle 7 reaches a predetermined distance, and the determination unit 123 performs the movement amount of the vehicle When it reaches a predetermined distance, the shift of the on-vehicle camera is determined based on the accumulated value. According to this, when camera shift has occurred, shift determination can be performed at the timing at which the difference between the estimated value and the value obtained from the actual measurement information appears clearly, and the reliability of shift determination can be improved. Can. The configuration of the present modification is also applicable to the flow to which the procedure for detecting an abnormal state shown in FIG. 12 or 13 is added.

<4-3. Third Modified Example>
FIG. 17 is a block diagram showing a configuration of a camera shift detection device 1A of a modification. The camera shift detection device 1A differs from the embodiment described above in that the camera shift detection device 1A further includes the shadow position estimation unit 124. The shadow position estimation unit 124 is a functional unit realized by the CPU performing arithmetic processing according to a program stored in the storage unit 13 by the CPU of the control unit 12. The shadow position estimation unit 124 estimates the shadow position of the vehicle 7 (own vehicle).

  The shadow position may be estimated based on, for example, time, season, weather, the traveling direction of the vehicle 7, or the like. Information on the season and the weather may be acquired from, for example, a server apparatus or the like via the Internet or the like. In addition, the traveling direction of the vehicle 7 may be acquired based on information obtained from the steering angle sensor 42 via the communication bus 50, for example.

  FIG. 18 is a flow chart showing an example of a camera shift detection flow by the camera shift detection device 1A of the modification. Also in the camera shift detection device 1A, steps S1 and S2 are executed similarly to the example shown in FIG. 3, and when the vehicle 7 travels straight at a low speed, the captured image for deriving the optical flow OF is captured Be done. When the photographed image is photographed, the shadow position estimation unit 124 estimates a shadow position (step S31).

When the shadow position is estimated, the derivation unit 121 sets the extraction range of the feature point FP based on the estimated shadow position (step S31). Specifically, the derivation unit 121 sets the extraction range of the feature point FP at a position deviated from the estimated shadow position (a position not overlapping the shadow position). After this, step S3 and subsequent steps shown in FIG. 3, FIG. 12, FIG. 13, or FIG. 16 described above are executed.
According to this, it is possible to reduce the possibility that the movement amount estimated value of the vehicle 7 based on the optical flow OF becomes inaccurate due to the influence of the shadow SH, and it is possible to quickly and appropriately determine the camera shift.

<4-4. Other>
The configurations of the embodiments and modifications in the present specification are merely examples of the present invention. The configurations of the embodiment and the modifications may be appropriately changed without departing from the technical concept of the present invention. Moreover, several embodiment and modification may be implemented in combination as much as possible.

  In the above, the data used for the shift determination of the on-vehicle cameras 21 to 24 is configured to be collected when the vehicle 7 is traveling straight. However, this is an example, and data used for shift judgment of in-vehicle cameras 21-24 may be collected when vehicles 7 do not go straight on. By using the speed information obtained from the vehicle speed sensor 41 and the information obtained from the steering angle sensor 42, it is possible to accurately obtain the actual amount of movement of the vehicle 7 in the longitudinal and lateral directions. The above-described deviation determination can be performed even when not being performed.

  In the above, the movement amount estimation unit 122 is configured to obtain the estimated value of the movement amount in the front-rear direction and the estimated value of the movement amount in the left-right direction. However, only one of them may be obtained. However, in a configuration in which only one of the estimated values is obtained, it is preferable that the movement amount estimation unit 122 obtain only the estimated value of the movement amount in the front-rear direction. In this case, for example, the determination unit 123 compares the estimated value obtained by the movement amount estimation unit 122 with the value obtained from the actual measurement information acquired by the sensor unit 4 with respect to the movement amount in the front-rear direction, You may make a decision.

1, 1A: camera deviation detection device 7: vehicle 21: front camera (vehicle-mounted camera)
22 · · · back camera (vehicle-mounted camera)
23 ··· Left side camera (vehicle-mounted camera)
24 ··· Right side camera (vehicle-mounted camera)
41 ... vehicle speed sensor (external sensor)
42 · · · rudder angle sensor (external sensor)
121: Derivation unit 122: Movement amount estimation unit 123: Judgment unit 124: Shadow position estimation unit FP: Feature point OF: Optical flow

Claims (8)

A camera shift detection device for detecting a shift in mounting of a camera mounted on a moving body, comprising:
A derivation unit that derives an optical flow, which is the movement of a feature point between two images input with a predetermined time by the camera, for each feature point;
A movement amount estimation unit configured to estimate a movement amount of a moving object equipped with the camera based on a plurality of the optical flows;
A determination unit that determines the displacement of the camera based on the estimation information obtained by the movement amount estimation unit and the measurement information related to the movement of the moving object input from an external sensor other than the camera;
Equipped with
The determination unit is configured to detect an abnormal state in which the moving object is estimated to be stopped based on the estimation information despite the fact that the moving object is recognized as moving based on the external sensor. A camera shift detection device for stopping the shift determination process of the camera.   In the abnormal state, it is determined that there is no movement in the plurality of optical flows derived by the deriving unit although the moving body is recognized as moving based on the external sensor. The camera shift detector according to claim 1, wherein the case where the optical flow is present at a predetermined rate or more is also included.   The camera shift detection device according to claim 1, wherein the determination unit continues stopping the shift determination process until the traveling direction of the moving body changes when the abnormal state is detected at least once. .   When the moving direction of the moving body changes, movement amount estimation by the movement amount estimation unit and the deviation determination processing by the determination unit are performed based on the plurality of optical flows newly acquired after the change of the movement direction. The camera shift detection device according to claim 3, which is performed.   When the position of the shadow of the moving body can be recognized when the abnormal state occurs, the displacement determination process is not stopped, and a plurality of the features are derived using the feature points located in the range shifted from the shadow. The camera displacement detection device according to any one of claims 1 to 4, wherein the movement amount estimation by the movement amount estimation unit and the deviation determination processing by the determination unit are performed based on the optical flow. When there are a plurality of the cameras and the abnormal state occurs in some of the plurality of cameras,
The camera shift detection device according to any one of claims 1 to 5, wherein the shift determination process is performed on the remaining cameras in which the abnormal state has not occurred. It further comprises a shadow position estimation unit for estimating the shadow position of the moving body,
The camera deviation detection device according to any one of claims 1 to 6, wherein the derivation unit sets an extraction range of the feature point based on the estimated shadow position. A camera shift detection method for detecting a mount shift of a camera mounted on a moving body, comprising:
A derivation step of deriving, for each feature point, an optical flow that is a motion of the feature point between two images input with a predetermined time by the camera;
Estimating an amount of movement of a mobile unit on which the camera is mounted based on a plurality of the optical flows;
A determination step of determining the deviation of the camera based on the estimation information obtained in the estimation step and the measurement information related to the movement of the moving object input from a sensor other than the camera;
Equipped with
The camera in the determination step when an abnormal state occurs in which it is estimated that the moving body is stopped in the estimation step despite the fact that the moving body is recognized to be moving based on the sensor. A camera shift detection method in which the shift determination process is stopped.

Images