JP2021034857A - Posture estimation device, anomaly detection device, correction device, and posture estimation method - Google Patents

Abstract

To accurately estimate the posture of a camera and restrict lengthening of a time required for the estimation.SOLUTION: A posture estimation device comprises an acquisition unit, an extraction unit, a specifying unit, and an estimation unit. The acquisition unit acquires a photographic image photographed by a camera mounted in a moving body. The extraction unit extracts feature points from respective photographic images photographed at first to fourth times. The second and third times are later than the first time, and the fourth time is later than the third time. The specifying unit selects a first feature point from the feature points extracted from the photographic images photographed at the first and second times, selects a second feature point from the feature points extracted from the photographic images photographed at the third and fourth times, calculates respective positional changes of the first feature point and the second feature point and, based on the respective positional changes of the first feature point and the second feature point, specifies two pairs of surfaces in which respective lines intersecting a predetermined flat surface are parallel. Based on the pairs of surfaces specified by the specifying unit, the estimation unit estimates a posture of the camera.SELECTED DRAWING: Figure 1

Description

The present invention relates to a posture estimation device that estimates the posture of the camera, an abnormality detection device that detects an abnormality in the camera, a correction device that corrects parameters of the camera, and a posture estimation method that estimates the posture of the camera.

Conventionally, driving support such as vehicle parking support has been provided by using an in-vehicle camera. The on-board camera is fixedly attached to the vehicle before it is shipped from the factory. However, the on-board camera may be displaced from the factory-installed state due to, for example, unexpected contact or aging. If the position of the in-vehicle camera (the position of the in-vehicle camera relative to the body of the vehicle) deviates, an error occurs in the steering amount of the steering wheel determined by using the camera image. is important.

Patent Document 1 discloses a technique for detecting an optical axis deviation of an in-vehicle camera. The optical axis deviation detecting device of the vehicle-mounted camera in Patent Document 1 includes an image processing means and a determining means. The image processing means detects the position information of the marking from the captured image of the vehicle-mounted camera that captures the range including the marking on the vehicle body portion that supports driving. The determination means determines the shooting optical axis deviation by comparing the initially set marking position information with the newly detected marking position information.

Japanese Unexamined Patent Publication No. 2004-173037

Since the image taken by the in-vehicle camera reflects the scenery around the vehicle, for example, when the marking is a specific shape of a part of the bonnet, the specific shape may not be easily extracted. If the specific shape is erroneously detected, the optical axis deviation of the in-vehicle camera may not be accurately determined. The optical axis deviation of the in-vehicle camera means that the attitude of the in-vehicle camera is not as designed.

In addition, if it takes a long time to determine whether or not the posture of the in-vehicle camera is as designed, if a state in which the posture of the in-vehicle camera is not as designed occurs, the implementation of countermeasures against the state will be delayed. .. Therefore, it is desirable that the time required to determine whether or not the posture of the in-vehicle camera is as designed is short.

In view of the above problems, it is an object of the present invention to provide a technique capable of accurately estimating the posture of a camera and suppressing a prolongation of the time required for estimation.

The posture estimation device according to the present invention includes an acquisition unit that acquires a photographed image taken by a camera mounted on a moving body, and two above-mentioned images taken at a first time and a second time after the first time. An extraction unit that extracts feature points from each of the captured images and extracts feature points from each of the two captured images captured at the third time after the first time and the fourth time after the third time. , The first feature points corresponding to each other are selected from the feature points extracted from the two captured images taken at the first time and the second time, and the position change of the first feature points is calculated. From the feature points extracted from the two captured images taken at the third time and the fourth time, different from the first feature point, the second feature points corresponding to each other are selected, and the second feature is selected. A specific unit that calculates the position change of a point and specifies two sets of surfaces whose lines of intersection with a predetermined plane are parallel to each other based on the position change of the first feature point and the position change of the second feature point. It is a configuration (first configuration) including an estimation unit that estimates the posture of the camera based on a set of surfaces specified by the specific unit.

In the posture estimation device having the first configuration, the combination of the position change of the first feature point and the position change of the second feature point is determined based on the movement information of the moving body (second configuration). ) May be.

In the posture estimation device having the second configuration, the movement information may have a configuration (third configuration) including steering angle information of the moving body.

In the posture estimation device having the second or third configuration, the movement information may be configured to include the speed information of the moving body (fourth configuration).

In the posture estimation device having any of the second to fourth configurations, the movement information may be supplied to the posture estimation device via the communication bus (fifth configuration).

In the posture estimation device having any of the second to fourth configurations, the movement information is the result of coordinate-converting the position change of the candidate of the first feature point on the predetermined plane, and the second feature point. It may be a configuration (sixth configuration) that is information estimated based on the result of coordinate conversion of the position change of the candidate of the above on the predetermined plane.

In the posture estimation device having any of the first to sixth configurations, the combination of the position change of the first feature point and the position change of the second feature point is at least one of the first time and the second time. The configuration may be determined based on the time difference between the time related to the above and the time related to at least one of the third time and the fourth time (seventh configuration).

In the posture estimation device having any of the first to seventh configurations, the specific unit is the speed of the moving body from the first time to the second time and the speed of the moving body from the third time to the fourth time. Based on the difference from the speed of the moving body, at least one of the position change of the first feature point and the position change of the second feature point is corrected, and at least one of the corrected position changes of the first feature point and the said Based on the change in the position of the second feature point, two sets of surfaces whose intersection lines with the predetermined plane are parallel to each other may be specified (eighth configuration).

The abnormality detection device according to the present invention is in a state in which the mounting of the camera is misaligned based on the posture estimation device having any of the first to eighth configurations and the posture of the camera estimated by the estimation unit. It is a configuration (nineth configuration) including a determination unit for determining whether or not there is a presence.

The correction device according to the present invention includes a posture estimation device having any of the first to eighth configurations, a correction unit that corrects the parameters of the camera based on the posture of the camera estimated by the estimation unit, and a correction unit. (10th configuration).

The posture estimation method according to the present invention includes an acquisition step of acquiring a captured image taken by a camera mounted on a moving body, and two above-mentioned two images taken at a first time and a second time after the first time. An extraction step of extracting feature points from each of the captured images and extracting feature points from each of the two captured images captured at the third time after the first time and the fourth time after the third time. , The first feature points corresponding to each other are selected from the feature points extracted from the two captured images taken at the first time and the second time, and the position change of the first feature points is calculated. From the feature points extracted from the two captured images taken at the third time and the fourth time, different from the first feature point, the second feature points corresponding to each other are selected, and the second feature is selected. A specific step of calculating the position change of a point and specifying two sets of surfaces whose intersection lines with a predetermined plane are parallel to each other based on the position change of the first feature point and the position change of the second feature point. It is a configuration (11th configuration) including an estimation step of estimating the posture of the camera based on the set of surfaces specified in the specific step.

According to the present invention, the posture of the camera can be estimated with high accuracy, and the time required for the estimation can be suppressed from being prolonged.

Block diagram showing the configuration of the anomaly detection system The figure which illustrates the position where the in-vehicle camera is arranged in a vehicle The figure which shows an example of the photographed image schematically The figure which shows the other example of the photographed image schematically. The figure which shows the other example of the photographed image schematically. A flowchart showing an example of the posture estimation flow of the camera by the posture estimation device. The figure which shows typically the photographed image before the L frame from the current frame. Diagram showing the current frame schematically A diagram schematically showing a captured image N frames before the current frame. The figure which shows typically the photographed image M frame before the current frame. The figure which shows typically the photographed image before the L frame from the current frame. Diagram showing the current frame schematically Diagram schematically showing a virtually formed quadrangle The figure for demonstrating the method of specifying two sets of planes whose intersection lines with a predetermined plane are parallel to each other. A diagram for explaining a method of finding the direction of intersection between faces based on one of the specified sets of faces. A diagram for explaining a method of finding the direction of intersection between faces based on the other side of the specified set of faces. Block diagram showing the configuration of the correction system

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the following description, a vehicle will be described as an example of a moving body, but the moving body is not limited to the vehicle. Vehicles include a wide range of vehicles with wheels, such as automobiles, trains, and automatic guided vehicles. Examples of moving objects other than vehicles include ships and aircraft.

Further, in the following description, the direction in which the vehicle travels straight from the driver's seat to the steering wheel is referred to as the "forward direction". Further, the direction in which the vehicle travels straight from the steering wheel to the driver's seat is defined as the "rear direction". Further, the direction from the right side to the left side of the driver who is facing the front direction, which is the direction perpendicular to the straight traveling direction and the vertical straight line of the vehicle, is defined as the "left direction". Further, the direction from the left side to the right side of the driver who is facing the front direction, which is the direction perpendicular to the straight line and the vertical direction of the vehicle, is defined as the "right direction".

<1. Anomaly detection system>
FIG. 1 is a block diagram showing a configuration of an abnormality detection system SYS1 according to an embodiment. In the present embodiment, the abnormality is a state in which the mounting of the camera is misaligned. That is, the abnormality detection system SYS1 is a system that detects the misalignment of the camera (vehicle-mounted camera) mounted on the vehicle. As shown in FIG. 1, the abnormality detection system SYS1 includes an abnormality detection device 1 and a photographing unit 2.

The abnormality detection device 1 is a device that detects an abnormality based on a captured image taken by an in-vehicle camera. The abnormality detection device 1 is provided for each vehicle equipped with an in-vehicle camera. In the present embodiment, the abnormality detection device 1 acquires a photographed image from the photographing unit 2.

The photographing unit 2 is provided for the purpose of monitoring the situation around the vehicle. The photographing unit 2 includes four cameras 21 to 24. The four cameras 21 to 24 are in-vehicle cameras. FIG. 2 is a diagram illustrating positions where four in-vehicle cameras 21 to 24 are arranged in the vehicle 5.

The in-vehicle camera 21 is provided at the front end of the vehicle 5. Therefore, the in-vehicle camera 21 is also referred to as a front camera 21. The optical axis 21a of the front camera 21 is along the front-rear direction of the vehicle 5 in a plan view from above. The front camera 21 photographs the front direction of the vehicle 5. The in-vehicle camera 22 is provided at the rear end of the vehicle 5. Therefore, the in-vehicle camera 22 is also referred to as a back camera 22. The optical axis 22a of the back camera 22 is along the front-rear direction of the vehicle 5 in a plan view from above. The back camera 22 captures the rear direction of the vehicle 5. The front camera 21 and the back camera 22 are preferably mounted at the center of the left and right sides of the vehicle 5, but may be slightly deviated from the center of the left and right sides in the left-right direction.

The in-vehicle camera 23 is provided on the left side door mirror 51 of the vehicle 5. Therefore, the in-vehicle camera 23 is also referred to as a left side camera 23. The optical axis 23a of the left side camera 23 is along the left-right direction of the vehicle 5 in a plan view from above. The left side camera 23 photographs the left direction of the vehicle 5. The in-vehicle camera 24 is provided on the right door mirror 52 of the vehicle 5. Therefore, the in-vehicle camera 24 is also referred to as a right side camera 24. The optical axis 24a of the right side camera 24 is along the left-right direction of the vehicle 5 in a plan view from above. The right side camera 24 photographs the vehicle 5 in the right direction. When the vehicle 5 is a so-called door mirrorless vehicle, the left side camera 23 is attached around the rotation axis (hinge portion) of the left side door without passing through the door mirror, and the right side camera 24 is attached to the right side door. It can be attached around the rotating shaft (hinge part) without going through a door mirror.

The horizontal angle of view θ of each of the vehicle-mounted cameras 21 to 24 is 180 degrees or more. Therefore, the vehicle-mounted cameras 21 to 24 can photograph the entire circumference of the vehicle 5 in the horizontal direction. Further, the body of the vehicle 5 equipped with the in-vehicle cameras 21 to 24 is reflected in the images taken by the in-vehicle cameras 21 to 24.

In the present embodiment, the number of in-vehicle cameras is four, but this number may be changed as appropriate, and may be plural or singular. For example, when an in-vehicle camera is mounted for the purpose of assisting the vehicle 5 to park in the back, the in-vehicle camera is composed of three, a back camera 22, a left side camera 23, and a right side camera 24. May be good.

Returning to FIG. 1, in the present embodiment, the output of the steering angle sensor 3 for detecting the rotation angle of the steering wheel (steering wheel) and the output of the vehicle speed sensor 4 for detecting the speed of the vehicle 5 are CAN (Controller Area Network). It is input to the abnormality detection device 1 via the communication bus B1 such as a bus.

<2. Anomaly detection device>
As shown in FIG. 1, the abnormality detection device 1 includes an acquisition unit 11, a control unit 12, and a storage unit 13.

The acquisition unit 11 continuously acquires analog or digital captured images from the vehicle-mounted cameras 21 to 24 at a predetermined cycle (for example, a 1/30 second cycle). That is, the aggregate of the captured images acquired by the acquisition unit 11 is a moving image captured by the in-vehicle cameras 21 to 24. Then, when the acquired captured image is analog, the acquisition unit 11 converts the analog captured image into a digital captured image (A / D conversion). The acquisition unit 11 outputs the acquired photographed image or the acquired and converted photographed image to the control unit 12. One captured image output from the acquisition unit 11 becomes one frame image.

The control unit 12 is, for example, a microcomputer, and controls the entire abnormality detection device 1 in an integrated manner. The control unit 12 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). The storage unit 13 is, for example, a non-volatile memory such as a flash memory, and stores various types of information. The storage unit 13 stores a program as firmware and various data.

The control unit 12 includes an extraction unit 121, a specific unit 122, an estimation unit 123, and a determination unit 124. At least one of the extraction unit 121, the identification unit 122, the estimation unit 123, and the determination unit 124 of the control unit 12 is hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). It may be configured. Further, the extraction unit 121, the specific unit 122, the estimation unit 123, and the determination unit 124 are conceptual components. The functions executed by one component may be distributed to a plurality of components, or the functions of the plurality of components may be integrated into one component. Further, the acquisition unit 11 may be configured to be realized by the CPU of the control unit 12 performing arithmetic processing according to a program.

The extraction unit 121 extracts a plurality of feature points including the first feature point and the second feature point from the images taken from the in-vehicle cameras 21 to 24. The feature point is a point that can be conspicuously detected in the captured image, such as an intersection of edges in the captured image. The feature points are, for example, the edge of a white line drawn on the road surface, cracks on the road surface, stains on the road surface, gravel on the road surface, and the like. A large number of feature points are usually present in one captured image. The extraction unit 121 extracts feature points using, for example, a known method such as a Harris operator. The extraction unit 121 selects the first feature point and the second feature point from the feature points on the road surface. The first feature point and the second feature point are different feature points from each other. In this embodiment, a road surface is assumed as a predetermined plane.

The specific unit 122 derives each optical flow of the first feature point and the second feature point extracted from the two captured images taken at different times. The optical flow is a motion vector showing the movement of feature points between two captured images taken at different times. The interval between different times may be the same as the frame period of the acquisition unit 11, or may be a plurality of times the frame period of the acquisition unit 11.

The specific unit 122 uses the optical flows of the first feature point and the second feature point to change the positions of the first feature point and the second feature point extracted from the two captured images taken at different times. calculate.

Based on the change in the position of the first feature point and the change in the position of the second feature point, the identification unit 122 identifies two sets of faces whose lines of intersection with a predetermined plane are parallel to each other.

In some cases, only one of the first feature point and the second feature point can be extracted from a single captured image. In addition, the first feature point and the second feature point extracted from a single captured image may not have an appropriate positional relationship.

As an example in which only one of the first feature point and the second feature point can be extracted from a single captured image, the captured image shown in FIG. 3 and the captured image shown in FIG. 4 can be mentioned.

FIG. 3 schematically shows a photographed image taken by the front camera 21. The captured image shown in FIG. 3 includes a region BO in which the body of the vehicle 5 is reflected. The shaded area in the captured image shown in FIG. 3 shows a shadow, and the inside of the dotted square in the captured image shown in FIG. 3 is the ROI (region of interest) in which the feature points can be extracted. In the captured image shown in FIG. 3, only one feature point FP existing on the road surface RS can be extracted due to the influence of shadows.

FIG. 4 schematically shows a photographed image taken by the left side camera 23. The captured image shown in FIG. 4 includes a region BO in which the body of the vehicle 5 is reflected. The inside of the dotted square in the captured image shown in FIG. 4 is the ROI in which the feature points can be extracted. In the captured image shown in FIG. 4, since there is only one white line corner in the ROI, only one feature point FP existing on the road surface RS can be extracted.

As an example of the case where the first feature point and the second feature point extracted from a single captured image do not have an appropriate positional relationship, the captured image shown in FIG. 5 can be mentioned.

FIG. 5 schematically shows a photographed image taken by the front camera 21. The captured image shown in FIG. 5 includes a region BO in which the body of the vehicle 5 is reflected. The shaded areas in the captured image shown in FIG. 5 show shadows, and the inside of the dotted square in the captured image shown in FIG. 5 is the ROI in which the feature points can be extracted. In the captured image shown in FIG. 5, the feature point FP existing on the road surface RS can be extracted only at a local location due to the influence of the shadow. If the first feature point and the second feature point are unevenly distributed in local locations, the accuracy of camera posture estimation may be inferior in a part of the PAN axis, TILT axis, and ROLL axis.

When only one of the first feature point and the second feature point can be extracted from a single captured image, or the first feature point and the second feature point extracted from a single captured image have an appropriate positional relationship. If not, the extraction unit 121 extracts feature points from each of the two captured images taken at the first time and the second time after the first time, and extracts the feature points from the first time and the third time and the third time after the first time. Feature points may be extracted from each of the two captured images captured at the later fourth time. Then, the specific unit 122 selects the first feature points corresponding to each other from the feature points extracted from the two captured images taken at the first time and the second time, and calculates the position change of the first feature point. Then, from the feature points extracted from the two captured images taken at the 3rd time and the 4th time, unlike the 1st feature point, the 2nd feature points corresponding to each other are selected, and the positions of the 2nd feature points are selected. The change may be calculated, and based on the position change of the first feature point and the position change of the second feature point, two sets of faces whose intersection lines with a predetermined plane are parallel to each other may be specified.

The estimation unit 123 estimates the posture of the camera (camera mounting angle) based on the set of surfaces specified by the specific unit 122. For example, when the extraction unit 121 extracts the first feature point and the second feature point from the front camera 21, the estimation unit 123 estimates the posture (camera mounting angle) of the front camera 21.

The determination unit 124 determines whether or not the camera is in a misaligned state based on the posture of the camera estimated by the estimation unit 123. If it is determined that the camera is misaligned, the abnormality detection device 1 has detected an abnormality in the camera. For example, when the abnormality detection device 1 notifies the user of the vehicle 5 of this abnormality, the user of the vehicle 5 can take measures such as requesting the camera mounting adjustment by the dealer.

The posture estimation device 14 is composed of the acquisition unit 11, the extraction unit 121, the specific unit 122, and the estimation unit 123. In other words, the abnormality detection device 1 includes a posture estimation device 14. FIG. 6 is a flowchart showing an example of the posture estimation flow of the camera by the posture estimation device 14. Hereinafter, a case where the posture of the front camera 21 is estimated will be described as an example.

As shown in FIG. 6, the extraction unit 121 extracts feature points (step S1). More specifically, in step S1, the extraction unit 121 extracts the feature points of the current frame, which is the latest captured image, and the feature points of the captured image L (L is a natural number) before the current frame.

Next, the specific unit 122 derives the optical flow (step S2). More specifically, in step S2, the extraction unit 121 derives the optical flow of the feature points extracted from the latest captured image (current frame) and the captured image L (L is a natural number) frame before the current frame.

If no optical flow is derived in step S2, a frame one frame after the current frame is set as a new current frame, and the processes of steps S1 and S2 are executed for the new current frame.

The storage unit 13 stores the optical flow derived in step S2 as the latest optical flow. The latest optical flow changes to the past optical flow by updating the current frame. The storage unit 13 stores the past optical flow from the time when the latest optical flow changes to the past optical flow until a certain time elapses.

When the derivation of the optical flow in step S2 is completed, the specific unit 122 performs coordinate conversion on the feature points using the internal parameters of the front camera 21 stored in the storage unit 13. In the coordinate transformation, aberration correction and distortion correction of the front camera 21 are performed. Aberration correction is performed to correct distortion due to aberration in the optical system of the front camera 21. Specifically, it is the correction of distortion aberrations such as barrel distortion and pincushion distortion. The distortion correction is performed to correct the distortion of the optical system itself of the front camera 21. Specifically, it is fisheye correction. By the coordinate conversion, the coordinates of the feature points are converted into the coordinates obtained on the two-dimensional image when the subject is photographed by perspective projection.

When the coordinate transformation of the feature points described above is completed, the specific unit 122 determines whether or not the quadrangle can be virtually formed only by the latest optical flow (step S3).

When it is determined that the quadrangle can be virtually formed only by the latest optical flow (YES in step S3), the specific unit 122 virtually forms the quadrangle only by the latest optical flow (step S4). Hereinafter, for the sake of explanation, geometric elements such as a quadrangle and the sides of the quadrangle are virtually formed and used. However, in the actual processing, the processing may be based on vector operations such as the coordinates of feature points and the direction of a straight line, and may not be based on geometric elements having the same effect.

Here, a specific example in the case where the processing is executed in the order of steps S1, S2, S3, and S4 will be described with reference to FIGS. 7A, 7B, and 8. FIG. 7A is a diagram schematically showing a captured image before the L (L is a natural number) frame from the current frame. FIG. 7B is a diagram schematically showing the current frame. FIG. 8 is a diagram schematically showing a quadrangle formed virtually.

The photographed image P1 shown in FIG. 7A and the photographed image P2 shown in FIG. 7B are photographed by the front camera 21. The captured images P1 and P2 include a region BO in which the body of the vehicle 5 is reflected. The first feature point FP1 and the second feature point FP2 exist on the road surface RS. In FIGS. 7A and 7B, the first feature point FP1 and the second feature point FP2 are present at the corners of the white lines drawn on the road surface.

From the time when the photographed image P1 is photographed to the time when the photographed image P2 is photographed, the vehicle 5 is traveling straight ahead. The circled FP1P shown in FIG. 7B indicates the position of the first feature point FP1 at the time of shooting the captured image P1. The circled FP2P shown in FIG. 7B indicates the position of the second feature point FP2 at the time of shooting the captured image P1.

As shown in FIG. 7B, when the vehicle 5 goes straight forward, the first feature point FP1 and the second feature point FP2 existing in front of the vehicle 5 approach the vehicle 5. That is, the first feature point FP1 and the second feature point FP2 appear at different positions in the captured image P2 and the captured image P1. The identification unit 122 associates the first feature point FP1 of the captured image P2 with the first feature point FP1 of the captured image P1 based on the pixel values in the vicinity thereof, and associates the positions of the first feature points FP1 with each other. Based on, the optical flow OF1 of the first feature point FP1 is derived. Similarly, the specific unit 122 associates the second feature point FP2 of the captured image P2 with the second feature point FP2 of the captured image P1 based on the pixel values in the vicinity thereof, and associates the second feature point FP2 with the associated second feature point FP2. Based on each position, the optical flow OF2 of the second feature point FP2 is derived.

When the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2 are derived, the specific unit 122 sends an internal parameter to the front camera 21 stored in the storage unit 13 with respect to the feature point. Use to perform coordinate transformation. In the coordinate transformation, aberration correction and distortion correction of the front camera 21 are performed. Aberration correction is performed to correct distortion due to aberration in the optical system of the front camera 21. Specifically, it is the correction of distortion aberrations such as barrel distortion and pincushion distortion. The distortion correction is performed to correct the distortion of the optical system itself of the front camera 21. Specifically, it is fisheye correction. By the coordinate conversion, the coordinates of the feature points are converted into the coordinates obtained on the two-dimensional image when the subject is photographed by perspective projection.

When the coordinate conversion of the feature points is performed, as shown in FIG. 8, the specific unit 122 uses the start point of the optical flow OF1 of the first feature point FP1 as the apex SP1 and the end point of the optical flow OF1 of the first feature point FP1 as the apex. A square QL is virtually formed in which the start point of the optical flow OF2 of the EP1 and the second feature point FP1 is the apex SP2, and the end point of the optical flow OF2 of the second feature point FP2 is the apex EP.

Returning to FIG. 6, step S5 of the flowchart will be described. When it is determined that the quadrangle cannot be virtually formed only by the latest optical flow (NO in step S3), the specific unit 122 virtually forms the quadrangle using the past optical flow and the latest optical flow (step). S5).

Here, a specific example in the case where the processing is executed in the order of steps S1, S2, S3, and S5 will be described with reference to FIGS. 7C to 7F and FIG. FIG. 7C is a diagram schematically showing a captured image N (N is a natural number) frame before the current frame. FIG. 7D is a diagram schematically showing a captured image M (M is a natural number) frame before the current frame. FIG. 7E is a diagram schematically showing a captured image before the L (L is a natural number) frame from the current frame. FIG. 7F is a diagram schematically showing the current frame. In the present embodiment, N> M> L for the above N, M, and L, but M and L may be the same, and L may be larger than M and smaller than N.

The photographed image P11 shown in FIG. 7C is an example of the photographed image photographed at the first time. The captured image P12 shown in FIG. 7D is an example of a captured image captured at a second time after the first time. The photographed image P13 shown in FIG. 7E is an example of the photographed image photographed at the third time after the second time. The photographed image P14 shown in FIG. 7F is an example of the photographed image photographed at the fourth time after the third time.

The captured images P11 to P14 shown in FIGS. 7C to 7F are captured by the front camera 21. The captured images P11 to P14 include a region BO in which the body of the vehicle 5 is reflected. The first feature point FP1 and the second feature point FP2 exist on the road surface RS. In FIGS. 7C and 7D, the first feature point FP1 exists at the corner of the white line drawn on the road surface. In FIGS. 7E and 7F, the second feature point FP2 exists at the corner of the white line drawn on the road surface.

From the time when the photographed image P11 is photographed to the time when the photographed image P12 is photographed, the vehicle 5 is traveling straight ahead. The circled FP1P shown in FIG. 7D indicates the position of the first feature point FP1 at the time of shooting the captured image P11. Similarly, between the time when the photographed image P13 is photographed and the time when the photographed image P14 is photographed, the vehicle 5 is traveling straight ahead. The circled FP2P shown in FIG. 7F indicates the position of the second feature point FP2 at the time of shooting the captured image P13.

As shown in FIG. 7D, when the vehicle 5 goes straight forward, the first feature point FP1 existing in front of the vehicle 5 approaches the vehicle 5. That is, the first feature point FP1 appears at different positions in the captured image P12 and the captured image P11. The identification unit 122 associates the first feature point FP1 of the captured image P12 with the first feature point FP1 of the captured image P11 based on the pixel values in the vicinity thereof, and associates the positions of the first feature points FP1 with each other. Based on, the optical flow OF1 of the first feature point FP1 is derived.

As shown in FIG. 7F, when the vehicle 5 goes straight forward, the second feature point FP2 existing in front of the vehicle 5 approaches the vehicle 5. That is, the second feature point FP2 appears at different positions in the captured image P14 and the captured image P13. The identification unit 122 associates the second feature point FP2 of the captured image P14 with the second feature point FP2 of the captured image P13 based on the pixel values in the vicinity thereof, and associates the positions of the second feature points FP2 with each other. Based on, the optical flow OF2 of the second feature point FP2 is derived.

When the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2 are derived, the specific unit 122 sends an internal parameter to the front camera 21 stored in the storage unit 13 with respect to the feature point. Use to perform coordinate transformation. In the coordinate transformation, aberration correction and distortion correction of the front camera 21 are performed. Aberration correction is performed to correct distortion due to aberration in the optical system of the front camera 21. Specifically, it is the correction of distortion aberrations such as barrel distortion and pincushion distortion. The distortion correction is performed to correct the distortion of the optical system itself of the front camera 21. Specifically, it is fisheye correction. By the coordinate conversion, the coordinates of the feature points are converted into the coordinates obtained on the two-dimensional image when the subject is photographed by perspective projection.

When the coordinate conversion of the feature points is performed, as shown in FIG. 8, the specific unit 122 uses the start point of the optical flow OF1 of the first feature point FP1 as the apex SP1 and the end point of the optical flow OF1 of the first feature point FP1 as the apex. A square QL is virtually formed in which the start point of the optical flow OF2 of the EP1 and the second feature point FP1 is the apex SP2, and the end point of the optical flow OF2 of the second feature point FP2 is the apex EP.

Returning to FIG. 6, the processing after step S5 of the flowchart will be described.

When the quadrangle QL is virtually formed, the specific unit 122 uses the quadrangle QL and the internal parameters of the front camera 21 stored in the storage unit 13 to display the projection surface IMG of the front camera 21 in the three-dimensional space. The quadrangle QL is moved upward to virtually generate the quadrangle QL1 on the projection plane IMG (step S6).

For the sake of explanation, the sides of the quadrangle QL1 are defined as follows (see FIG. 9). The first side SD1 of the quadrangle QL1 corresponds to the side connecting the vertices SP1 and the vertices SP2 in the quadrangle QL. That is, it corresponds to the side connecting the start point of the optical flow OF1 of the first feature point FP1 and the start point of the optical flow OF2 of the second feature point FP2. Similarly, the second side SD2 of the quadrangle QL1 corresponds to the side connecting the vertices SP2 and the vertices EP2 in the quadrangle QL. That is, it corresponds to the optical flow OF2 of the second feature point FP2. Similarly, the third side SD3 of the quadrangle QL1 corresponds to the side connecting the vertices EP1 and EP2 in the quadrangle QL. That is, it corresponds to the side connecting the end point of the optical flow OF1 of the first feature point FP1 and the end point of the optical flow OF2 of the second feature point FP2. Similarly, the fourth side SD4 of the quadrangle QL1 corresponds to the side connecting the vertices SP1 and the vertices EP1 in the quadrangle QL. That is, it corresponds to the optical flow OF1 of the first feature point FP1.

In addition, the surface is defined as follows (see FIG. 9). The surface including the first side SD1 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as the first surface F1. Similarly, the surface including the second side SD2 of the quadrangle QL1 and the optical center OC of the front camera 21 is referred to as the second surface F2. Similarly, the surface including the third side SD3 of the quadrangle QL1 and the optical center OC of the front camera 21 is referred to as the third surface F3. Similarly, the surface including the fourth side SD4 of the quadrangle QL1 and the optical center OC of the front camera 21 is referred to as the fourth surface F4.

In step S7 following step S6, the identification unit 122 identifies two sets of faces whose lines of intersection with a predetermined plane are parallel to each other. A predetermined plane is a plane whose normal is known in advance. Specifically, it is a plane on which the vehicle 5 is moving, that is, a road surface. The predetermined plane does not have to be a strict plane, as long as the specific portion 122 can be regarded as a plane when specifying two sets of planes whose intersection lines with the predetermined plane are parallel to each other. Good.

In the present embodiment, the first feature point and the second feature point are extracted from a stationary object located on a predetermined plane such as a road surface. Therefore, each optical flow of the first feature point and the second feature point derived by the specific unit 122 represents a relative position change of the stationary object with respect to the vehicle 5 in the real world. That is, it is a movement vector of the vehicle 5 having the opposite direction.

Since the second side SD2 and the fourth side SD4 of the quadrangle QL1 both correspond to the optical flow of the feature points, they both correspond to the movement vector of the vehicle 5 in the real world. Therefore, it is assumed that they are parallel to each other on the road surface.

Further, since the first side SD1 and the third side SD3 of the quadrangle QL1 are both in the positional relationship between the feature points, they correspond to the positional relationship between the stationary objects accompanying the movement of the vehicle 5 in the real world. The positional relationship before the movement corresponds to the first side SD1, and the positional relationship after the movement corresponds to the third side SD3. At this time, since the position of the stationary object does not change, the positional relationship does not change before and after the movement. Therefore, it is assumed that they are also parallel to each other on the road surface.

Therefore, the specific portion 122 has two sets of a pair of the second surface F2 and the fourth surface F4 and a pair of the first surface F1 and the third surface F3 as planes whose intersection lines with the road surface are parallel. To identify. That is, the specific unit 122 specifies a total of two sets, with the faces including the optical flow as one set and the faces including the feature points photographed at the same time as other sets.

In addition, in FIG. 9, when the square QL2 is virtually formed by using the optical flow obtained by the photographed image P1 shown in FIG. 7A and the photographed image P2 shown in FIG. 7B, the first feature at the time of photographing the photographed image P2 is The position of the point FP1 in the three-dimensional space (real world), the position of the second feature point FP2 at the time of shooting the captured image P2 in the three-dimensional space, and the position of the first feature point FP1 at the time of shooting the captured image P1. It is a quadrangle whose apex is the position in the three-dimensional space and the position in the three-dimensional space of the second feature point FP2 at the time of shooting the captured image P1.

On the other hand, in FIG. 9, the square QL2 is virtual using the optical flow obtained from the captured image P11 shown in FIG. 7C, the captured image P12 shown in FIG. 7D, the captured image P13 shown in FIG. 7E, and the captured image P14 shown in FIG. 7F. On the three-dimensional space (real world) of the first feature point FP1 at the time of shooting the captured image P12, and on the three-dimensional space of the second feature point FP2 at the time of capturing the captured image P14. A square whose apex is the position, the position of the first feature point FP1 at the time of shooting the captured image P11 in the three-dimensional space, and the position of the second feature point FP2 at the time of shooting the captured image P13 in the three-dimensional space. is there.

The first surface F1 includes the first side SD11 of the quadrangle QL2. Similarly, the second surface F2 includes the second side SD12 of the quadrangle QL2, the third surface F3 includes the third side SD13 of the quadrangle QL2, and the fourth surface F4 includes the fourth side SD14 of the quadrangle QL2. At this time, as described above, the quadrangle QL2 is assumed to be a parallelogram formed on the road surface.

Next, the estimation unit 123 calculates the normal of the road surface (step S8). First, the estimation unit 123 obtains the direction of the line of intersection between the surfaces based on the first surface F1 and the third surface F3, which are one of the set of surfaces specified by the specific unit 122. Specifically, the direction of the line of intersection CL1 between the first surface F1 and the third surface F3 is obtained (see FIG. 10). The direction vector V1 of the line of intersection CL1 is a vector perpendicular to each of the normal vector of the first surface F1 and the normal vector of the third surface F3. Therefore, the estimation unit 123 obtains the direction vector V1 of the line of intersection CL1 by the outer product of the normal vector of the first surface F1 and the normal vector of the third surface F3. Since the line of intersection of the first surface F1 and the third surface F3 is parallel to the road surface, the direction vector V1 is parallel to the road surface.

Similarly, the estimation unit 123 obtains the direction of the line of intersection between the second surface F2 and the fourth surface F4, which is the other side of the set of surfaces specified by the specific unit 122. Specifically, the direction of the line of intersection CL2 between the second surface F2 and the fourth surface F4 is obtained (see FIG. 11). The direction vector V2 of the line of intersection CL2 is a vector perpendicular to each of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Therefore, the estimation unit 123 obtains the direction vector V2 of the line of intersection CL2 by the outer product of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Similarly, the second surface F2 and the fourth surface F4 have parallel lines of intersection with the road surface, so that the direction vector V2 is parallel to the road surface.

The estimation unit 123 calculates the normal of the surface of the quadrangle QL2, that is, the normal of the road surface by the outer product of the direction vector V1 and the direction vector V2. Since the normal of the road surface calculated by the estimation unit 123 is calculated by the camera coordinate system of the front camera 21, the coordinate system of the three-dimensional space is obtained from the difference from the vertical direction which is the normal of the actual road surface, and the front with respect to the road surface. The posture of the camera 21 can be estimated. From the estimation result, the estimation unit 123 estimates the posture of the front camera 21 with respect to the vehicle 5 (step S9). The calculation process in step S7 can be executed using, for example, a known ARToolkit.

When the posture estimation of the front camera 21 in step S9 is completed, the flow shown in FIG. 6 is completed.

The attitude estimation device 14 autonomously identifies two sets of faces whose intersection lines with a predetermined plane are parallel to each other by using the movement of the vehicle 5, and only the extraction error of the feature points is estimated accuracy. It is possible to estimate the attitude of the camera that affects the image. That is, the posture estimation device 14 can estimate the posture of the camera with few error factors. Therefore, the posture estimation device 14 can accurately estimate the posture of the camera.

Further, as described above, the posture estimation device 14 virtually forms a quadrangle by using the past optical flow and the latest optical flow when the quadrangle cannot be virtually formed only by the latest optical flow. Therefore, the posture estimation device 14 is estimated as compared with a configuration in which a quadrangle cannot be virtually formed only by the latest optical flow, that is, a configuration in which steps S3 and S5 are removed from the flowchart shown in FIG. It is possible to suppress the prolongation of the time required for.

However, when a quadrangle is virtually formed using the past optical flow and the latest optical flow, compared to the case where the quadrangle is virtually formed only by the latest optical flow without using an appropriate optical flow. The accuracy of camera posture estimation may be significantly reduced.

Whether the optical flow used in the process of step S5 shown in FIG. 6 is appropriate depends on the movement condition of the vehicle 5. Therefore, when executing the process of step S5 shown in FIG. 6, it is desirable that the combination of the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point is determined based on the movement information of the vehicle 5. ..

As described above, the posture estimation device 14 estimates the posture of the camera on the assumption that the quadrangle QL2 is a parallelogram formed on the road surface.

The smaller the difference between the steering angle of the vehicle 5 when the optical flow OP1 of the first feature point occurs and the steering angle of the vehicle 5 when the optical flow OP2 of the second feature point occurs, the more the above assumption and reality The divergence becomes smaller. Therefore, when the rudder angle information of the vehicle 5 is included in the movement information of the vehicle 5 and the process of step S5 shown in FIG. 6 is executed, the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point It is desirable that the combination of the above is determined based on the steering angle information of the vehicle 5. Hereinafter, this combination determination method is referred to as a first determination method. For example, if the output of the steering angle sensor 3 supplied from the steering angle sensor 3 to the attitude estimation device 14 via the communication bus B1 is used as the steering angle information of the vehicle 5, the steering angle information of the vehicle 5 is the estimation of the steering angle. Since the actual measurement information of the steering angle is used instead of the information, the accuracy of the steering angle information of the vehicle 5 can be improved.

For example, the first feature is such that the difference between the steering angle of the vehicle 5 at the time of photographing the photographed image P12 shown in FIG. 7D and the steering angle of the vehicle 5 at the time of photographing the photographed image P14 shown in FIG. 7F is within a predetermined angle. The combination of the point optical flow OP1 and the second feature point optical flow OP2 may be determined. Further, for example, the average value of the steering angles of the vehicle 5 from the time of shooting the photographed image P11 shown in FIG. 7C to the time of photographing the photographed image P12 shown in FIG. 7D and the time of photographing the photographed image P13 shown in FIG. 7E are shown in FIG. 7F. The combination of the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point is set so that the difference from the average value of the steering angles of the vehicle 5 up to the time of shooting the photographed image P14 is within a predetermined angle. You may decide.

The smaller the difference between the speed of the vehicle 5 when the optical flow OP1 of the first feature point occurs and the speed of the vehicle 5 when the optical flow OP2 of the second feature point occurs, the more the difference between the above assumption and the reality is. Becomes smaller. Therefore, when the speed information of the vehicle 5 is included in the movement information of the vehicle 5 and the process of step S5 shown in FIG. 6 is executed, the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point The combination is preferably determined based on the speed information of the vehicle 5. Hereinafter, this combination determination method is referred to as a second determination method. For example, when the output of the vehicle speed sensor 4 supplied from the vehicle speed sensor 4 to the attitude estimation device 14 via the communication bus B1 is used as the speed information of the vehicle 5, the speed information of the vehicle 5 is not the speed estimation information. Since the actual speed information is used, the accuracy of the speed information of the vehicle 5 can be improved.

For example, the first feature point is set so that the difference between the speed of the vehicle 5 at the time of shooting the captured image P12 shown in FIG. 7D and the speed of the vehicle 5 at the time of capturing the captured image P14 shown in FIG. 7F is within a predetermined speed. The combination of the optical flow OP1 and the optical flow OP2 of the second feature point may be determined. Further, for example, the average value of the speeds of the vehicle 5 from the time when the photographed image P11 shown in FIG. 7C is photographed to the time when the photographed image P12 shown in FIG. 7D is photographed and the time when the photographed image P13 shown in FIG. 7E is photographed are shown in FIG. 7F. The combination of the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point is determined so that the difference from the average value of the speeds of the vehicle 5 up to the time of shooting the captured image P14 is within a predetermined speed. You may.

If the time difference between the time when the optical flow OP1 of the first feature point is generated and the time when the optical flow OP1 of the second feature point is generated is short, it can be expected that the change in the movement condition of the vehicle 5 is small. Therefore, when the process of step S5 shown in FIG. 6 is executed, the combination of the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point is the time of shooting of the captured image P11 shown in FIG. 7C and FIG. 7D. Determined based on the time difference between the time related to at least one of the shooting time of the captured image P12 shown in FIG. 7E and the time related to at least one of the shooting time of the captured image P13 shown in FIG. 7E and the shooting time of the captured image P14 shown in FIG. 7F. It is desirable to be done. Hereinafter, this combination determination method will be referred to as a third determination method.

As the time related to at least one of the shooting time of the captured image P11 shown in FIG. 7C and the shooting time of the captured image P12 shown in FIG. 7D, for example, the time at the time of capturing the captured image P11 shown in FIG. 7C is shown in FIG. 7D. Examples include the time at the time of shooting the captured image P12, the time at the midpoint between the time of shooting the captured image P11 shown in FIG. 7C and the time at which the captured image P12 is photographed shown in FIG. 7D. Similarly, as the time related to at least one of the shooting time of the captured image P13 shown in FIG. 7E and the capturing time of the captured image P14 shown in FIG. 7F, for example, the time at the time of capturing the captured image P13 shown in FIG. 7E, FIG. Examples include the time at which the photographed image P14 shown on the 7th floor is photographed, the time at the midpoint between the time when the photographed image P13 shown in FIG. 7E is photographed and the time when the photographed image P14 shown on the 7th floor is photographed.

The first determination method, the second determination method, and the third determination method described above may be performed individually or in combination of any two or more.

Further, since each optical flow of the first feature point and the second feature point is a movement vector of the vehicle 5 having the opposite directions, the above-mentioned movement information is used for each of the first feature point and the second feature point optical. The information may be estimated based on the result of coordinate conversion of the flow on the road surface. The speed of the vehicle 5 can be estimated by converting the coordinates of each optical flow of the first feature point and the second feature point onto the road surface and calculating the length on the road surface, and by calculating the inclination on the road surface. The steering angle of the vehicle 5 can be estimated. That is, it can be implemented as a substitute for the above-mentioned first determination method and second determination method. When the above-mentioned movement information is used as information estimated based on the result of coordinate conversion of each optical flow of the first feature point and the second feature point on the road surface, it is not necessary to connect the attitude estimation device 14 and the communication bus B1. become.

Further, the specific unit 122 is the speed of the vehicle 5 when the optical flow OP1 of the first feature point is generated (the measured value detected by the vehicle speed sensor 4 or the estimated speed described above may be used). Based on the difference between the speed of the vehicle 5 when the optical flow OP2 of the second feature point is generated (the measured value detected by the vehicle speed sensor 4 or the estimated speed described above). At least one of the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point is corrected, and at least one of them intersects with the road surface based on the corrected optical flows of the first feature point and the second feature point. It is desirable to identify two sets of faces whose lines are parallel to each other. As a result, the accuracy of camera posture estimation can be improved.

For example, when the speed of the vehicle 5 when the optical flow OP1 of the first feature point is generated is larger than the speed of the vehicle 5 when the optical flow OP2 of the second feature point is generated, the optical flow OP1 of the first feature point is It may be corrected so that it becomes shorter, and it may be corrected so that the optical flow OP2 of the second feature point becomes longer, the optical flow OP1 of the first feature point becomes shorter, and the optical flow OP2 of the second feature point becomes longer. It may be corrected as follows.

For example, when the road surface RS is made of concrete, it is difficult to extract feature points. Therefore, the extraction error of the feature points tends to be large. However, for example, when dirt is present on the road surface RS made of concrete, when the first feature point FP1 and the second feature point FP2 are extracted from the dirty portion, the feature points having a high degree of feature are the first feature point FP1 and the first feature point FP1. It will be selected as the second feature point FP2, and it is considered that the extraction error of the first feature point FP1 and the second feature point FP2 can be suppressed.

Not limited to dirt, when there is a high-density region where feature points are more concentrated than others, by extracting the first feature point FP1 and the second feature point FP2 from the high-density region, It is considered that the extraction error of the first feature point FP1 and the second feature point FP2 can be suppressed. Therefore, when there is a high-density region in which the feature points are more concentrated than others, even if the first feature point FP1 and the second feature point FP2 are extracted from the high-density region by the extraction unit 121. Good.

Further, in the extraction of feature points having a high degree of cornering, the extraction error becomes small. Therefore, the first feature point and the second feature point may be designated as specific feature points whose corner degree indicating corneriness is equal to or higher than a predetermined corner degree threshold value. A corner is the intersection of two edges. At the corners, the degree of corners is high. The degree of cornering can be determined by using a known detection method such as a Harris operator or a KLT (Kanade-Lucas-Tomasi) tracker.

In the above description, the optical center OC of the front camera 21 and the plane including one side of the quadrangle QL1 are specified, but this is not the case. The plane may be specified by specifying the normal direction of the plane. For example, the normal direction of the plane may be obtained from the outer product of the direction vectors from the optical center OC to each vertex, and the surface may be specified by the normal direction. In this case, the normal direction of the first surface F1 and the normal direction of the third surface F3 are regarded as one set, and the normal direction of the second surface F2 and the normal direction of the fourth surface F4 are regarded as another set. It is good to identify two pairs.

Moreover, the surface may be translated. For example, instead of the first surface F1, the surface obtained by translating the first surface F1 may be paired with the third surface F3. This is because the direction of the line of intersection with a predetermined plane does not change even if the translation is performed.

<3. Correction system>
FIG. 12 is a block diagram showing a configuration of the correction system SYS2 according to the embodiment. In FIG. 12, the same parts as those in FIG. 1 are designated by the same reference numerals, and detailed description thereof will be omitted. As shown in FIG. 12, the correction system SYS2 includes a correction device 1'and a photographing unit 2.

The correction device 1'is the same as the configuration in which the determination unit 124 is removed from the abnormality detection device 1 shown in FIG. 1 and the correction unit 125 is added. That is, the correction device 1'includes a posture estimation device 14 and a correction unit 125.

The correction unit 125 corrects the camera parameters based on the camera posture estimated by the estimation unit 123. As a result, even if the camera mounting is misaligned, the camera is calibrated according to the misalignment, and the adverse effect of the camera mounting misalignment on the captured image can be suppressed.

<4. Notes>
The configurations of the embodiments and examples in the present specification are merely examples of the present invention. The configurations of the embodiments and modifications may be appropriately changed without exceeding the technical idea of the present invention. Moreover, a plurality of embodiments and modifications may be carried out in combination to the extent possible.

The present invention can be used to estimate the posture of a camera that assists driving a moving object such as automatic parking, as in the above-described embodiment. Further, the present invention can be used to estimate the posture of a camera that records driving information of a drive recorder or the like. Further, the present invention can be used as a correction device or the like for correcting a captured image by using the estimation information of the posture of the camera as in the above-described embodiment. Further, the present invention can be used for a device or the like that operates in cooperation with a center provided so as to be able to communicate with a plurality of mobile bodies via a network. For example, when transmitting a photographed image to the center, the device may be configured to transmit the abnormality information of the camera and the estimation information of the posture as a set with the photographed image. Then, at the center, various image processing (processing for changing the viewpoint / viewing direction of the image in consideration of the camera posture, for example, the viewpoint / viewing direction is changed to an image in the front direction of the vehicle body) using the estimation information of the camera posture. (Generate an image, etc.), correct the camera posture in the measurement process using the image, statistically process the secular change of the camera posture (data of many vehicles), etc. to generate useful data to be provided to the user. And so on.

1 Anomaly detection device 1'Correction device 14 Posture estimation device 21 to 24 On-board camera 11 Acquisition unit 121 Extraction unit 122 Specific unit 123 Estimate unit 124 Judgment unit 125 Correction unit

Claims (11)

An acquisition unit that acquires captured images taken by a camera mounted on a moving body,
Feature points are extracted from each of the two captured images taken at the first time and the second time after the first time, and the third time after the first time and the third time after the third time. An extraction unit that extracts feature points from each of the two captured images taken at 4 hours,
From the feature points extracted from the two captured images taken at the first time and the second time, the first feature points corresponding to each other are selected, and the position change of the first feature points is calculated.
From the feature points extracted from the two captured images taken at the third time and the fourth time, different from the first feature point, the second feature points corresponding to each other are selected, and the second feature is selected. Calculate the position change of the point,
Based on the change in the position of the first feature point and the change in the position of the second feature point, a specific portion that specifies two sets of faces whose intersection lines with a predetermined plane are parallel to each other,
An estimation unit that estimates the posture of the camera based on the set of surfaces specified by the specific unit, and an estimation unit.
A posture estimation device. The posture estimation device according to claim 1, wherein the combination of the position change of the first feature point and the position change of the second feature point is determined based on the movement information of the moving body. The posture estimation device according to claim 2, wherein the movement information includes steering angle information of the moving body. The posture estimation device according to claim 2 or 3, wherein the movement information includes speed information of the moving body. The posture estimation device according to any one of claims 2 to 4, wherein the movement information is supplied to the posture estimation device via a communication bus. The movement information is the result of coordinate-converting the position change of the candidate of the first feature point on the predetermined plane and the result of coordinate-converting the position change of the candidate of the second feature point on the predetermined plane. The posture estimation device according to claim 2 to 4, which is information estimated based on the above. The combination of the position change of the first feature point and the position change of the second feature point is a time related to at least one of the first time and the second time, and at least the third time and the fourth time. The posture estimation device according to any one of claims 1 to 6, which is determined based on a time difference from a time related to one of them. The specific unit of the first feature point is based on the difference between the speed of the moving body from the first time to the second time and the speed of the moving body from the third time to the fourth time. At least one of the position change and the position change of the second feature point is corrected, and based on the position change of the first feature point and the position change of the second feature point in which at least one is corrected, the position change with the predetermined plane The posture estimation device according to any one of claims 1 to 7, wherein two sets of planes whose intersection lines are parallel to each other are specified. The posture estimation device according to any one of claims 1 to 8, and the posture estimation device.
Based on the posture of the camera estimated by the estimation unit, a determination unit for determining whether or not the mounting of the camera is misaligned, and a determination unit.
Anomaly detection device. The posture estimation device according to any one of claims 1 to 8, and the posture estimation device.
A correction unit that corrects the parameters of the camera based on the posture of the camera estimated by the estimation unit, and a correction unit.
A correction device. The acquisition process to acquire the captured image taken by the camera mounted on the moving body, and
Feature points are extracted from each of the two captured images taken at the first time and the second time after the first time, and the third time after the first time and the third time after the third time. An extraction step of extracting feature points from each of the two captured images taken at 4 hours, and
From the feature points extracted from the two captured images taken at the first time and the second time, the first feature points corresponding to each other are selected, the position change of the first feature points is calculated, and the above. From the feature points extracted from the two captured images taken at the third time and the fourth time, different from the first feature point, the second feature points corresponding to each other are selected, and the second feature point is selected. Based on the position change of the first feature point and the position change of the second feature point, a specific step of specifying two sets of planes whose intersection lines with a predetermined plane are parallel to each other. ,
An estimation process that estimates the posture of the camera based on the set of surfaces specified in the specific process, and an estimation process.
A posture estimation method.

Images