JP7256734B2 - Posture estimation device, anomaly detection device, correction device, and posture estimation method - Google Patents

Description

The present invention relates to a posture estimation device for estimating the posture of a camera, an anomaly detection device for detecting anomalies in a camera, a correction device for correcting camera parameters, and a posture estimation method for estimating the posture of a camera.

2. Description of the Related Art Conventionally, vehicle-mounted cameras are used to provide driving assistance such as parking assistance for vehicles. An on-board camera is fixedly attached to a vehicle before the vehicle leaves the factory. However, the on-vehicle camera may be displaced from the mounting state at the time of shipment from the factory due to, for example, accidental contact or aging. If the position of the in-vehicle camera (relative position of the in-vehicle camera to the vehicle body) shifts, an error will occur in the amount of steering wheel steering determined using the camera image. is important.

Patent Literature 1 discloses a technique for detecting optical axis deviation of a vehicle-mounted camera. The optical axis deviation detection device for an on-vehicle camera in Patent Document 1 includes image processing means and determination means. The image processing means detects the position information of the marking from the captured image of the vehicle-mounted camera that captures the range including the marking on the vehicle body for driving assistance. The judging means judges the misalignment of the photographing optical axis by comparing the initially set marking position information and the newly detected marking position information.

JP 2004-173037 A

Since the captured image of the on-vehicle camera shows the scenery around the vehicle, for example, when the marking is a specific shape of a part of the hood, the specific shape may not be easily extracted. If the specific shape is erroneously detected, there is a possibility that the misalignment of the optical axis of the in-vehicle camera may not be accurately determined. It should be noted that the deviation of the optical axis of the vehicle-mounted camera means that the posture of the vehicle-mounted camera is not as designed.

SUMMARY OF THE INVENTION In view of the above problems, it is an object of the present invention to provide a technology capable of estimating the pose of a camera with high accuracy and suppressing a decrease in accuracy of pose estimation that depends on the position of the optical flow used for pose estimation. and

A posture estimation apparatus according to the present invention includes an acquisition unit that acquires a captured image captured by a camera mounted on a mobile object, and extracts a plurality of feature points including a first feature point and a second feature point from the captured image. a flow derivation unit for deriving an optical flow for each of the plurality of feature points; and based on the position of the optical flow derived by the flow derivation unit, the optical flow derived by the flow derivation unit a selection unit that selects the optical flow of the first feature point and the optical flow of the second feature point from the optical flow of the first feature point and the optical flow of the second feature point selected by the selection unit; and an estimating unit for estimating the orientation of the camera based on the sets of planes specified by the specifying unit. It is a configuration (first configuration).

In the posture estimation device having the first configuration, the selection unit determines that the distance in a predetermined direction between the real-world position of the first feature point and the real-world position of the second feature point is equal to or greater than a predetermined value. The configuration (second configuration) may be such that the optical flow of the first feature point and the optical flow of the second feature point are selected so that

In the posture estimation device having the second configuration, the selection unit selects the position of the first feature point in the real world and the position of the second feature point in the real world in a direction perpendicular to the predetermined direction. The predetermined value may be increased as the distance is increased (third configuration).

In the posture estimation device having the second or third configuration, the selection unit selects an optical flow of the first feature point and an optical flow of the second feature point located within a predetermined distance from the center of the optical axis of the camera. A configuration (fourth configuration) for selecting a flow may be used.

In the posture estimation device having the fourth configuration, the camera may be configured to view the front or rear of the moving object (fifth configuration).

In the posture estimation apparatus having the second or third configuration, the selection unit selects the optical flow of the first feature points located within a predetermined distance from the center of the optical axis of the camera and the optical flow from the center of the optical axis of the camera. The configuration (sixth configuration) may be such that the optical flow of the second feature point located in the area away from the predetermined distance is selected.

In the posture estimation device having the sixth configuration, the camera may be configured to view the left or right side of the moving object (seventh configuration).

An anomaly detection apparatus according to the present invention provides a posture estimation device having any one of the first to seventh configurations described above, and a posture estimation device, based on the posture of the camera estimated by the estimation unit, in a state in which the mounting of the camera is deviated. and a determination unit for determining whether or not there is a configuration (eighth configuration).

A correction device according to the present invention includes: a posture estimation device having any one of the first to seventh configurations; and a correction unit that corrects parameters of the camera based on the camera posture estimated by the estimation unit. It is a configuration (ninth configuration) provided.

A posture estimation method according to the present invention includes an acquisition step of acquiring a photographed image taken by a camera mounted on a mobile object, and extracting a plurality of feature points including a first feature point and a second feature point from the photographed image. a flow derivation step of deriving an optical flow for each of the plurality of feature points; and based on the position of the optical flow derived in the flow derivation step, the optical flow derived in the flow derivation step a selection step of selecting the optical flow of the first feature point and the optical flow of the second feature point from the optical flow of the first feature point and the optical flow of the second feature point selected in the selection step; and an estimating step of estimating the posture of the camera based on the pairs of planes identified in the identifying step. This is the configuration (tenth configuration).

According to the present invention, it is possible to accurately estimate the pose of a camera, and to prevent the accuracy of pose estimation from deteriorating depending on the position of the optical flow used for pose estimation.

Block diagram showing the configuration of the anomaly detection system Diagram exemplifying the position where the on-board camera is arranged in the vehicle Flowchart showing an example of a camera posture estimation flow by a posture estimation device Diagram for explaining the method of extracting feature points A diagram for explaining a technique for deriving an optical flow of feature points. A diagram showing a rectangle formed virtually from optical flow FIG. 4 is a diagram for explaining a technique for identifying two sets of planes whose intersection lines with a predetermined plane are parallel to each other; Diagram for explaining the method of determining the direction of the line of intersection between the faces based on one of the identified pairs of faces. Diagram for explaining the method of determining the direction of the line of intersection between the faces based on the other of the identified faces Graph showing the error characteristics when the posture estimation device estimates the posture of the front camera using the captured image of the front camera Graph showing error characteristics when the posture estimation device estimates the posture of the right-side camera using the captured image of the right-side camera Graph showing error characteristics when the posture estimation device estimates the posture of the right-side camera using the captured image of the right-side camera Graph showing error characteristics when the posture estimation device estimates the posture of the right-side camera using the captured image of the right-side camera Graph showing error characteristics when the posture estimation device estimates the posture of the right-side camera using the captured image of the right-side camera Graph showing error characteristics when the posture estimation device estimates the posture of the right-side camera using the captured image of the right-side camera Block diagram showing the configuration of the correction system

Exemplary embodiments of the invention are described in detail below with reference to the drawings. In the following description, a vehicle is taken as an example of a moving object, but the moving object is not limited to a vehicle. Vehicles broadly include vehicles with wheels, such as automobiles, trains, and automatic guided vehicles. Examples of mobile objects other than vehicles include ships and aircraft.

Further, in the following description, the direction in which the vehicle travels straight ahead, which is the direction from the driver's seat to the steering wheel, is referred to as the "forward direction." In addition, the direction from the steering wheel to the driver's seat, which is the direction in which the vehicle travels straight, is defined as the "rearward direction." In addition, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and from the right side to the left side of the driver who is facing forward is referred to as the "left direction." Also, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and from the left side to the right side of the driver who is facing forward is referred to as the "right direction."

<1. Anomaly detection system>
FIG. 1 is a block diagram showing the configuration of an abnormality detection system SYS1 according to the embodiment. In this embodiment, the abnormality is a state in which the mounting of the camera is misaligned. That is, the abnormality detection system SYS1 is a system that detects misalignment of a camera mounted on a vehicle (in-vehicle camera). As shown in FIG. 1 , an abnormality detection system SYS1 includes an abnormality detection device 1 and an imaging unit 2 .

The abnormality detection device 1 is a device that detects an abnormality based on an image captured by an in-vehicle camera. The abnormality detection device 1 is provided for each vehicle equipped with an in-vehicle camera. In this embodiment, the abnormality detection device 1 acquires a captured image from the imaging unit 2 .

The photographing unit 2 is provided for the purpose of monitoring the situation around the vehicle. The imaging unit 2 includes four cameras 21-24. The four cameras 21-24 are in-vehicle cameras. FIG. 2 is a diagram illustrating positions where the four vehicle-mounted cameras 21 to 24 are arranged on the vehicle 4. As shown in FIG.

An in-vehicle camera 21 is provided at the front end of the vehicle 4 . For this reason, the vehicle-mounted camera 21 is also called a front camera 21 . An optical axis 21a of the front camera 21 extends along the longitudinal direction of the vehicle 4 in plan view from above. The front camera 21 is a camera that faces the front of the vehicle 4 and photographs the front of the vehicle 4 . An in-vehicle camera 22 is provided at the rear end of the vehicle 4 . Therefore, the vehicle-mounted camera 22 is also called a back camera 22 . An optical axis 22a of the back camera 22 extends along the longitudinal direction of the vehicle 4 when viewed from above. The back camera 22 is a camera that looks behind the vehicle 4 and photographs the rear direction of the vehicle 4 . The mounting position of the front camera 21 and the back camera 22 is preferably the center in the left and right direction of the vehicle 4, but may be slightly shifted in the left and right direction from the center in the left and right direction.

The vehicle-mounted camera 23 is provided on the left side door mirror 41 of the vehicle 4 . Therefore, the in-vehicle camera 23 is also called a left side camera 23 . The optical axis 23a of the left side camera 23 extends along the left-right direction of the vehicle 4 in plan view from above. The left side camera 23 is a camera that looks to the left of the vehicle 4 and captures the left direction of the vehicle 4 . The vehicle-mounted camera 24 is provided on the right side door mirror 42 of the vehicle 4 . Therefore, the in-vehicle camera 24 is also called a right side camera 24 . An optical axis 24a of the right side camera 24 extends along the left-right direction of the vehicle 4 in plan view from above. The right side camera 24 is a camera that faces the right side of the vehicle 4 and photographs the right side of the vehicle 4 . When the vehicle 4 is a so-called door mirrorless vehicle, the left side camera 23 is attached around the rotation axis (hinge portion) of the left side door without interposing the door mirror, and the right side camera 24 is attached to the right side door. It can be attached around the rotating shaft (hinge part) without interposing the door mirror.

The horizontal angle of view θ of each vehicle-mounted camera 21 to 24 is 180 degrees or more. Therefore, the vehicle-mounted cameras 21 to 24 can photograph the entire periphery of the vehicle 4 in the horizontal direction. Also, the body of the vehicle 4 on which the on-board cameras 21-24 are mounted is reflected in the images captured by the on-board cameras 21-24.

In this embodiment, the number of vehicle-mounted cameras is four, but this number may be changed as appropriate, and may be plural or singular. For example, when an in-vehicle camera is installed for the purpose of assisting the vehicle 4 to park in reverse, the in-vehicle camera is composed of a rear camera 22, a left side camera 23, and a right side camera 24. good too.

Returning to FIG. 1, in this embodiment, the output of the steering angle sensor 3 that detects the rotation angle of the steering wheel (steering wheel) is sent to the abnormality detection device 1 via a communication bus B1 such as a CAN (Controller Area Network) bus. is entered.

<2. Abnormality detection device>
As shown in FIG. 1 , the abnormality detection device 1 includes an acquisition section 11 , a control section 12 and a storage section 13 .

The obtaining unit 11 temporally continuously obtains analog or digital captured images from the vehicle-mounted cameras 21 to 24 at a predetermined cycle (for example, 1/30 second cycle). In other words, an aggregate of captured images acquired by the acquisition unit 11 is a moving image captured by the vehicle-mounted cameras 21-24. Then, when the acquired captured image is analog, the acquisition unit 11 converts (A/D converts) the analog captured image into a digital captured image. The acquisition unit 11 outputs the acquired photographed image or the acquired and converted photographed image to the control unit 12 . One captured image output from the acquisition unit 11 becomes one frame image.

The control unit 12 is, for example, a microcomputer, and controls the entire abnormality detection device 1 in an integrated manner. The control unit 12 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). The storage unit 13 is, for example, a non-volatile memory such as a flash memory, and stores various kinds of information. The storage unit 13 stores a program as firmware and various data.

The control unit 12 includes an extraction unit 121 , a flow derivation unit 122 , a selection unit 123 , an identification unit 124 , an estimation unit 125 and a determination unit 126 . At least one of the extraction unit 121, the flow derivation unit 122, the selection unit 123, the identification unit 124, the estimation unit 125, and the determination unit 126 of the control unit 12 is an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array) or other hardware may be used. Also, the extraction unit 121, the flow derivation unit 122, the selection unit 123, the identification unit 124, the estimation unit 125, and the determination unit 126 are conceptual components. A function performed by one component may be distributed to a plurality of components, or a function possessed by a plurality of components may be integrated into one component. Further, the acquisition unit 11 may be implemented by the CPU of the control unit 12 performing arithmetic processing according to a program.

The extraction unit 121 extracts a plurality of feature points including a first feature point and a second feature point from images captured by the vehicle-mounted cameras 21-24. A feature point is a point that can be conspicuously detected in a captured image, such as an intersection of edges in the captured image. The feature points are, for example, edges of white lines drawn on the road surface, cracks on the road surface, stains on the road surface, gravel on the road surface, and the like. A large number of feature points usually exist in one captured image. The extraction unit 121 extracts feature points using, for example, a known method such as a Harris operator. The extraction unit 121 selects a first feature point and a second feature point from the feature points on the road surface. In addition, in this embodiment, a road surface is assumed as a predetermined plane.

The flow derivation unit 122 derives an optical flow for each feature point extracted by the extraction unit 121 from two captured images captured at different times. Optical flow is a motion vector that indicates the motion of feature points between two captured images captured at different times. The interval between the different times may be the same as the frame period of the acquisition unit 11 or multiple times the frame period of the acquisition unit 11 .

The selection unit 123 selects the optical flow of the first feature point and the optical flow of the second feature point from the optical flows derived by the flow derivation unit 122 based on the positions of the optical flows derived by the flow derivation unit 122. select. Details of the selection method will be described later.

The specifying unit 124 uses the respective optical flows of the first feature point and the second feature point to determine the positional changes of the first feature point and the second feature point extracted from the two captured images captured at different times. calculate.

The specifying unit 124 specifies two sets of planes whose intersection lines with a predetermined plane are parallel to each other, based on the positional change of the first feature point and the positional change of the second feature point.

The estimation unit 125 estimates the orientation of the camera (mounting angle of the camera) based on the set of surfaces identified by the identification unit 124 . For example, when the extraction unit 121 extracts the first feature point and the second feature point from the front camera 21, the estimation unit 125 estimates the orientation of the front camera 21 (camera mounting angle).

The determination unit 126 determines whether or not the camera is in a state of misalignment based on the orientation of the camera estimated by the estimation unit 125 . If it is determined that the camera is in a state of mounting misalignment, the abnormality detection device 1 has detected an abnormality in the camera. For example, if the abnormality detection device 1 notifies the user of the vehicle 4 of this abnormality, the user of the vehicle 4 can take countermeasures such as requesting the dealer to adjust the mounting of the camera.

<3. Posture Estimation Device>
The posture estimation device 14 is configured by the acquisition unit 11, the extraction unit 121, the flow derivation unit 122, the selection unit 123, the identification unit 124, and the estimation unit 125 described above. In other words, the anomaly detection device 1 includes the attitude estimation device 14 . FIG. 3 is a flow chart showing an example of a camera pose estimation flow by the pose estimation device 14 . An example of a camera posture estimation flow by the posture estimation device 14 will be described below, taking the case of estimating the posture of the front camera 21 as an example.

As shown in FIG. 3, first, the control unit 12 monitors whether the vehicle 4 equipped with the front camera 21 is traveling straight (step S1). Whether or not the vehicle 4 is traveling straight can be determined, for example, based on the rotation angle information of the steering wheel obtained from the steering angle sensor 3 . For example, when the vehicle 4 moves completely straight when the rotation angle of the steering wheel is zero, not only when the rotation angle is zero, but also when the rotation angle is within a certain range between positive and negative directions. It may be determined that the vehicle 4 is traveling straight, including the case where there is. It should be noted that straight running includes both forward straight running and backward straight running.

The control unit 12 repeats the monitoring of step S1 until it detects that the vehicle 4 is traveling straight. In other words, the controller 12 does not estimate the orientation of the camera unless the vehicle 4 is traveling straight. According to this, the orientation of the camera is estimated using the positional change of the first feature point and the positional change of the second feature point during the straight movement of the moving object, and when the traveling direction of the vehicle 4 is curved, Since the orientation of the camera is not estimated using the information of , it is possible to avoid complication of the information processing for estimating the orientation of the camera, more specifically, the processing of step S3, which will be described later.

In the process of step S3, which will be described later, the orientation of the camera can be estimated if the optical flow of the first feature point and the optical flow of the second feature point, which do not overlap each other and are not located on the same straight line, can be derived. Therefore, unlike the present embodiment, the orientation of the camera may be estimated when the traveling direction of the vehicle 4 is curved.

In this embodiment, when it is determined that the vehicle 4 is traveling straight (YES in step S1), the extraction unit 121 extracts feature points (step S2).

FIG. 4 is a diagram for explaining a technique for extracting feature points. FIG. 4 schematically shows a captured image P1 captured by the front camera. The captured image P includes an area BO in which the body of the vehicle 4 is reflected. A plurality of feature points FP exist on the road surface RS. In FIG. 4, there are a plurality of feature points FP at the corners of the white lines drawn on the road surface.

After extracting the plurality of feature points FP, the flow derivation unit 122 derives an optical flow for each feature point (step S3).

FIG. 5 is a diagram for explaining a method of deriving the optical flow of feature points. FIG. 5 is a schematic diagram shown for convenience similar to FIG. FIG. 5 shows a photographed image P2 photographed by the front camera 21 after a predetermined period of time has elapsed after the photographed image P1 shown in FIG. 4 was photographed. After the photographed image P1 shown in FIG. 4 is photographed, the vehicle 4 is traveling forward in a straight line until a predetermined time elapses. The white circle marks shown in FIG. 5 indicate the position of each feature point at the time of photographing the photographed image P1 shown in FIG. The black circle marks shown in FIG. 5 indicate the positions of the feature points at the time when the photographed image P2 shown in FIG. 5 was photographed.

As shown in FIG. 5 , when the vehicle 4 moves straight forward, each characteristic point present in front of the vehicle 4 approaches the vehicle 4 . That is, each feature point appears at a different position between the captured image P2 shown in FIG. 5 and the captured image P1 shown in FIG. The flow derivation unit 122 associates each feature point of the captured image P2 shown in FIG. 5 with each feature point of the captured image P2 shown in FIG. Based on the position of , an optical flow is derived for each associated feature point (tracked feature point) (step S3).

Next, the selection unit 123 selects the optical flow of the first feature point and the optical flow of the second feature point from the optical flows derived by the flow derivation unit 122 (step S3).

When the optical flow of the first feature point and the optical flow of the second feature point are derived, the specifying unit 124 performs coordinate transformation on the feature points using the internal parameters of the front camera 21 stored in the storage unit 13. I do. In the coordinate conversion, aberration correction and distortion correction of the front camera 21 are performed. Aberration correction is performed to correct distortion due to aberration of the optical system of the front camera 21 . Specifically, it is correction of distortion such as barrel distortion and pincushion distortion. Distortion correction is performed to correct distortion of the optical system itself of the front camera 21 . Specifically, it is fisheye correction. Coordinate transformation converts the coordinates of the feature points into coordinates obtained on a two-dimensional image when the subject is photographed by perspective projection.

When the feature point coordinate transformation is performed, the specifying unit 124 sets the starting point of the optical flow OF1 of the first feature point to the vertex SP1, the end point of the optical flow OF1 of the first feature point to the vertex EP1, and A quadrangle QL is virtually formed having the vertex SP2 as the starting point of the optical flow OF2 of the second feature points and the vertex EP as the ending point of the optical flow OF2 of the second feature points (step S4). Hereinafter, for the sake of explanation, geometric elements such as quadrilaterals and sides of the quadrilaterals are virtually formed and used. However, in actual processing, processing may be based on vector operations such as the coordinates of feature points and the direction of straight lines, and may not be based on geometric elements having equivalent effects.

When the quadrangle QL is virtually formed, the specifying unit 124 uses the quadrangle QL and the internal parameters of the front camera 21 stored in the storage unit 13 to determine the projection plane IMG of the front camera 21 in the three-dimensional space. The quadrangle QL is moved upward to virtually generate a quadrangle QL1 on the projection plane IMG (step S5).

For the sake of explanation, the edges are defined as follows. A first side SD1 of quadrangle QL1 corresponds to a side connecting vertex SP1 and vertex SP2 in quadrangle QL. That is, it corresponds to the side connecting the first feature point and the second feature point in the captured image P1. Similarly, the second side SD2 of quadrangle QL1 corresponds to the side connecting vertex SP2 and vertex EP2 in quadrangle QL. That is, it corresponds to the optical flow OF2 of the second feature point. Similarly, the third side SD3 of quadrangle QL1 corresponds to the side connecting vertex EP1 and vertex EP2 in quadrangle QL. That is, it corresponds to the side connecting the first feature point and the second feature point in the captured image P2. Similarly, the fourth side SD4 of quadrangle QL1 corresponds to the side connecting vertex SP1 and vertex EP1 in quadrangle QL. That is, it corresponds to the optical flow OF1 of the first feature point.

Also, a surface is defined as follows (see FIG. 7). A plane including the first side SD1 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a first plane F1. Similarly, a plane including the second side SD2 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a second plane F2. Similarly, a surface including the third side SD3 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a third surface F3. Similarly, a plane including the fourth side SD4 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a fourth plane F4.

Next, the identifying unit 124 identifies two sets of planes whose intersection lines with a predetermined plane are parallel to each other (Step S6). A predetermined plane is a plane whose normal is known in advance. Specifically, it is the plane on which the vehicle is moving, that is, the road surface. The predetermined plane does not have to be a strict plane, as long as it can be regarded as a plane when the specifying unit 124 specifies two sets of planes whose intersection lines with the predetermined plane are parallel to each other. good.

In this embodiment, when the vehicle 4 is traveling straight ahead, the abnormality detection device 1 extracts feature points from two images taken at different times and calculates the optical flow of the feature points. Also, the feature points are extracted from a stationary object positioned on a predetermined plane such as a road surface. Therefore, the calculated optical flow represents relative position change of the stationary object with respect to the vehicle 4 in the real world. That is, it is the movement vector of the vehicle 4 whose direction is reversed.

The second side SD2 and the fourth side SD4 of the quadrangle QL1 both correspond to the optical flow of the feature points, and therefore both correspond to the movement vector of the vehicle 4 in the real world. Therefore, it is assumed that they are parallel to each other on the road surface.

The first side SD1 and the third side SD3 of the quadrangle QL1 both correspond to the positional relationship between the feature points, and thus correspond to the positional relationship between the stationary objects accompanying the movement of the vehicle 4 in the real world. The positional relationship before movement corresponds to the first side SD1, and the positional relationship after movement corresponds to the third side SD3. Since the position of the stationary object does not change at this time, the positional relationship does not change before and after the movement. Therefore, it is assumed that they are also parallel to each other on the road surface.

Therefore, the identification unit 124 has two sets of surfaces having parallel lines of intersection with the road surface: the set of the second surface F2 and the fourth surface F4, and the set of the first surface F1 and the third surface F3. identify. In other words, the specifying unit 124 specifies a total of two pairs of surfaces including optical flows as one pair and surfaces including feature points captured at the same time as another pair.

In FIG. 7, a quadrangle QL2 represents the position of the first feature point in the three-dimensional space (real world) at the time of capturing the captured image P2, and the position of the second feature point in the three-dimensional space at the time of capturing the captured image P2. , the position of the first feature point in the three-dimensional space at the time of capturing the captured image P1, and the position of the second feature point in the three-dimensional space at the time of capturing the captured image P1. . The first face F1 includes the first side SD11 of the quadrangle QL2. Similarly, the second surface F2 includes the second side SD12 of the quadrilateral QL2, the third surface F3 includes the third side SD13 of the quadrilateral QL2, and the fourth surface F4 includes the fourth side SD14 of the quadrilateral QL2. At this time, the quadrangle QL2 is assumed to be a parallelogram formed on the road surface as described above.

Next, the estimation unit 125 calculates the normal line of the road surface (step S7). First, the estimation unit 125 obtains the direction of the line of intersection between the surfaces based on the first surface F1 and the third surface F3, which are one of the pairs of surfaces identified by the identification unit 124 . Specifically, the direction of the line of intersection CL1 between the first surface F1 and the third surface F3 is determined (see FIG. 8). A direction vector V1 of the line of intersection CL1 is a vector perpendicular to each of the normal vector of the first surface F1 and the normal vector of the third surface F3. Therefore, the estimating unit 125 obtains the direction vector V1 of the line of intersection CL1 by the outer product of the normal vector of the first surface F1 and the normal vector of the third surface F3. Since the first plane F1 and the third plane F3 are parallel to the road surface, the direction vector V1 is parallel to the road surface.

Similarly, the estimation unit 125 obtains the direction of the line of intersection between the second surface F2 and the fourth surface F4 that are the other of the set of surfaces identified by the identification unit 124 . Specifically, the direction of the intersection line CL2 between the second surface F2 and the fourth surface F4 is obtained (see FIG. 9). A direction vector V2 of the line of intersection CL2 is a vector perpendicular to each of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Therefore, the estimation unit 125 obtains the direction vector V2 of the line of intersection CL2 from the cross product of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Similarly, the second surface F2 and the fourth surface F4 also have parallel lines of intersection with the road surface, so the direction vector V2 is parallel with the road surface.

The estimating unit 125 calculates the normal to the surface of the quadrangle QL2, that is, the normal to the road surface, from the outer product of the direction vector V1 and the direction vector V2. Since the normal line of the road surface calculated by the estimating unit 125 is calculated in the camera coordinate system of the front camera 21, the coordinate system of the three-dimensional space is obtained from the difference from the vertical direction, which is the normal line of the actual road surface, and the front surface of the road surface is calculated. The pose of the camera 21 can be estimated. Based on the estimation result, the estimation unit 125 estimates the posture of the front camera 21 with respect to the vehicle 4 (step S8). Note that the calculation process in step S7 can be executed using, for example, the known ARToolkit.

When the posture estimation of the front camera 21 in step S8 ends, the flow shown in FIG. 5 ends.

The posture estimation device 14 uses the movement of the vehicle 4 to autonomously specify two sets of planes whose intersection lines with a predetermined plane are parallel to each other, so that only the feature point extraction error is the estimation accuracy. can be used to estimate the pose of the camera, which affects the In other words, the posture estimation device 14 can estimate the posture of the camera with few error factors. Therefore, the orientation estimation device 14 can accurately estimate the orientation of the camera.

Since the degree of influence of feature point extraction errors on pose estimation varies depending on the position of the optical flow, as described above, the selection unit 123 causes the flow derivation unit 122 to perform The optical flow of the first feature point and the optical flow of the second feature point are selected from the derived optical flows. This makes it possible to use the optical flow of the first feature point and the optical flow of the second feature point, which reduce the degree of influence of feature point extraction errors on pose estimation, for pose estimation. Therefore, pose estimation apparatus 14 can suppress a decrease in the accuracy of pose estimation that depends on the position of the optical flow used for pose estimation.

In the above description, the plane including the optical center OC of the front camera 21 and one side of the quadrangle QL1 is specified, but this is not the only option. A plane may be specified by specifying the normal direction of the plane. For example, the normal direction of the plane may be obtained from the outer product of the direction vectors from the optical center OC to each vertex, and the plane may be identified by the normal direction. In this case, the normal direction of the first surface F1 and the normal direction of the third surface F3 are set as one set, and the normal direction of the second surface F2 and the normal direction of the fourth surface F4 are set as another set. Two sets may be identified.

Also, the plane may be translated. For example, instead of the first plane F1, a plane translated from the first plane F1 may be paired with the third plane F3. This is because the direction of the line of intersection with a predetermined plane does not change even if it is translated.

<4. Selecting Optical Flow>
Next, detailed operation of the selection unit 123 will be described.

FIG. 10 shows the error (PAN, TILT, and ROLL errors) with respect to the expected value of the result of estimating the posture of the front camera 21 by the posture estimation device 14 using the captured image of the front camera 21 on the vertical axis. It is a graph in which the horizontal axis represents the distance in the horizontal direction between the real-world position of the point and the real-world position of the second feature point.

As is clear from the graph shown in FIG. 10, when the posture estimation device 14 estimates the posture of the front camera 21 using the captured image of the front camera 21, the selection unit 123 selects The optical flow of the first feature point and the optical flow of the second feature point are adjusted so that the distance in a predetermined direction between the position and the real-world position of the second feature point is greater than or equal to a first predetermined value V1 (see FIG. 10). It is preferred to select flow. This is because the degree of influence of feature point extraction errors on posture estimation is reduced, and the accuracy of posture estimation is increased.

Here, the predetermined direction is preferably a direction parallel to the road surface and perpendicular to the traveling direction of the vehicle. For example, in the case of the front camera 21 and the back camera 22, the predetermined direction is the horizontal direction (horizontal direction) on the image because the optical axis of the camera coincides with the traveling direction of the vehicle. In the case of the left side camera 23 and the right side camera 24, the predetermined direction is the vertical direction (depth direction) on the image because the optical axis of the camera and the traveling direction of the vehicle are orthogonal.

Also, the distance in a given direction is an index of how far apart two points are in a given direction. Specifically, when a position vector is formed between two points, the length of the vector obtained by projecting the position vector in a predetermined direction is calculated.

However, as is clear from the graph shown in FIG. 10, the distance in a predetermined direction between the real-world position of the first feature point and the real-world position of the second feature point is a second predetermined value V2 (V2>V1 , FIG. 10), the accuracy of attitude estimation is reduced due to the influence of distortion.

Therefore, in order to prevent the distance in a predetermined direction between the real-world position of the first feature point and the real-world position of the second feature point from becoming equal to or greater than the second predetermined value V2, the selection unit 123 , the optical flow of the first feature point and the optical flow of the second feature point located within a predetermined distance from the center of the optical axis of the front camera 21 are preferably selected. In other words, the selection unit 123 preferably does not select the optical flow of the first feature point and the optical flow of the second feature point from the periphery of the captured image.

Here, the case where posture estimation device 14 estimates the posture of front camera 21 using an image captured by front camera 21 has been described. The same is true when estimating .

In FIG. 11, the abscissa represents the error (PAN, TILT, and ROLL errors) with respect to the expected value of the result of estimating the pose of the right-side camera 24 by the pose estimation device 14 using the captured image of the right-side camera 24. Based on the feature point closer to the vehicle 4 out of the first feature point and the second feature point, that is, the position in the real world of the feature point closer to the vehicle 4 out of the first feature point and the second feature point It is a graph in which the vertical axis represents the distance in the horizontal direction between the position of the first feature point in the real world and the position of the second feature point in the real world.

In FIG. 12, the abscissa represents the error (PAN, TILT, and ROLL errors) with respect to the expected value of the result of estimating the pose of the right side camera 24 by the pose estimation device 14 using the captured image of the right side camera 24. Based on the feature point closer to the vehicle 4 out of the first feature point and the second feature point, that is, the position in the real world of the feature point farther from the vehicle 4 out of the first feature point and the second feature point It is a graph in which the vertical axis represents the distance in the horizontal direction between the position of the first feature point in the real world and the position of the second feature point in the real world.

As is clear from the graphs shown in FIGS. 11 and 12, when posture estimation device 14 estimates the posture of right side camera 24 using the captured image of right side camera 24, selection unit 123 selects the right The optical flow of the first feature point located within the predetermined distance from the center of the optical axis of the side camera 24 and the optical flow of the second feature point located in the area away from the center of the optical axis of the right side camera 24 by the predetermined distance. is preferred. This is because the degree of influence of feature point extraction errors on posture estimation is reduced, and the accuracy of posture estimation is increased.

As is clear from the comparison of the graphs shown in FIG. 11 and the graphs shown in FIG. 12, the selection unit 123 uses the feature point closer to the vehicle 4 out of the first feature point and the second feature point as a reference, that is, It is preferable to fix the position in the real world of the feature point closer to the vehicle 4 than the first feature point and the second feature point and perform the selection process. Using the feature point closer to the vehicle 4 out of the first feature point and the second feature point as the reference is better than using the feature point farther from the vehicle 4 out of the first feature point and the second feature point as the reference. Also, when the distance in the left-right direction between the real-world position of the first feature point and the real-world position of the second feature point is small, it is possible to suppress a decrease in the accuracy of posture estimation due to the effects of distortion. It is from.

Here, the case where posture estimation device 14 estimates the posture of right side camera 24 using the image captured by right side camera 24 has been described. The same is true when estimating the orientation of the camera 23 .

FIGS. 13 to 15 show the errors (PAN, TILT, and ROLL errors) with respect to the expected values of the result of estimating the posture of the right side camera 24 by the posture estimation device 14 using the captured image of the right side camera 24. The vertical axis (the vertical axis on the left side of the page) is taken, and the amount of deviation in the horizontal direction of the captured image between the optical flow of the first feature point and the optical flow of the second feature point is taken on the second vertical axis (the vertical axis on the right side of the page). is a graph.

FIG. 13 is a graph when the distance in the horizontal direction between the position of the first feature point in the real world and the position of the second feature point in the real world is the smallest among FIGS. FIG. 14 is a graph when the distance in the horizontal direction between the real-world position of the first feature point and the real-world position of the second feature point is the second smallest in FIGS. FIG. 15 is a graph when the distance in the horizontal direction between the position of the first feature point in the real world and the position of the second feature point in the real world is the smallest in FIGS.

As is clear from the graphs shown in FIGS. 13 to 15, when posture estimation device 14 estimates the posture of right side camera 24 using the captured image of right side camera 24, the first feature points in the real world The optical flow of the first feature point and the optical flow of the second feature point are selected so that the distance in the horizontal direction between the position and the real-world position of the second feature point is equal to or greater than a predetermined value, and the optical flow of the first feature point is selected. It is preferable to increase the predetermined value as the distance in the direction perpendicular to the front-rear direction between the real-world position and the real-world position of the second feature point increases. This is because it is possible to prevent the accuracy of posture estimation from deteriorating when the amount of deviation in the horizontal direction of the captured image between the optical flow of the first feature point and the optical flow of the second feature point is large.

Here, the case where posture estimation device 14 estimates the posture of right side camera 24 using the image captured by right side camera 24 has been described. The same is true when estimating the orientation of the camera 23 . When estimating the posture of the front camera 21 using the image captured by the front camera 21, or when estimating the posture of the back camera 22 using the image captured by the back camera 22, the left-right direction and the front-back direction are interchanged. is the same.

<5. Correction system>
FIG. 16 is a block diagram showing the configuration of the correction system SYS2 according to the embodiment. In FIG. 16, the same parts as those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted. As shown in FIG. 16, the correction system SYS2 includes a correction device 1' and an imaging unit 2. FIG.

The correction device 1' has the same configuration as that of the abnormality detection device 1 shown in FIG. That is, the correction device 1 ′ includes the posture estimation device 14 and the correction section 127 .

The correction unit 127 corrects the parameters of the camera based on the camera orientation estimated by the estimation unit 125 . As a result, even if there is a camera mounting misalignment, the camera is calibrated according to the misalignment, and the adverse effects of the camera mounting misalignment on the captured image can be suppressed.

<6. Notes>
The configurations of the embodiments and examples herein are merely illustrative of the present invention. The configurations of the embodiments and modifications may be changed as appropriate without departing from the technical idea of the present invention. Also, multiple embodiments and modifications may be implemented in combination within a possible range.

1 anomaly detection device 1' correction device 14 attitude estimation device 21 to 24 vehicle-mounted camera 11 acquisition unit 121 extraction unit 122 flow derivation unit 123 selection unit 124 identification unit 125 estimation unit 126 determination unit 127 correction unit

Claims (11)

an acquisition unit that acquires a captured image captured by a camera mounted on a moving body;
an extraction unit that extracts a plurality of feature points including a first feature point and a second feature point from the captured image;
a flow derivation unit that derives an optical flow for each of the plurality of feature points;
Selection of selecting the optical flow of the first feature point and the optical flow of the second feature point from the optical flows derived by the flow derivation unit based on the positions of the optical flows derived by the flow derivation unit. Department and
a specifying unit that specifies two sets of planes whose intersection lines with a predetermined plane are parallel to each other based on the optical flow of the first feature point and the optical flow of the second feature point selected by the selecting unit; ,
an estimation unit that estimates the orientation of the camera based on the set of planes identified by the identification unit;
A posture estimation device comprising: The selection unit selects the first feature point so that a distance in a predetermined direction between a real-world position of the first feature point and a real-world position of the second feature point is equal to or greater than a predetermined value. 2. The pose estimation device according to claim 1, wherein the optical flow and the optical flow of the second feature point are selected. The selection unit increases the predetermined value as the distance between the real-world position of the first feature point and the real-world position of the second feature point in a direction perpendicular to the predetermined direction increases. 3. The attitude estimation device according to claim 2. 4. The selection unit according to claim 2, wherein the selection unit selects the optical flow of the first feature point and the optical flow of the second feature point located within a predetermined distance from the center of the optical axis of the camera. pose estimator. 5. The posture estimation device according to claim 4, wherein said camera faces the front or rear of said moving object. The selecting unit selects optical flows of the first feature points located within a predetermined distance from the center of the optical axis of the camera, and the second feature located in a region away from the center of the optical axis of the camera by more than the predetermined distance. 4. A pose estimator according to claim 2 or claim 3, which selects an optical flow of points. 7. The posture estimation device according to claim 6, wherein said camera faces left or right of said moving body. a posture estimation device according to any one of claims 1 to 7;
a determination unit that determines whether or not there is a mounting deviation of the camera based on the orientation of the camera estimated by the estimation unit;
An anomaly detection device. a posture estimation device according to any one of claims 1 to 7;
a correction unit that corrects parameters of the camera based on the orientation of the camera estimated by the estimation unit;
a compensator. an acquisition step of acquiring a photographed image photographed by a camera mounted on a mobile body;
an extracting step of extracting a plurality of feature points including a first feature point and a second feature point from the captured image;
a flow derivation step of deriving an optical flow for each of the plurality of feature points;
A selection of selecting the optical flow of the first feature point and the optical flow of the second feature point from among the optical flows derived in the flow derivation step based on the positions of the optical flows derived in the flow derivation step. process and
a specifying step of specifying two sets of planes whose intersection lines with a predetermined plane are parallel to each other based on the optical flow of the first feature point and the optical flow of the second feature point selected in the selecting step; ,
an estimating step of estimating the orientation of the camera based on the set of planes identified in the identifying step;
A pose estimation method comprising: A posture estimation device comprising a control unit for estimating the posture of a camera mounted on a moving object,
The control unit
Acquiring a captured image captured by the camera,
extracting a plurality of feature points including a first feature point and a second feature point from the captured image;
deriving an optical flow for each of the plurality of feature points;
selecting the optical flow of the first feature point and the optical flow of the second feature point from the optical flows based on the derived positions of the optical flows;
Based on the optical flow of the selected first feature point and the optical flow of the second feature point, two pairs of planes whose intersection lines with a predetermined plane are parallel to each other are specified,
estimating a pose of the camera based on the identified set of planes;
Posture estimation device.

Images