JP7237773B2 - Posture estimation device, anomaly detection device, correction device, and posture estimation method - Google Patents

Description

The present invention relates to a posture estimation device for estimating the posture of a camera, an anomaly detection device for detecting anomalies in a camera, a correction device for correcting camera parameters, and a posture estimation method for estimating the posture of a camera.

2. Description of the Related Art Conventionally, vehicle-mounted cameras are used to provide driving assistance such as parking assistance for vehicles. An on-board camera is fixedly attached to a vehicle before the vehicle leaves the factory. However, the on-vehicle camera may be displaced from the mounting state at the time of shipment from the factory due to, for example, accidental contact or aging. If the position of the in-vehicle camera (relative position of the in-vehicle camera to the vehicle body) shifts, an error will occur in the amount of steering wheel steering determined using the camera image. is important.

Patent Literature 1 discloses a technique for detecting optical axis deviation of a vehicle-mounted camera. The optical axis deviation detection device for an on-vehicle camera in Patent Document 1 includes image processing means and determination means. The image processing means detects the position information of the marking from the captured image of the vehicle-mounted camera that captures the range including the marking on the vehicle body for driving assistance. The judging means judges the misalignment of the photographing optical axis by comparing the initially set marking position information and the newly detected marking position information.

JP 2004-173037 A

Since the captured image of the on-vehicle camera shows the scenery around the vehicle, for example, when the marking is a specific shape of a part of the hood, the specific shape may not be easily extracted. If the specific shape is erroneously detected, there is a possibility that the misalignment of the optical axis of the in-vehicle camera may not be accurately determined. It should be noted that the deviation of the optical axis of the vehicle-mounted camera means that the posture of the vehicle-mounted camera is not as designed.

SUMMARY OF THE INVENTION In view of the above problems, it is an object of the present invention to provide a technique capable of estimating the pose of a camera with high accuracy and suppressing a decrease in pose estimation accuracy that depends on the positions of feature points used for pose estimation. and

A posture estimation apparatus according to one aspect of the present invention includes an acquisition unit that acquires a captured image captured by a camera mounted on a mobile object, and an image captured at a first time and at a second time after the first time. Extracting feature points from each of the two captured images, and extracting feature points from each of the two captured images captured at a third time after the first time and at a fourth time after the third time. and when it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated by a first predetermined distance or more in a predetermined direction, the first time and the Selecting the first feature points corresponding to each other from the feature points extracted from the two captured images captured at the second time, calculating the positional change of the first feature points, calculating the third time and the Different from the first feature points, the second feature points corresponding to each other are selected from the feature points extracted from the two captured images captured at a fourth time, and positional changes of the second feature points are selected. a specifying unit that calculates and specifies two sets of planes whose intersection lines with a predetermined plane are parallel to each other based on the positional change of the first feature point and the positional change of the second feature point; and an estimating unit that estimates the orientation of the camera based on the set of planes specified in (1).

In addition, in the first configuration and other configurations described later, the captured image captured at the first time and the captured image captured at the third time are the same image, and the captured image is captured at the second time. The photographed image taken at the fourth time and the photographed image photographed at the fourth time may be the same image, and the photographed image photographed at the first time and the photographed image photographed at the third time may be different images, and the captured image captured at the second time and the captured image captured at the fourth time may be different images.

In the posture estimation device having the first configuration, the specifying unit determines that the real-world position of the first feature point and the real-world position of the second feature point are in the first predetermined direction in the predetermined direction. In the case where it can be estimated that they are separated by less than a distance, a configuration (second configuration) that does not specify a set of planes whose lines of intersection with the predetermined plane are parallel to each other may be employed.

In the posture estimation device having the first configuration, the extracting unit sets a plurality of regions aligned in the predetermined direction in the real world within the captured image, and sets them at different positions within the captured image. The first feature point is extracted from one of the two regions and the second feature point is extracted from the other. A configuration (a third configuration) may be used in which it is estimated that the position of the point on the real world is separated by the first predetermined distance or more in the predetermined direction.

In the posture estimation device having the first or second configuration described above, the specifying unit performs the position of the first feature point on the captured image captured at the first time and the position of the first feature point captured at the third time. At least one of the position of the second feature point on the captured image, and the position of the first feature point on the captured image captured at the first time and the position of the first feature point captured at the third time based on the distance from the position of the second feature point on the captured image and/or at the position of the first feature point on the captured image captured at the second time and at the fourth time and/or the position of the first feature point on the captured image captured at the second time and the position of the first feature point on the captured image captured at the fourth time. the position of the first feature point in the real world and the position of the second feature point in the real world based on the distance from the position of the second feature point on the photographed image obtained in the predetermined direction; may be a configuration (fourth configuration) for estimating whether or not the distance is greater than or equal to the first predetermined distance.

In the posture estimation device having any one of the first to fourth configurations, the identifying unit determines that the real-world position of the first feature point and the real-world position of the second feature point are aligned in the predetermined direction. , the position of the first feature point in the real world and the position of the second feature point in the real world can be estimated to be separated by the first predetermined distance or more in the predetermined direction. If it can be estimated that they are separated by a second predetermined distance longer than the distance, a configuration (fifth configuration) in which a set of planes whose intersection lines with the predetermined plane are parallel to each other is not specified.

A posture estimation apparatus according to another aspect of the present invention includes an acquisition unit that acquires a captured image captured by a camera mounted on a mobile object, and an image captured at a first time and a second time after the first time. A feature point is extracted from each of the two photographed images, and a feature point is extracted from each of the two photographed images photographed at a third time after the first time and at a fourth time after the third time. an extracting unit, and an amount of movement of the moving body from the first time to the second time in the real world and an amount of movement of the moving body from the third time to the fourth time in the real world, respectively; selecting first feature points corresponding to each other from feature points extracted from the two captured images captured at the first time and the second time when it can be estimated that the distance is equal to or greater than a third predetermined distance; A change in position of the first feature point is calculated, and from the feature points extracted from the two photographed images photographed at the third time and the fourth time, different from the first feature point, corresponding to each other Selecting a second feature point, calculating a positional change of the second feature point, and based on the positional change of the first feature point and the positional change of the second feature point, lines of intersection with a predetermined plane being parallel to each other and an estimating unit for estimating the orientation of the camera based on the sets of surfaces specified by the specifying unit (sixth configuration).

In the posture estimation device having the sixth configuration, the specifying unit calculates the movement amount from the first time in the real world to the second time in the real world and the third time in the real world for the mobile. to the fourth time, and when it can be estimated that each of the movement amounts is less than the third predetermined distance, a configuration that does not specify a set of planes whose intersection lines with the predetermined plane are parallel to each other (a seventh configuration).

In the posture estimation device having the sixth configuration, when the speed of the moving body is equal to or less than a first predetermined speed, the identifying unit performs and the movement amount of the moving body from the third time to the fourth time in the real world are each estimated to be less than the third predetermined distance (eighth configuration). good.

In the posture estimation device having the sixth configuration, when the speed of the moving object is equal to or lower than a first predetermined speed, the extracting unit extracts an interval between the first time and the second time and an interval between the third time and the The interval from the fourth time is made longer than when the speed of the moving object is higher than the first predetermined speed, and the specifying unit is configured to set the interval from the first time in the real world of the moving object to the second time. and the movement amount of the moving body from the third time to the fourth time in the real world are each estimated to be equal to or greater than the third predetermined distance (ninth configuration), good too.

In the posture estimation device having the sixth or seventh configuration, the specifying unit performs the position of the first feature point on the captured image captured at the first time and the position of the first feature point captured at the second time. At least one of the position of the first feature point on the captured image, and the position of the first feature point on the captured image captured at the first time and the position of the first feature point captured at the second time Based on the distance from the position of the first feature point on the captured image, and the position of the second feature point on the captured image captured at the third time and the position of the second feature point captured at the fourth time At least one of the position of the second feature point on the captured image, and the position of the second feature point on the captured image captured at the third time and the position of the second feature point captured at the fourth time Based on the distance from the position of the second feature point on the captured image, the amount of movement of the moving body from the first time point in the real world to the second time point and the moving amount of the moving body in the real world A configuration (a tenth configuration) may be used in which it is estimated whether or not the amount of movement from the third time to the fourth time is equal to or greater than a third predetermined distance.

In the posture estimating device having any one of the sixth to tenth configurations, the specifying unit includes a moving amount from the first time to the second time in the real world of the moving body and a movement amount of the moving body in the real world movement amount from the third time to the fourth time can be estimated to be equal to or greater than a third predetermined distance, movement of the moving body from the first time to the second time in the real world and the amount of movement of the moving body from the third time to the fourth time in the real world can be estimated to be equal to or greater than a fourth predetermined distance that is longer than the third predetermined distance, the predetermined plane A configuration (eleventh configuration) in which a set of planes whose intersection lines are parallel to each other is not specified.

An anomaly detection apparatus according to the present invention is a posture estimation device having any one of the first to eleventh configurations described above, and the camera posture estimated by the estimating unit in a state in which the camera is attached with a misalignment. and a determination unit for determining whether or not there is a configuration (a twelfth configuration).

A correction device according to the present invention includes a posture estimation device having any one of the first to eleventh configurations; a correction unit that corrects parameters of the camera based on the posture of the camera estimated by the estimation unit; (13th configuration).

A posture estimation method according to an aspect of the present invention includes an acquisition step of acquiring a photographed image photographed by a camera mounted on a mobile body, and Extracting feature points from each of the two captured images, and extracting feature points from each of the two captured images captured at a third time after the first time and at a fourth time after the third time. and when it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated by a first predetermined distance or more in a predetermined direction, the first time and the Selecting the first feature points corresponding to each other from the feature points extracted from the two captured images captured at the second time, calculating the positional change of the first feature points, calculating the third time and the Different from the first feature points, the second feature points corresponding to each other are selected from the feature points extracted from the two captured images captured at a fourth time, and positional changes of the second feature points are selected. a specifying step of specifying two sets of planes whose lines of intersection with a predetermined plane are parallel to each other based on the positional change of the first feature point and the positional change of the second feature point; and an estimating step of estimating the orientation of the camera based on the set of planes specified in (14th configuration).

A posture estimation method according to another aspect of the present invention includes an obtaining step of obtaining a photographed image photographed by a camera mounted on a mobile object, and A feature point is extracted from each of the two photographed images, and a feature point is extracted from each of the two photographed images photographed at a third time after the first time and at a fourth time after the third time. an extraction step, and an amount of movement of the moving object from the first time to the second time in the real world and an amount of movement of the moving object from the third time to the fourth time in the real world, respectively. selecting first feature points corresponding to each other from feature points extracted from the two captured images captured at the first time and the second time when it can be estimated that the distance is equal to or greater than a third predetermined distance; A change in position of the first feature point is calculated, and from the feature points extracted from the two photographed images photographed at the third time and the fourth time, different from the first feature point, corresponding to each other Selecting a second feature point, calculating a positional change of the second feature point, and based on the positional change of the first feature point and the positional change of the second feature point, lines of intersection with a predetermined plane being parallel to each other and an estimating step of estimating the orientation of the camera based on the pairs of planes specified in the specifying step (a fifteenth configuration).

According to the present invention, it is possible to accurately estimate the pose of a camera, and to prevent the accuracy of pose estimation from deteriorating depending on the positions of feature points used for pose estimation.

Block diagram showing the configuration of the anomaly detection system Diagram exemplifying the position where the on-board camera is arranged in the vehicle A diagram schematically showing an example of a captured image A diagram schematically showing another example of a captured image The figure which shows typically another example of a picked-up image. Flowchart showing an example of a camera posture estimation flow by a posture estimation device FIG. 4 is a diagram schematically showing a photographed image L frames before the current frame; Schematic diagram of the current frame A diagram schematically showing a photographed image N frames before the current frame. FIG. 4 is a diagram schematically showing a photographed image M frames before the current frame; FIG. 4 is a diagram schematically showing a photographed image L frames before the current frame; Schematic diagram of the current frame A diagram schematically showing a virtually formed quadrangle FIG. 4 is a diagram for explaining a technique for identifying two sets of planes whose intersection lines with a predetermined plane are parallel to each other; Diagram for explaining the method of determining the direction of the line of intersection between the faces based on one of the identified pairs of faces. Diagram for explaining the method of determining the direction of the line of intersection between the faces based on the other of the identified faces Diagram for explaining a setting example of ROI Diagram for explaining another setting example of ROI Block diagram showing the configuration of the correction system

Exemplary embodiments of the invention are described in detail below with reference to the drawings. In the following description, a vehicle is taken as an example of a moving object, but the moving object is not limited to a vehicle. Vehicles broadly include vehicles with wheels, such as automobiles, trains, and automatic guided vehicles. Examples of mobile objects other than vehicles include ships and aircraft.

Further, in the following description, the direction in which the vehicle travels straight ahead, which is the direction from the driver's seat to the steering wheel, is referred to as the "forward direction." In addition, the direction from the steering wheel to the driver's seat, which is the direction in which the vehicle travels straight, is defined as the "rearward direction." In addition, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and from the right side to the left side of the driver who is facing forward is referred to as the "left direction." Also, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and from the left side to the right side of the driver who is facing forward is referred to as the "right direction."

<1. Anomaly detection system>
FIG. 1 is a block diagram showing the configuration of an abnormality detection system SYS1 according to the embodiment. In this embodiment, the abnormality is a state in which the mounting of the camera is misaligned. That is, the abnormality detection system SYS1 is a system that detects misalignment of a camera mounted on a vehicle (in-vehicle camera). As shown in FIG. 1 , an abnormality detection system SYS1 includes an abnormality detection device 1 and an imaging unit 2 .

The abnormality detection device 1 is a device that detects an abnormality based on an image captured by an in-vehicle camera. The abnormality detection device 1 is provided for each vehicle equipped with an in-vehicle camera. In this embodiment, the abnormality detection device 1 acquires a captured image from the imaging unit 2 .

The photographing unit 2 is provided for the purpose of monitoring the situation around the vehicle. The imaging unit 2 includes four cameras 21-24. The four cameras 21-24 are in-vehicle cameras. FIG. 2 is a diagram illustrating the positions where the four vehicle-mounted cameras 21 to 24 are arranged on the vehicle 5. As shown in FIG.

An in-vehicle camera 21 is provided at the front end of the vehicle 5 . For this reason, the vehicle-mounted camera 21 is also called a front camera 21 . The optical axis 21a of the front camera 21 extends along the longitudinal direction of the vehicle 5 in plan view from above. The front camera 21 photographs the forward direction of the vehicle 5 . The vehicle-mounted camera 22 is provided at the rear end of the vehicle 5 . Therefore, the vehicle-mounted camera 22 is also called a back camera 22 . An optical axis 22a of the back camera 22 extends along the longitudinal direction of the vehicle 5 in plan view from above. A back camera 22 photographs the rear direction of the vehicle 5 . The mounting position of the front camera 21 and the rear camera 22 is preferably the center in the left and right direction of the vehicle 5, but may be slightly shifted in the left and right direction from the center in the left and right direction.

The vehicle-mounted camera 23 is provided on the left side door mirror 51 of the vehicle 5 . Therefore, the in-vehicle camera 23 is also called a left side camera 23 . The optical axis 23a of the left side camera 23 extends along the left-right direction of the vehicle 5 in plan view from above. The left side camera 23 photographs the left direction of the vehicle 5 . The vehicle-mounted camera 24 is provided on the right side door mirror 52 of the vehicle 5 . Therefore, the in-vehicle camera 24 is also called a right side camera 24 . The optical axis 24a of the right side camera 24 extends along the left-right direction of the vehicle 5 in plan view from above. The right side camera 24 photographs the right direction of the vehicle 5 . When the vehicle 5 is a so-called door mirrorless vehicle, the left side camera 23 is attached around the rotation axis (hinge portion) of the left side door without interposing the door mirror, and the right side camera 24 is attached to the right side door. It can be attached around the rotating shaft (hinge part) without interposing the door mirror.

The horizontal angle of view θ of each vehicle-mounted camera 21 to 24 is 180 degrees or more. Therefore, the vehicle-mounted cameras 21 to 24 can photograph the entire circumference of the vehicle 5 in the horizontal direction. The body of the vehicle 5 on which the on-board cameras 21-24 are mounted is reflected in the images captured by the on-board cameras 21-24.

In this embodiment, the number of vehicle-mounted cameras is four, but this number may be changed as appropriate, and may be plural or singular. For example, when an in-vehicle camera is installed for the purpose of assisting the vehicle 5 to park in reverse, the in-vehicle camera is composed of a back camera 22, a left side camera 23, and a right side camera 24. good too.

Returning to FIG. 1, in this embodiment, the output of the steering angle sensor 3 that detects the rotation angle of the steering wheel (steering wheel) and the output of the vehicle speed sensor 4 that detects the speed of the vehicle 5 are connected to CAN (Controller Area Network). It is input to the abnormality detection device 1 via a communication bus B1 such as a bus.

<2. Abnormality detection device>
As shown in FIG. 1 , the abnormality detection device 1 includes an acquisition section 11 , a control section 12 and a storage section 13 .

The obtaining unit 11 temporally continuously obtains analog or digital captured images from the vehicle-mounted cameras 21 to 24 at a predetermined cycle (for example, 1/30 second cycle). In other words, an aggregate of captured images acquired by the acquisition unit 11 is a moving image captured by the vehicle-mounted cameras 21-24. Then, when the acquired captured image is analog, the acquisition unit 11 converts (A/D converts) the analog captured image into a digital captured image. The acquisition unit 11 outputs the acquired photographed image or the acquired and converted photographed image to the control unit 12 . One captured image output from the acquisition unit 11 becomes one frame image.

The control unit 12 is, for example, a microcomputer, and controls the entire abnormality detection device 1 in an integrated manner. The control unit 12 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). The storage unit 13 is, for example, a non-volatile memory such as a flash memory, and stores various kinds of information. The storage unit 13 stores a program as firmware and various data.

The control unit 12 includes an extraction unit 121 , a specification unit 122 , an estimation unit 123 and a determination unit 124 . At least one of the extraction unit 121, the identification unit 122, the estimation unit 123, and the determination unit 124 of the control unit 12 is hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). may be configured. Also, the extraction unit 121, the identification unit 122, the estimation unit 123, and the determination unit 124 are conceptual components. A function performed by one component may be distributed to a plurality of components, or a function possessed by a plurality of components may be integrated into one component. Further, the acquisition unit 11 may be implemented by the CPU of the control unit 12 performing arithmetic processing according to a program.

The extraction unit 121 extracts a plurality of feature points including a first feature point and a second feature point from images captured by the vehicle-mounted cameras 21-24. A feature point is a point that can be conspicuously detected in a captured image, such as an intersection of edges in the captured image. The feature points are, for example, edges of white lines drawn on the road surface, cracks on the road surface, stains on the road surface, gravel on the road surface, and the like. A large number of feature points usually exist in one captured image. The extraction unit 121 extracts feature points using, for example, a known method such as a Harris operator. The extraction unit 121 selects a first feature point and a second feature point from the feature points on the road surface. The first feature point and the second feature point are feature points different from each other. In addition, in this embodiment, a road surface is assumed as a predetermined plane.

The specifying unit 122 derives optical flows of the first feature points and the second feature points extracted from the two captured images captured at different times. Optical flow is a motion vector that indicates the motion of feature points between two captured images captured at different times. The interval between the different times may be the same as the frame period of the acquisition unit 11 or multiple times the frame period of the acquisition unit 11 .

The identifying unit 122 can estimate that the real-world position of the first feature point and the real-world position of the second feature point are separated from each other by a first predetermined distance or more and less than a second predetermined distance in a predetermined direction, and , when it can be estimated that the amount of movement of the vehicle 5 at the intervals of the different times in the real world is equal to or greater than the third predetermined distance and less than the fourth predetermined distance, the optical flows of the first feature point and the second feature point are are used to calculate positional changes of the first feature point and the second feature point extracted from two captured images captured at different times. The second predetermined distance is longer than the first predetermined distance, and the fourth predetermined distance is longer than the third predetermined distance. The first predetermined distance and the third predetermined distance may be the same or different. The second predetermined distance and the fourth predetermined distance may be the same or different. If the captured image is an image captured by the front camera 21 or the back camera 22, the predetermined direction may be the horizontal direction. On the other hand, when the captured image is an image captured by the left side camera 23 or the right side camera 24, the predetermined direction may be the front-rear direction.

The identifying unit 122 identifies two sets of planes whose intersection lines with a predetermined plane are parallel to each other, based on the positional change of the first feature point and the positional change of the second feature point.

In some cases, only one of the first feature point and the second feature point can be extracted from a single captured image. Also, the first feature point and the second feature point extracted from a single captured image may not have an appropriate positional relationship.

Examples of the case where only one of the first feature point and the second feature point can be extracted from a single captured image include the captured image shown in FIG. 3 and the captured image shown in FIG.

FIG. 3 schematically shows a photographed image photographed by the front camera 21. As shown in FIG. The captured image shown in FIG. 3 includes an area BO in which the body of the vehicle 5 is reflected. A shaded area in the photographed image shown in FIG. 3 indicates a shadow, and a dotted-line square in the photographed image shown in FIG. 3 is an ROI (region of interest) where feature points can be extracted. In the photographed image shown in FIG. 3, only one feature point FP existing on the road surface RS can be extracted due to the influence of the shadow.

FIG. 4 schematically shows a photographed image photographed by the left side camera 23. As shown in FIG. The captured image shown in FIG. 4 includes an area BO in which the body of the vehicle 5 is reflected. The inside of the dotted-line square in the photographed image shown in FIG. 4 is the ROI, which is an extractable region for feature points. In the photographed image shown in FIG. 4, only one feature point FP existing on the road surface RS can be extracted because there is only one white line corner in the ROI.

As an example of a case where the first feature point and the second feature point extracted from a single captured image do not have an appropriate positional relationship, the captured image shown in FIG. 5 can be mentioned.

FIG. 5 schematically shows a photographed image photographed by the front camera 21. As shown in FIG. The captured image shown in FIG. 5 includes an area BO in which the body of the vehicle 5 is reflected. A shaded area in the photographed image shown in FIG. 5 indicates a shadow, and a dotted-line square in the photographed image shown in FIG. In the captured image shown in FIG. 5, feature points FP existing on the road surface RS can only be extracted at local locations due to the influence of shadows. If the first feature points and the second feature points are unevenly distributed at local locations, there is a risk that the accuracy of camera orientation estimation may be degraded in some of the PAN, TILT, and ROLL axes.

When only one of the first feature point and the second feature point can be extracted from a single captured image, or when the first feature point and the second feature point extracted from the single captured image have an appropriate positional relationship Otherwise, the extracting unit 121 extracts the feature points from each of the two captured images captured at the first time and the second time after the first time, and extracts the feature points from the third time and the third time after the first time. Feature points may be extracted from each of the two captured images captured at the later fourth time. Then, the specifying unit 122 selects the first feature points corresponding to each other from the feature points extracted from the two captured images captured at the first time and the second time, and calculates the position change of the first feature points. Then, from the feature points extracted from the two captured images captured at the third time and the fourth time, the first feature points that are different from the first feature points and correspond to each other are selected, and the positions of the second feature points are selected. Two pairs of planes whose intersection lines with a predetermined plane are parallel to each other may be identified based on the positional changes of the first feature point and the positional changes of the second feature point.

The estimation unit 123 estimates the orientation of the camera (mounting angle of the camera) based on the set of surfaces identified by the identification unit 122 . For example, when the extraction unit 121 extracts the first feature point and the second feature point from the front camera 21, the estimation unit 123 estimates the orientation of the front camera 21 (camera mounting angle).

The determination unit 124 determines whether or not the camera has been misaligned based on the orientation of the camera estimated by the estimation unit 123 . If it is determined that the camera is in a state of mounting misalignment, the abnormality detection device 1 has detected an abnormality in the camera. For example, if the abnormality detection device 1 notifies the user of the vehicle 5 of this abnormality, the user of the vehicle 5 can take countermeasures such as requesting the dealer to adjust the mounting of the camera.

A posture estimation device 14 is configured by the acquisition unit 11 , the extraction unit 121 , the identification unit 122 , and the estimation unit 123 . In other words, the anomaly detection device 1 includes the attitude estimation device 14 . FIG. 6 is a flowchart showing an example of a camera pose estimation flow by the pose estimation device 14 . A case of estimating the orientation of the front camera 21 will be described below as an example.

As shown in FIG. 6, the extraction unit 121 extracts feature points (step S1). More specifically, in step S1, the extraction unit 121 extracts the feature points of the current frame, which is the latest captured image, and the feature points of the captured image L frames before the current frame.

Next, the specifying unit 122 derives an optical flow (step S2). More specifically, in step S2, the extraction unit 121 derives an optical flow of feature points extracted from the latest captured image (current frame) and the captured image L (L is a natural number) frames before the current frame.

If no optical flow is derived in step S2, the frame one frame after the current frame is set as a new current frame, and the processes of steps S1 and S2 are executed for the new current frame.

The storage unit 13 stores the optical flow derived in step S2 as the latest optical flow. The latest optical flow changes to the past optical flow by updating the current frame. The storage unit 13 stores the past optical flow for a certain period of time after the latest optical flow changes to the past optical flow.

After the derivation of the optical flow in step S2 is finished, the specifying unit 122 performs coordinate transformation on the feature points using the internal parameters of the front camera 21 stored in the storage unit 13 . In the coordinate conversion, aberration correction and distortion correction of the front camera 21 are performed. Aberration correction is performed to correct distortion due to aberration of the optical system of the front camera 21 . Specifically, it is correction of distortion such as barrel distortion and pincushion distortion. Distortion correction is performed to correct distortion of the optical system itself of the front camera 21 . Specifically, it is fisheye correction. Coordinate transformation converts the coordinates of the feature points into coordinates obtained on a two-dimensional image when the subject is photographed by perspective projection.

When the coordinate transformation of the feature points described above is completed, the specifying unit 122 determines whether or not a quadrangle can be virtually formed only with the latest optical flow (step S3).

It can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated from each other by a first predetermined distance or more and less than a second predetermined distance in a predetermined direction. can be estimated to be a third predetermined distance or more and less than a fourth predetermined distance, and the latest optical flow If it is determined that a quadrangle can be virtually formed using only the latest optical flow (YES in step S3), the specifying unit 122 virtually forms a quadrangle using only the latest optical flow (step S4). Hereinafter, for the sake of explanation, geometric elements such as quadrilaterals and sides of the quadrilaterals are virtually formed and used. However, in actual processing, processing may be based on vector operations such as the coordinates of feature points and the direction of straight lines, and may not be based on geometric elements having equivalent effects.

Here, a specific example in which the processes are executed in the order of steps S1, S2, S3, and S4 will be described with reference to FIGS. 7A, 7B, and 8. FIG. FIG. 7A is a diagram schematically showing a photographed image L (L is a natural number) frames before the current frame. FIG. 7B is a diagram schematically showing the current frame. FIG. 8 is a diagram schematically showing a virtually formed quadrangle.

A captured image P1 shown in FIG. 7A and a captured image P2 shown in FIG. 7B are captured by the front camera 21 . The captured images P1 and P2 include an area BO in which the body of the vehicle 5 is reflected. The first feature point FP1 and the second feature point FP2 are present on the road surface RS. In FIGS. 7A and 7B, the first feature point FP1 and the second feature point FP2 are present at the corners of the white lines drawn on the road surface.

The vehicle 5 is traveling straight forward from the time when the photographed image P1 is taken until the time when the photographed image P2 is taken. A circle FP1P shown in FIG. 7B indicates the position of the first feature point FP1 at the time of capturing the captured image P1. A circle FP2P shown in FIG. 7B indicates the position of the second feature point FP2 at the time of capturing the captured image P1.

As shown in FIG. 7B , when the vehicle 5 moves straight ahead, the first feature point FP1 and the second feature point FP2 existing in front of the vehicle 5 approach the vehicle 5 . That is, the first feature point FP1 and the second feature point FP2 appear at different positions in the captured image P2 and the captured image P1. The specifying unit 122 associates the first feature point FP1 of the captured image P2 with the first feature point FP1 of the captured image P1 based on pixel values in the neighborhood of the first feature point FP1, and determines the position of each of the associated first feature points FP1. Based on, the optical flow OF1 of the first feature point FP1 is derived. Similarly, the identifying unit 122 associates the second feature point FP2 of the captured image P2 with the second feature point FP2 of the captured image P1 based on pixel values in the neighborhood of the second feature point FP2, and An optical flow OF2 of the second feature point FP2 is derived based on each position.

When the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2 are derived, the identifying unit 122 applies internal parameters to the front camera 21 stored in the storage unit 13 for the feature points. Coordinate transformation is performed using In the coordinate conversion, aberration correction and distortion correction of the front camera 21 are performed. Aberration correction is performed to correct distortion due to aberration of the optical system of the front camera 21 . Specifically, it is correction of distortion such as barrel distortion and pincushion distortion. Distortion correction is performed to correct distortion of the optical system itself of the front camera 21 . Specifically, it is fisheye correction. Coordinate transformation converts the coordinates of the feature points into coordinates obtained on a two-dimensional image when the subject is photographed by perspective projection.

When the feature point coordinate transformation is performed, the identifying unit 122 sets the starting point of the optical flow OF1 of the first feature point FP1 to the vertex SP1 and the end point of the optical flow OF1 of the first feature point FP1 to the vertex, as shown in FIG. A quadrangle QL is virtually formed in which the starting point of the optical flow OF2 of EP1 and the second feature point FP1 is the vertex SP2, and the end point of the optical flow OF2 of the second feature point FP2 is the vertex EP.

Returning to FIG. 6, step S5 of the flowchart will be described. If it is determined that a quadrangle cannot be virtually formed using only the latest optical flow (NO in step S3), the identifying unit 122 determines the real-world position of the first feature point and the real-world position of the second feature point. can be estimated to be at least a first predetermined distance but less than a second predetermined distance in a predetermined direction, and at the time of photographing the current frame in the real world of the vehicle 5 and at L (L is a natural number) frames before the current frame. and the interval between the photographing point of time N (N is a natural number) frames before the current frame and the photographing point of time M (M is a natural number) frames before the current frame. If it can be estimated that it is less than the distance, a rectangle is virtually formed using the past optical flow and the latest optical flow (step S5). If the above estimation cannot be performed, the flowchart ends without executing steps S5 to S9. In this embodiment, N>M>L for N, M, and L, but M and L may be the same, and L may be larger than M and smaller than N.

Here, a specific example in which the processes are executed in the order of steps S1, S2, S3, and S5 will be described with reference to FIGS. 7C to 7F and 8. FIG. FIG. 7C is a diagram schematically showing a photographed image N frames before the current frame. FIG. 7D is a diagram schematically showing a photographed image M frames before the current frame. FIG. 7E is a diagram schematically showing a photographed image L frames before the current frame. FIG. 7F is a diagram schematically showing the current frame.

A captured image P11 shown in FIG. 7C is an example of a captured image captured at the first time. A captured image P12 shown in FIG. 7D is an example of a captured image captured at a second time after the first time. A captured image P13 shown in FIG. 7E is an example of a captured image captured at a third time after the second time. A captured image P14 shown in FIG. 7F is an example of a captured image captured at a fourth time after the third time.

Photographed images P11 to P14 shown in FIGS. 7C to 7F are photographed by the front camera 21. FIG. The captured images P11 to P14 include an area BO in which the body of the vehicle 5 is reflected. The first feature point FP1 and the second feature point FP2 are present on the road surface RS. In FIGS. 7C and 7D, the first feature point FP1 exists at the corner of the white line drawn on the road surface. In FIGS. 7E and 7F, the second feature point FP2 exists at the corner of the white line drawn on the road surface.

The vehicle 5 is traveling straight forward from the time when the captured image P11 is captured to the time when the captured image P12 is captured. A circle FP1P shown in FIG. 7D indicates the position of the first feature point FP1 at the time of capturing the captured image P11. Similarly, the vehicle 5 is traveling straight forward from the time when the captured image P13 is captured to the time when the captured image P14 is captured. A circle FP2P shown in FIG. 7F indicates the position of the second feature point FP2 at the time of capturing the captured image P13.

As shown in FIG. 7D , when the vehicle 5 moves straight forward, the first feature point FP<b>1 existing in front of the vehicle 5 approaches the vehicle 5 . That is, the first feature point FP1 appears at different positions in the captured image P12 and the captured image P11. The specifying unit 122 associates the first feature point FP1 of the captured image P12 with the first feature point FP1 of the captured image P11 based on pixel values in the vicinity thereof, and determines the position of each of the associated first feature points FP1. Based on, the optical flow OF1 of the first feature point FP1 is derived.

As shown in FIG. 7F , when the vehicle 5 moves straight ahead, the second feature point FP2 existing in front of the vehicle 5 approaches the vehicle 5 . That is, the second feature point FP2 appears at different positions in the captured image P14 and the captured image P13. The specifying unit 122 associates the second feature point FP2 of the captured image P14 with the second feature point FP2 of the captured image P13 based on pixel values in the vicinity thereof, and determines the position of each of the associated second feature points FP2. Based on, the optical flow OF2 of the second feature point FP2 is derived.

When the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2 are derived, the identifying unit 122 applies internal parameters to the front camera 21 stored in the storage unit 13 for the feature points. Coordinate transformation is performed using In the coordinate conversion, aberration correction and distortion correction of the front camera 21 are performed. Aberration correction is performed to correct distortion due to aberration of the optical system of the front camera 21 . Specifically, it is correction of distortion such as barrel distortion and pincushion distortion. Distortion correction is performed to correct distortion of the optical system itself of the front camera 21 . Specifically, it is fisheye correction. Coordinate transformation converts the coordinates of the feature points into coordinates obtained on a two-dimensional image when the subject is photographed by perspective projection.

When the feature point coordinate transformation is performed, the identifying unit 122 sets the starting point of the optical flow OF1 of the first feature point FP1 to the vertex SP1 and the end point of the optical flow OF1 of the first feature point FP1 to the vertex, as shown in FIG. A quadrangle QL is virtually formed in which the starting point of the optical flow OF2 of EP1 and the second feature point FP1 is the vertex SP2, and the end point of the optical flow OF2 of the second feature point FP2 is the vertex EP.

Returning to FIG. 6, the processing after step S5 in the flowchart will be described.

When the quadrangle QL is virtually formed, the specifying unit 122 uses the quadrangle QL and the internal parameters of the front camera 21 stored in the storage unit 13 to determine the projection plane IMG of the front camera 21 in the three-dimensional space. The quadrangle QL is moved upward to virtually generate a quadrangle QL1 on the projection plane IMG (step S6).

For the sake of explanation, the sides of quadrangle QL1 are defined as follows (see FIG. 9). A first side SD1 of quadrangle QL1 corresponds to a side connecting vertex SP1 and vertex SP2 in quadrangle QL. That is, it corresponds to the side connecting the starting point of the optical flow OF1 of the first feature point FP1 and the starting point of the optical flow OF2 of the second feature point FP2. Similarly, the second side SD2 of quadrangle QL1 corresponds to the side connecting vertex SP2 and vertex EP2 in quadrangle QL. That is, it corresponds to the optical flow OF2 of the second feature point FP2. Similarly, the third side SD3 of quadrangle QL1 corresponds to the side connecting vertex EP1 and vertex EP2 in quadrangle QL. That is, it corresponds to the side connecting the end point of the optical flow OF1 of the first feature point FP1 and the end point of the optical flow OF2 of the second feature point FP2. Similarly, the fourth side SD4 of quadrangle QL1 corresponds to the side connecting vertex SP1 and vertex EP1 in quadrangle QL. That is, it corresponds to the optical flow OF1 of the first feature point FP1.

Also, a surface is defined as follows (see FIG. 9). A plane including the first side SD1 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a first plane F1. Similarly, a plane including the second side SD2 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a second plane F2. Similarly, a surface including the third side SD3 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a third surface F3. Similarly, a plane including the fourth side SD4 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a fourth plane F4.

In step S7 following step S6, the identifying unit 122 identifies two sets of planes whose intersection lines with a predetermined plane are parallel to each other. A predetermined plane is a plane whose normal is known in advance. Specifically, it is a plane on which the vehicle 5 is moving, that is, a road surface. The predetermined plane does not have to be a strict plane, as long as it can be regarded as a plane when the specifying unit 122 specifies two sets of planes whose intersection lines with the predetermined plane are parallel to each other. good.

In this embodiment, the first feature point and the second feature point are extracted from a stationary object positioned on a predetermined plane such as a road surface. Therefore, each optical flow of the first feature point and the second feature point derived by the identifying unit 122 represents the relative position change of the stationary object with respect to the vehicle 5 in the real world. That is, it is the movement vector of the vehicle 5 whose direction is reversed.

The second side SD2 and the fourth side SD4 of the quadrangle QL1 both correspond to the optical flow of the feature points, and thus both correspond to the movement vector of the vehicle 5 in the real world. Therefore, it is assumed that they are parallel to each other on the road surface.

Further, since the first side SD1 and the third side SD3 of the quadrangle QL1 are both in the positional relationship between feature points, they correspond to the positional relationship between stationary objects accompanying the movement of the vehicle 5 in the real world. The positional relationship before movement corresponds to the first side SD1, and the positional relationship after movement corresponds to the third side SD3. Since the position of the stationary object does not change at this time, the positional relationship does not change before and after the movement. Therefore, it is assumed that they are also parallel to each other on the road surface.

Therefore, the identification unit 122 has two sets of surfaces having parallel lines of intersection with the road surface: the set of the second surface F2 and the fourth surface F4, and the set of the first surface F1 and the third surface F3. identify. In other words, the specifying unit 122 specifies a total of two pairs of surfaces including optical flows as one pair and surfaces including feature points captured at the same time as another pair.

In addition, in FIG. 9, when the quadrangle QL2 is virtually formed using the optical flow obtained from the captured image P1 shown in FIG. 7A and the captured image P2 shown in FIG. The position of the point FP1 in the three-dimensional space (real world), the position of the second feature point FP2 in the three-dimensional space at the time of capturing the captured image P2, and the first feature point FP1 at the time of capturing the captured image P1. It is a quadrangle whose vertices are the position in the dimensional space and the position in the 3D space of the second feature point FP2 at the time of photographing the photographed image P1.

On the other hand, the quadrangle QL2 in FIG. 9 is a virtual square using the optical flow obtained from the captured image P11 shown in FIG. 7C, the captured image P12 shown in FIG. 7D, the captured image P13 shown in FIG. 7E, and the captured image P14 shown in FIG. , the position of the first feature point FP1 in the three-dimensional space (real world) at the time of capturing the captured image P12, and the position of the second feature point FP2 in the three-dimensional space at the time of capturing the captured image P14. position, the position in the three-dimensional space of the first feature point FP1 at the time of capturing the captured image P11, and the position in the three-dimensional space of the second feature point FP2 at the time of capturing the captured image P13. be.

The first face F1 includes the first side SD11 of the quadrangle QL2. Similarly, the second surface F2 includes the second side SD12 of the quadrilateral QL2, the third surface F3 includes the third side SD13 of the quadrilateral QL2, and the fourth surface F4 includes the fourth side SD14 of the quadrilateral QL2. At this time, the quadrangle QL2 is assumed to be a parallelogram formed on the road surface as described above.

Next, the estimation unit 123 calculates the normal line of the road surface (step S8). First, the estimation unit 123 obtains the direction of the line of intersection between the surfaces based on the first surface F1 and the third surface F3, which are one of the pairs of surfaces identified by the identification unit 122 . Specifically, the direction of the line of intersection CL1 between the first surface F1 and the third surface F3 is obtained (see FIG. 10). A direction vector V1 of the line of intersection CL1 is a vector perpendicular to each of the normal vector of the first surface F1 and the normal vector of the third surface F3. Therefore, the estimation unit 123 obtains the direction vector V1 of the line of intersection CL1 from the cross product of the normal vector of the first surface F1 and the normal vector of the third surface F3. Since the first plane F1 and the third plane F3 are parallel to the road surface, the direction vector V1 is parallel to the road surface.

Similarly, the estimation unit 123 obtains the direction of the line of intersection between the second surface F2 and the fourth surface F4, which are the other of the set of surfaces identified by the identification unit 122 . Specifically, the direction of the intersection line CL2 between the second surface F2 and the fourth surface F4 is obtained (see FIG. 11). A direction vector V2 of the line of intersection CL2 is a vector perpendicular to each of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Therefore, the estimation unit 123 obtains the direction vector V2 of the line of intersection CL2 from the cross product of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Similarly, the second surface F2 and the fourth surface F4 also have parallel lines of intersection with the road surface, so the direction vector V2 is parallel with the road surface.

The estimating unit 123 calculates the normal to the surface of the quadrangle QL2, that is, the normal to the road surface, from the outer product of the direction vector V1 and the direction vector V2. Since the normal line of the road surface calculated by the estimating unit 123 is calculated in the camera coordinate system of the front camera 21, the coordinate system of the three-dimensional space is obtained from the difference from the vertical direction, which is the normal line of the actual road surface. The pose of the camera 21 can be estimated. Based on the estimation result, the estimation unit 123 estimates the posture of the front camera 21 with respect to the vehicle 5 (step S9). Note that the calculation process in step S7 can be executed using, for example, the known ARToolkit.

When the posture estimation of the front camera 21 in step S9 ends, the flow shown in FIG. 6 ends.

The posture estimation device 14 uses the movement of the vehicle 5 to autonomously specify two sets of planes whose intersection lines with a predetermined plane are parallel to each other, so that only the feature point extraction error is the estimation accuracy. can be used to estimate the pose of the camera, which affects the In other words, the posture estimation device 14 can estimate the posture of the camera with few error factors. Therefore, the orientation estimation device 14 can accurately estimate the orientation of the camera.

Further, when it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated by a first predetermined distance or more in a predetermined direction, posture estimation apparatus 14 performs the estimation. The identification process in the unit 122 and the estimation process in the estimation unit 123 are executed. As a result, it is possible to prevent the angle α (α<90°) between the second surface F2 and the fourth surface F4 shown in FIG. 11 from becoming too small. can be prevented from decreasing, and the degree of influence when dot misalignment occurs in each coordinate of the first feature point and the second feature point can be suppressed. As a result, it is possible to prevent the accuracy of camera orientation estimation from deteriorating depending on the position of the first feature point and the position of the second feature point.

For example, as in the captured image shown in FIG. 12A , the extraction unit 121 sets two ROIs 5A and 5B aligned in a predetermined direction in the real world and separated by a first predetermined distance or more in the captured image, and captures each other. The first feature point may be extracted from one of the two ROIs 5A and 5B set at different positions in the image, and the second feature point may be extracted from the other. As a result, the position of the first feature point in the real world and the position of the second feature point in the real world are inevitably separated by the first predetermined distance or more in the predetermined direction. and the real-world position of the second feature point are estimated to be separated by a first predetermined distance or more in a predetermined direction. In other words, the estimation process for estimating whether or not the real-world position of the first feature point and the real-world position of the second feature point are separated by a first predetermined distance or more in a predetermined direction is substantially unnecessary. Become.

Further, as in the captured image shown in FIG. 12B, the extraction unit 121 sets three or more (four in FIG. 12B) ROIs 5C to 5F arranged along a predetermined direction in the real world in the captured image, The first feature point may be extracted from one of two non-adjacent ROIs set at different positions in the captured image, and the second feature point may be extracted from the other. Also in this case, it is necessarily possible to separate the real-world position of the first feature point from the real-world position of the second feature point by the first predetermined distance or more in a predetermined direction. Three or more ROIs are set in the captured image, the first feature points are extracted from one of the two ROIs that are set at different positions in the captured image and are not adjacent to each other, and the second feature points are extracted from the other. When extracting, adjacent ROIs may be in contact with each other.

Further, the specifying unit 122 determines the real-world position of the first feature point and the second feature point based on the distance between the starting point of the optical flow of the first feature point and the starting point of the optical flow of the second feature point on the captured image. It may be estimated whether or not the position of the feature point in the real world is separated by a first predetermined distance or more in a predetermined direction. For example, if the distance between the starting point of the optical flow of the first feature point and the starting point of the optical flow of the second feature point on the captured image is equal to or greater than a threshold, the position of the first feature point in the real world and the second feature point It may be estimated that the position of the point in the real world is separated from the point in a predetermined direction by a first predetermined distance or more.

The distance between the starting point of the optical flow of the first feature point and the starting point of the optical flow of the second feature point on the captured image is, for example, when the captured image is an image captured by the front camera 21 or the back camera 22. The horizontal distance of the image may be used, and if the captured image is an image captured by the front camera 21 or the back camera 22, for example, the horizontal distance of the captured image may be used. In the case of an image shot by the camera 24, for example, the vertical distance of the shot image may be used.

Instead of the distance between the starting point of the optical flow of the first feature point on the captured image and the starting point of the optical flow of the second feature point, or the starting point of the optical flow of the first feature point on the captured image and the second In addition to the distance from the start point of the optical flow of the feature point, the distance between the end point of the optical flow of the first feature point and the end point of the optical flow of the second feature point on the captured image may be used.

Since the relationship between the distance on the captured image and the distance in the real world changes depending on the position of the captured image, the specifying unit 122 determines the position of the starting point of the optical flow of the first feature point on the captured image, At least one of the position of the start point of the optical flow of the second feature point, the position of the end point of the optical flow of the first feature point on the captured image, and the position of the end point of the optical flow of the second feature point on the captured image , it may be estimated whether or not the real-world position of the first feature point and the real-world position of the second feature point are separated from each other by a first predetermined distance or more in a predetermined direction. Thereby, the real-world position of the first feature point and the real-world position of the second feature point are determined only from the optical flow of the first feature point and the optical flow of the second feature point on the captured image. , it can be estimated whether or not they are separated by a first predetermined distance or more in the direction of . For example, the above threshold may be the position of the starting point of the optical flow of the first feature point on the captured image, the position of the starting point of the optical flow of the second feature point on the captured image, and the position of the starting point of the optical flow of the second feature point on the captured image. and the position of the end point of the optical flow of the second feature point on the captured image.

Note that in the present embodiment, when it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated by less than a first predetermined distance in a predetermined direction, the estimation By not executing the specifying process in the unit 122 and the estimation process in the estimating unit 123, it is possible to prevent the accuracy of camera pose estimation from decreasing depending on the position of the first feature point and the position of the second feature point. ing. However, unlike the present embodiment, even if it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are less than the first predetermined distance in a predetermined direction, Even if the specific processing in the estimation unit 122 and the estimation processing in the estimation unit 123 are executed, and, for example, the posture estimation device 14 outputs that the accuracy of the camera posture estimation is low together with the camera posture estimation result. good.

Moreover, even if posture estimation apparatus 14 can estimate that the real-world position of the first feature point and the real-world position of the second feature point are separated from each other by a first predetermined distance or more in a predetermined direction, the first If it can be estimated that the real-world position of the feature point and the real-world position of the second feature point are separated from each other by a second predetermined distance or more in a predetermined direction, the identification processing in the estimation unit 122 and the estimation unit 123 Do not perform the estimation process in . As a result, it is possible to prevent the first feature point and the second feature point from being extracted at positions where the distortion is large in the captured image, so that it is possible to suppress deterioration in the accuracy of estimating the pose of the camera due to the influence of the distortion.

In addition, the posture estimation device 14 calculates the movement amount of the vehicle 5 in the real world when the first feature point moves from the start point to the end point of the optical flow, and the movement amount when the second feature point moves from the start point to the end point of the optical flow. When it can be estimated that the amount of movement of the vehicle 5 in the real world is equal to or greater than the third predetermined distance, the specific process in the estimating section 122 and the estimating process in the estimating section 123 are executed. As a result, it is possible to prevent the angle β (β<90°) between the first surface F1 and the third surface F3 shown in FIG. 10 from becoming too small. can be prevented from decreasing, and the degree of influence when dot misalignment occurs in each coordinate of the first feature point and the second feature point can be suppressed. As a result, it is possible to prevent the accuracy of camera orientation estimation from deteriorating depending on the position of the first feature point and the position of the second feature point.

For example, when the speed of the vehicle 5 is equal to or less than the first predetermined speed, the specifying unit 122 determines the movement amount of the vehicle 5 in the real world when the first feature point moves from the start point to the end point of the optical flow, and the second It can be estimated that the amount of movement of the vehicle 5 in the real world when the feature point moves from the start point to the end point of the optical flow is equal to or greater than the third predetermined distance. This facilitates the estimation. The speed of the vehicle 5 may be a measured value detected by the vehicle speed sensor 4, or an estimated speed estimated from the optical flow at the first feature point and the optical flow at the second feature point.

For example, when the speed of the vehicle 5 is equal to or lower than the first predetermined speed, the extraction unit 121 increases the value of L in the above-described L frame compared to when the speed of the vehicle 5 is higher than the first predetermined speed. The difference between M and N in the resulting M frame and N frame may be made larger than when the speed of the vehicle 5 is higher than the first predetermined speed. As a result, the movement amount of the vehicle 5 in the real world when the first feature point moves from the start point to the end point of the optical flow and the real world movement amount of the vehicle 5 when the second feature point moves from the start point to the end point of the optical flow It is possible to increase the number of cases where it can be estimated that each of the above movement amounts is equal to or greater than the third predetermined distance.

In addition, based on the size of the optical flow of the first feature point on the captured image, the identifying unit 122 determines the movement amount of the vehicle 5 in the real world when the first feature point moves from the start point to the end point of the optical flow. and the movement amount of the vehicle 5 in the real world when the second feature point moves from the start point to the end point of the optical flow, respectively, may be estimated as to whether or not it is equal to or greater than a third predetermined distance. For example, if the magnitude of the optical flow of the first feature point on the captured image is equal to or greater than the threshold, the amount of movement of the vehicle 5 in the real world when the first feature point moves from the start point to the end point of the optical flow and It can be estimated that the amount of movement of the vehicle 5 in the real world when the second feature point moves from the start point to the end point of the optical flow is equal to or greater than the third predetermined distance.

Instead of the magnitude of the optical flow of the first feature point on the captured image, or in addition to the magnitude of the optical flow of the first feature point on the captured image, the magnitude of the second feature point on the captured image The magnitude of optical flow may also be used.

Since the relationship between the distance on the captured image and the distance in the real world changes depending on the position of the captured image, the specifying unit 122 determines the position of the starting point of the optical flow of the first feature point on the captured image, At least one of the position of the start point of the optical flow of the second feature point, the position of the end point of the optical flow of the first feature point on the captured image, and the position of the end point of the optical flow of the second feature point on the captured image Based on, the amount of movement of the vehicle 5 in the real world when the first feature point moves from the start point to the end point of the optical flow and the real world of the vehicle 5 when the second feature point moves from the start point to the end point of the optical flow It may be estimated whether or not each of the above movement amounts is equal to or greater than a third predetermined distance. As a result, only the optical flow of the first feature point and the optical flow of the second feature point on the captured image can be used to determine the movement of the vehicle 5 in the real world when the first feature point moves from the start point to the end point of the optical flow. It is possible to estimate whether or not the movement amount of the vehicle 5 in the real world when the amount and the second feature point move from the start point to the end point of the optical flow are each a third predetermined distance or more. For example, the above threshold may be the position of the starting point of the optical flow of the first feature point on the captured image, the position of the starting point of the optical flow of the second feature point on the captured image, and the position of the starting point of the optical flow of the second feature point on the captured image. and the position of the end point of the optical flow of the second feature point on the captured image.

Note that in the present embodiment, the amount of movement of the vehicle 5 in the real world when the first feature point moves from the start point to the end point of the optical flow and the movement amount of the vehicle 5 when the second feature point moves from the start point to the end point of the optical flow 5 can be estimated to be less than the third predetermined distance, the position of the first feature point And, it suppresses the deterioration of the accuracy of the pose estimation of the camera depending on the position of the second feature point. However, unlike the present embodiment, the movement amount in the real world of the vehicle 5 when the first feature point moves from the start point to the end point of the optical flow and the movement amount when the second feature point moves from the start point to the end point of the optical flow Even if it can be estimated that the amount of movement of the vehicle 5 in the real world is less than the third predetermined distance, the specific processing in the estimating unit 122 and the estimation processing in the estimating unit 123 are executed, The posture estimation device 14 may output the fact that the accuracy is low together with the camera posture estimation result.

In addition, the posture estimation device 14 calculates the movement amount of the vehicle 5 in the real world when the first feature point moves from the start point to the end point of the optical flow, and the movement amount when the second feature point moves from the start point to the end point of the optical flow. Even if it can be estimated that the movement amount of the vehicle 5 in the real world is less than the third predetermined distance, the movement amount in the real world of the vehicle 5 when the first feature point moves from the start point to the end point of the optical flow and the third predetermined distance. When it can be estimated that the amount of movement of the vehicle 5 in the real world when the two feature points move from the start point to the end point of the optical flow is equal to or greater than the fourth predetermined distance, the specific processing in the estimation unit 122 and the estimation unit 123 Do not perform the estimation process in . As a result, the end point of the optical flow of the first feature point and the end point of the optical flow of the second feature point do not fall within the captured image, and the end point of the optical flow of the first feature point and the end point of the optical flow of the second feature point are It is possible to prevent the problem of being unable to extract.

Also, as described above, posture estimation apparatus 14 virtually forms a quadrangle using the past optical flow and the latest optical flow when a quadrangle cannot be virtually formed using only the latest optical flow. Therefore, the posture estimation device 14 has a configuration in which a quadrangle cannot be virtually formed using only the latest optical flow, that is, a configuration in which steps S3 and S5 are removed from the flowchart shown in FIG. It is possible to suppress the lengthening of the time required for posture estimation. In the present embodiment, it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated from each other by a first predetermined distance or more and less than a second predetermined distance in a predetermined direction. and the amount of movement of the vehicle 5 between the time of photographing the current frame in the real world and the time of photographing L (L is a natural number) frames before the current frame is greater than or equal to a third predetermined distance and less than a fourth predetermined distance. Since step S4 is not executed unless the condition that the estimation can be performed is satisfied, the above-described effect of suppressing the lengthening of the time required for estimating the orientation of the camera increases.

However, in the case of virtually forming a quadrangle using the past optical flow and the latest optical flow, if an appropriate optical flow is not used, compared to the case of virtually forming a quadrangle with only the latest optical flow, The accuracy of camera pose estimation may be significantly degraded.

Whether the optical flow used in the process of step S5 shown in FIG. 6 is appropriate depends on how the vehicle 5 moves. Therefore, when executing the process of step S5 shown in FIG. 6, it is desirable that the combination of the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point is determined based on the movement information of the vehicle 5. .

As described above, the posture estimation device 14 estimates the posture of the camera on the assumption that the quadrangle QL2 is a parallelogram formed on the road surface.

The smaller the difference between the steering angle of the vehicle 5 when the optical flow OP1 at the first feature point occurs and the steering angle of the vehicle 5 when the optical flow OP2 at the second feature point occurs, the closer the difference between the above assumption and reality. becomes smaller. Therefore, when the steering angle information of the vehicle 5 is included in the movement information of the vehicle 5 described above and the process of step S5 shown in FIG. is preferably determined based on the steering angle information of the vehicle 5 . Hereinafter, this combination determination method will be referred to as a first determination method. For example, if the output of the steering angle sensor 3, which is supplied from the steering angle sensor 3 to the attitude estimation device 14 via the communication bus B1, is used as the steering angle information of the vehicle 5, the steering angle information of the vehicle 5 can be used to estimate the steering angle. Since it is actual measurement information of the steering angle instead of information, the accuracy of the steering angle information of the vehicle 5 can be improved.

For example, the first characteristic is set so that the difference between the steering angle of the vehicle 5 at the time of capturing the captured image P12 shown in FIG. 7D and the steering angle of the vehicle 5 at the time of capturing the captured image P14 shown in FIG. 7F is within a predetermined angle. A combination of the point optical flow OP1 and the second feature point optical flow OP2 may be determined. Further, for example, the average value of the steering angle of the vehicle 5 from the photographing time of the photographed image P11 shown in FIG. 7C to the photographing time of the photographed image P12 shown in FIG. The optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point are combined so that the difference between the average value of the steering angle of the vehicle 5 and the average value of the steering angle of the vehicle 5 up to the time when the photographed image P14 shown in FIG. may decide.

The smaller the difference between the speed of the vehicle 5 when the optical flow OP1 at the first feature point occurs and the speed of the vehicle 5 when the optical flow OP2 at the second feature point occurs, the greater the divergence between the above assumption and reality. becomes smaller. Therefore, when the speed information of the vehicle 5 is included in the movement information of the vehicle 5 described above and the process of step S5 shown in FIG. The combination is desirably determined based on speed information of the vehicle 5 . Hereinafter, this combination determination method will be referred to as a second determination method. For example, if the output of the vehicle speed sensor 4, which is supplied from the vehicle speed sensor 4 to the attitude estimation device 14 via the communication bus B1, is used as the speed information of the vehicle 5, the speed information of the vehicle 5 is not speed estimation information. Since it becomes actual measurement information of the speed, the accuracy of the speed information of the vehicle 5 can be improved.

For example, the first feature point is set so that the difference between the speed of the vehicle 5 at the time of capturing the captured image P12 shown in FIG. 7D and the speed of the vehicle 5 at the time of capturing the captured image P14 shown in FIG. 7F is within a predetermined speed. A combination of the optical flow OP1 and the optical flow OP2 of the second feature point may be determined. Further, for example, the average value of the speed of the vehicle 5 from the time of capturing the captured image P11 shown in FIG. 7C to the time of capturing the captured image P12 shown in FIG. 7D and the average value of the speed of the vehicle 5 from the time of capturing the captured image P13 shown in FIG. A combination of the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point is determined so that the difference from the average value of the speed of the vehicle 5 up to the time of shooting the shot image P14 is within a predetermined speed. may

If the time difference between the occurrence of the optical flow OP1 at the first feature point and the occurrence of the optical flow OP1 at the second feature point is short, it can be expected that the movement of the vehicle 5 will change little. Therefore, when executing the process of step S5 shown in FIG. 6, the combination of the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point is the time point of the taken image P11 shown in FIG. 7C and the time point of the taken image P11 shown in FIG. and the time associated with at least one of the photographing times of the photographed image P13 shown in FIG. 7E and the photographing time of the photographed image P14 shown in FIG. 7F. It is desirable that Hereinafter, this combination determination method will be referred to as a third determination method.

Examples of the time related to at least one of the time point of the captured image P11 shown in FIG. 7C and the time point of the captured image P12 shown in FIG. Examples include the time when the captured image P12 is captured, and the midpoint time between the captured image P11 shown in FIG. 7C and the captured image P12 shown in FIG. 7D. Similarly, as the time related to at least one of the time of capturing the captured image P13 shown in FIG. 7E and the time of capturing the captured image P14 shown in FIG. 7F, for example, the time of capturing the captured image P13 shown in FIG. Examples include the time when the captured image P14 shown in 7F is captured, and the midpoint time between the time when the captured image P13 shown in FIG. 7E and the captured image P14 shown in FIG. 7F are captured.

The above-described first determination method, second determination method, and third determination method may be implemented independently, or any two or more of them may be combined.

Further, since the optical flows of the first feature point and the second feature point are movement vectors of the vehicle 5 whose direction is reversed, the movement information described above is converted to the optical flow of the first feature point and the second feature point. Information estimated based on the result of coordinate transformation of the flow onto the road surface may be used. The speed of the vehicle 5 can be estimated by transforming the coordinates of the optical flows of the first feature point and the second feature point onto the road surface, and calculating the length on the road surface. The steering angle of the vehicle 5 can be estimated. That is, it can be implemented as an alternative to the above-described first determination method and second determination method. If the movement information described above is information estimated based on the results of coordinate transformation of the respective optical flows of the first feature points and the second feature points on the road surface, there is no need to connect the attitude estimation device 14 and the communication bus B1. become.

Further, the specifying unit 122 determines the speed of the vehicle 5 when the optical flow OP1 of the first feature point occurs (it may be the actual measurement value detected by the vehicle speed sensor 4 or the estimated speed described above). and the speed of the vehicle 5 when the optical flow OP2 of the second feature point occurs (it may be the actual measured value detected by the vehicle speed sensor 4 or the estimated speed described above), At least one of the optical flow OP1 of the first feature point and the optical flow OP2 of the second feature point is corrected, and based on each optical flow of the first feature point and the second feature point, at least one of which is corrected, the intersection with the road surface It is desirable to identify two sets of planes whose lines are parallel to each other. As a result, the accuracy of camera orientation estimation can be improved.

For example, if the speed of the vehicle 5 when the optical flow OP1 at the first feature point occurs is greater than the speed of the vehicle 5 when the optical flow OP2 at the second feature point occurs, the optical flow OP1 at the first feature point is The optical flow OP2 of the second feature point may be corrected to be shorter, and the optical flow OP2 of the second feature point may be corrected to be longer, the optical flow OP1 of the first feature point is shortened, and the optical flow OP2 of the second feature point is longer. can be corrected as follows.

For example, when the road surface RS is made of concrete, it is difficult to extract feature points. For this reason, the feature point extraction error tends to increase. However, for example, when dirt exists on the road surface RS made of concrete, if the first feature point FP1 and the second feature point FP2 are extracted from the dirt portion, the feature points with high feature degrees are the first feature point FP1 and the second feature point FP2. It is considered that the extraction error of the first feature point FP1 and the second feature point FP2 can be suppressed by selecting the second feature point FP2.

When there is a high-density area in which feature points are more concentrated than others, regardless of dirt, by extracting the first feature point FP1 and the second feature point FP2 from the high-density area, It is considered that extraction errors of the first feature point FP1 and the second feature point FP2 can be suppressed. Therefore, when there is a high-density area where feature points are more concentrated than others, even if the extraction unit 121 extracts the first feature point FP1 and the second feature point FP2 from the high-density area, good.

In addition, the extraction error is small when extracting feature points with a high degree of cornering. Therefore, the first feature point and the second feature point may be specific feature points whose corner degree indicating cornerness is equal to or greater than a predetermined corner degree threshold value. A corner is where two edges meet. At corners, the degree of cornering increases. The corner degree can be determined using a known detection method such as Harris operator or KLT (Kanade-Lucas-Tomasi) tracker.

In the above description, the plane including the optical center OC of the front camera 21 and one side of the quadrangle QL1 is specified, but this is not the only option. A plane may be specified by specifying the normal direction of the plane. For example, the normal direction of the plane may be obtained from the outer product of the direction vectors from the optical center OC to each vertex, and the plane may be identified by the normal direction. In this case, the normal direction of the first surface F1 and the normal direction of the third surface F3 are set as one set, and the normal direction of the second surface F2 and the normal direction of the fourth surface F4 are set as another set. Two sets may be identified.

Also, the plane may be translated. For example, instead of the first plane F1, a plane translated from the first plane F1 may be paired with the third plane F3. This is because the direction of the line of intersection with a predetermined plane does not change even if it is translated.

<3. Correction system>
FIG. 13 is a block diagram showing the configuration of the correction system SYS2 according to the embodiment. In FIG. 13, the same parts as those in FIG. 1 are denoted by the same reference numerals, and detailed description thereof will be omitted. As shown in FIG. 13, the correction system SYS2 includes a correction device 1' and an imaging unit 2. FIG.

The correction device 1' has the same configuration as that of the abnormality detection device 1 shown in FIG. That is, the correction device 1 ′ includes the posture estimation device 14 and the correction section 125 .

The correction unit 125 corrects the parameters of the camera based on the camera orientation estimated by the estimation unit 123 . As a result, even if there is a camera mounting misalignment, the camera is calibrated according to the misalignment, and the adverse effects of the camera mounting misalignment on the captured image can be suppressed.

<4. Notes>
The configurations of the embodiments and examples herein are merely illustrative of the present invention. The configurations of the embodiments and modifications may be changed as appropriate without departing from the technical idea of the present invention. Also, multiple embodiments and modifications may be implemented in combination within a possible range.

INDUSTRIAL APPLICABILITY As in the above-described embodiments, the present invention can be used for estimating the orientation of a camera that assists driving of a moving object such as automatic parking. The present invention can also be used to estimate the orientation of a camera that records driving information, such as a drive recorder. Moreover, the present invention can be used in a correction device or the like that corrects a photographed image using the estimation information of the orientation of the camera, as in the above-described embodiments. Further, the present invention can be used for a device or the like that operates in cooperation with a center that is provided so as to be able to communicate with a plurality of moving bodies via a network. For example, when transmitting a photographed image to the center, the apparatus may be configured to transmit the abnormal information of the camera and the estimation information of the posture together with the photographed image. Then, at the center, using the estimated camera posture, various image processing (processing to change the viewpoint and viewing direction of the image considering the camera posture, such as changing the viewpoint and viewing direction to an image in front of the vehicle body) image generation, etc.), correction processing for camera posture in measurement processing using images, statistical processing of changes in camera posture over time (data from many vehicles), etc. to generate useful data to be provided to users. equal.

In the embodiment described above, it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated from each other by a first predetermined distance or more and less than a second predetermined distance in a predetermined direction, In addition, the amount of movement of the vehicle 5 between the time of photographing the current frame in the real world and the time of photographing L (L is a natural number) frames before the current frame is greater than or equal to a third predetermined distance and less than a fourth predetermined distance. If the condition that estimation is possible is not satisfied, step S4 is not executed.

However, even if it satisfies the condition that it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated from each other by a first predetermined distance or more and less than a second predetermined distance in a predetermined direction. Then, step S4 may be executed. Further, if the amount of movement between the time of photographing the current frame in the real world of the vehicle 5 and the time of photographing L (L is a natural number) frames before the current frame is greater than or equal to a third predetermined distance and less than a fourth predetermined distance. Step S4 may be executed as long as the condition that estimation is possible is satisfied. Similar modifications are possible for the execution of step S5. Compared to the above-described embodiment, these modified examples have a smaller effect of suppressing a decrease in posture estimation accuracy. can be suppressed.

In addition, compared to the above-described embodiment, the effect of suppressing deterioration of posture estimation accuracy is small, but the condition regarding the second predetermined distance may not be provided. Similarly, the condition regarding the fourth predetermined distance may not be provided.

1 Abnormality Detection Device 1' Correction Device 14 Posture Estimation Device 21-24 In-Vehicle Camera 11 Acquisition Unit 121 Extraction Unit 122 Identification Unit 123 Estimation Unit 124 Judgment Unit 125 Correction Unit

Claims (15)

an acquisition unit that acquires a captured image captured by a camera mounted on a mobile object;
A feature point is extracted from each of the two captured images captured at a first time and a second time after the first time, and a feature point is extracted at a third time after the first time and a fourth time after the third time. an extraction unit that extracts feature points from each of the two captured images captured at the same time;
the first time and the second time when it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated by a first predetermined distance or more in a predetermined direction; selecting the first feature points corresponding to each other from the feature points extracted from the two captured images captured in , calculating the position change of the first feature points, and calculating the third time and the fourth time Selecting the second feature points corresponding to each other, unlike the first feature points, from the feature points extracted from the two captured images captured in , calculating the positional change of the second feature points, a specifying unit that specifies two sets of planes whose lines of intersection with a predetermined plane are parallel to each other based on the positional change of the first feature point and the positional change of the second feature point;
an estimation unit that estimates the orientation of the camera based on the set of planes identified by the identification unit;
A posture estimation device comprising: When the identifying unit can estimate that the real-world position of the first feature point and the real-world position of the second feature point are separated from each other by less than the first predetermined distance in the predetermined direction 2. The posture estimation apparatus according to claim 1, wherein a set of planes whose intersection lines with said predetermined plane are parallel to each other is not specified. The extraction unit sets a plurality of regions aligned in the predetermined direction in the real world in the captured image, and selects one of the two regions set at different positions in the captured image from the first region. extracting one feature point and extracting the second feature point from the other;
The identifying unit estimates that the real-world position of the first feature point and the real-world position of the second feature point are separated by the first predetermined distance or more in the predetermined direction. 2. The posture estimation device according to 1. The identification unit
at least one of the position of the first feature point on the captured image captured at the first time and the position of the second feature point on the captured image captured at the third time; and based on the distance between the position of the first feature point on the captured image captured at the first time and the position of the second feature point on the captured image captured at the third time, and / or
at least one of the position of the first feature point on the captured image captured at the second time and the position of the second feature point on the captured image captured at the fourth time; and Based on the distance between the position of the first feature point on the captured image captured at the second time and the position of the second feature point on the captured image captured at the fourth time,
3. estimating whether or not the real-world position of the first feature point and the real-world position of the second feature point are separated by the first predetermined distance or more in the predetermined direction. Item 3. The posture estimation device according to item 2. Even if the identifying unit can estimate that the real-world position of the first feature point and the real-world position of the second feature point are separated by the first predetermined distance or more in the predetermined direction, When it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated by a second predetermined distance longer than the first predetermined distance in the predetermined direction. 5. The posture estimation apparatus according to claim 1, wherein does not specify a set of planes whose intersection lines with said predetermined plane are parallel to each other. an acquisition unit that acquires a captured image captured by a camera mounted on a mobile object;
A feature point is extracted from each of the two captured images captured at a first time and a second time after the first time, and a feature point is extracted at a third time after the first time and a fourth time after the third time. an extraction unit that extracts feature points from each of the two captured images captured at the same time;
A moving amount from the first time to the second time in the real world and a moving amount from the third time to the fourth time in the real world are each a third predetermined distance. When it can be estimated that the above is the case, a first feature point corresponding to each other is selected from the feature points extracted from the two captured images captured at the first time and the second time, and the first feature A change in the position of the point is calculated, and from the feature points extracted from the two photographed images photographed at the third time and the fourth time, a second feature point corresponding to each other, unlike the first feature point. and calculate the positional change of the second feature point, and based on the positional change of the first feature point and the positional change of the second feature point, the plane whose intersection line with a predetermined plane is parallel to each other a specifying unit that specifies two pairs;
an estimation unit that estimates the orientation of the camera based on the set of planes identified by the identification unit;
A posture estimation device comprising: The specifying unit determines that the amount of movement of the mobile body from the first time to the second time in the real world and the amount of movement of the mobile body from the third time to the fourth time in the real world are different. 7. The posture estimation apparatus according to claim 6, wherein when it can be estimated that each distance is less than the third predetermined distance, a set of planes whose intersection lines with the predetermined plane are parallel to each other is not specified. When the speed of the moving body is equal to or less than a first predetermined speed, the specifying unit determines a movement amount of the moving body from the first time to the second time in the real world and 7 . The posture estimation apparatus according to claim 6 , wherein the movement amount from the third time to the fourth time is estimated to be less than the third predetermined distance. when the speed of the moving object is equal to or lower than the first predetermined speed,
The extraction unit lengthens the interval between the first time and the second time and the interval between the third time and the fourth time compared to when the speed of the moving object is higher than the first predetermined speed. ,
The specifying unit determines that the amount of movement of the mobile body from the first time to the second time in the real world and the amount of movement of the mobile body from the third time to the fourth time in the real world are different. 7. The posture estimation device according to claim 6, wherein each distance is estimated to be equal to or greater than the third predetermined distance. The identification unit
at least one of the position of the first feature point on the captured image captured at the first time and the position of the first feature point on the captured image captured at the second time; and based on the distance between the position of the first feature point on the captured image captured at the first time and the position of the first feature point on the captured image captured at the second time, and ,
at least one of the position of the second feature point on the captured image captured at the third time and the position of the second feature point on the captured image captured at the fourth time; and Based on the distance between the position of the second feature point on the captured image captured at the third time and the position of the second feature point on the captured image captured at the fourth time,
A moving amount from the first time to the second time in the real world and a moving amount from the third time to the fourth time in the real world are each a third predetermined distance. 8. The posture estimation device according to claim 6, which estimates whether or not the posture is above. The specifying unit determines that the amount of movement of the mobile body from the first time to the second time in the real world and the amount of movement of the mobile body from the third time to the fourth time in the real world are different. Even if it can be estimated that they are at least the third predetermined distance, the moving amount from the first time in the real world to the second time in the real world and the movement amount from the third time in the real world to the moving amount If it can be estimated that the amount of movement up to the fourth time is equal to or greater than the fourth predetermined distance, which is longer than the third predetermined distance, the set of planes whose intersection lines with the predetermined plane are parallel to each other is not specified. A posture estimation device according to any one of claims 6 to 10. a posture estimation device according to any one of claims 1 to 11;
a determination unit that determines whether or not there is a mounting deviation of the camera based on the orientation of the camera estimated by the estimation unit;
An anomaly detection device. a posture estimation device according to any one of claims 1 to 11;
a correction unit that corrects parameters of the camera based on the orientation of the camera estimated by the estimation unit;
a compensator. an acquisition step of acquiring a photographed image photographed by a camera mounted on a mobile body;
A feature point is extracted from each of the two captured images captured at a first time and a second time after the first time, and a feature point is extracted at a third time after the first time and a fourth time after the third time. an extracting step of extracting feature points from each of the two captured images captured at the same time;
the first time and the second time when it can be estimated that the real-world position of the first feature point and the real-world position of the second feature point are separated by a first predetermined distance or more in a predetermined direction; selecting the first feature points corresponding to each other from the feature points extracted from the two captured images captured in , calculating the position change of the first feature points, and calculating the third time and the fourth time Selecting the second feature points corresponding to each other, unlike the first feature points, from the feature points extracted from the two captured images captured in , calculating the positional change of the second feature points, an identifying step of identifying two sets of planes whose lines of intersection with a predetermined plane are parallel to each other based on the positional change of the first feature point and the positional change of the second feature point;
an estimating step of estimating the orientation of the camera based on the set of planes identified in the identifying step;
A pose estimation method comprising: an acquisition step of acquiring a photographed image photographed by a camera mounted on a mobile body;
A feature point is extracted from each of the two captured images captured at a first time and a second time after the first time, and a feature point is extracted at a third time after the first time and a fourth time after the third time. an extracting step of extracting feature points from each of the two captured images captured at the same time;
A moving amount from the first time to the second time in the real world and a moving amount from the third time to the fourth time in the real world are each a third predetermined distance. When it can be estimated that the above is the case, a first feature point corresponding to each other is selected from the feature points extracted from the two captured images captured at the first time and the second time, and the first feature A change in the position of the point is calculated, and from the feature points extracted from the two photographed images photographed at the third time and the fourth time, a second feature point corresponding to each other, unlike the first feature point. and calculate the positional change of the second feature point, and based on the positional change of the first feature point and the positional change of the second feature point, the plane whose intersection line with a predetermined plane is parallel to each other an identifying step of identifying two pairs;
an estimating step of estimating the orientation of the camera based on the set of planes identified in the identifying step;
A pose estimation method comprising:

Images