JP7303064B2 - Image processing device and image processing method - Google Patents

Description

The present invention relates to an image processing device and an image processing method.

2. Description of the Related Art Conventionally, there is known a technique capable of obtaining the current mounting angle of an in-vehicle camera without using a figure whose shape and dimensions are known (see, for example, Patent Document 1).

The method disclosed in Patent Literature 1 includes the steps of capturing an image of the surroundings of the vehicle with a camera attached to the vehicle, detecting a motion vector based on a road surface image in the image captured by the camera, and detecting The motion vector is set around each of the three axes of the traveling direction axis of the vehicle, the road surface normal direction axis, and the road surface parallel direction axis orthogonal to the traveling direction axis and the road surface normal direction axis so that the motion vector becomes the motion vector of the directly above viewpoint. A step of creating a conversion formula to the motion vector of the top viewpoint, which is a formula for independent rotation correction, and a step of obtaining an installation angle of the camera with respect to the vehicle from the conversion formula to the motion vector of the top viewpoint. .

Note that the step of detecting the motion vector includes the step of determining whether the road surface image in the image captured by the camera is a road surface image including the feature points, and the step of determining whether the road surface image is the road surface image including the feature points. Once determined, detecting a motion vector based on the road surface image containing this feature point.

JP 2013-74325 A

By the way, when extracting feature points from an image captured by a camera, there is an advantage that a larger number of feature points can be extracted by extracting feature points from a wider range of the image. However, if the extraction range of the feature points is simply set wide, for example, when it is desired to extract the target feature points (feature points on the road surface), it may be difficult to extract the feature points from other vehicles on the road surface. There is concern that the existence of the object increases the processing load and reduces the accuracy of feature point extraction.

In addition, if the extraction range of feature points is simply set wide, when the speed of a vehicle equipped with a camera increases, the accuracy of feature point extraction decreases on the closer side of the captured image due to the effects of image blur, for example. may decline. Further, lens distortion tends to increase at the outer edge of the lens, and the accuracy of extracting feature points tends to be lower on the outer edge side of the captured image than on the central side. In addition to the influence of such a tendency, there is a possibility that the extraction accuracy of feature points on the outer edge side of the photographed image may decrease when the speed of the vehicle on which the camera is mounted increases.

In view of the above problems, it is an object of the present invention to provide a technique capable of extracting as many feature points as possible from an image captured by a camera mounted on a mobile object while increasing the accuracy of extracting desired feature points. aim.

In order to achieve the above object, the image processing apparatus of the present invention includes an acquisition unit that acquires a photographed image from a camera mounted on a mobile body, and an extraction unit that extracts feature points from an extraction region set in the photographed image. , and a setting unit that sets and changes the extraction area based on the external environment of the moving object in the shooting direction of the camera (first configuration).

The image processing apparatus having the first configuration further includes a storage unit that stores a reference extraction region as data, and the setting unit changes the reference extraction region to set the extraction region (second extraction region). configuration).

Further, in the image processing apparatus having the second configuration, the setting unit changes the reference extraction area based on the position of an object existing away from the moving body in the photographing direction of the camera to determine the extraction area. A setting configuration (third configuration) is preferable.

Further, in the image processing apparatus having the third configuration, the setting unit narrows the width of the reference extraction area in the direction toward the moving object in the captured image when the object exists within a predetermined distance. or setting the extraction region by increasing the width of the reference extraction region in the direction away from the moving object in the captured image when the object does not exist within a predetermined distance. A configuration (fourth configuration) is preferred.

Further, in the image processing apparatus having any one of the first to fourth configurations, the setting unit is configured to set and change the extraction region based on the speed of the moving object (fifth configuration). is preferred.

Further, the image processing apparatus having the fifth configuration described above includes the storage section, and the setting section moves the reference point in the direction away from the moving body in the captured image when the speed of the moving body becomes equal to or higher than a first speed. setting the extraction region by narrowing the width of the extraction region, or widening the width of the reference extraction region in the direction toward the moving object in the captured image when the speed of the moving object is lower than a first speed; It is preferable that the configuration (sixth configuration) sets the extraction region by

Further, the image processing apparatus having the fifth or sixth configuration includes the storage unit, and the setting unit changes the direction of moving away from the moving body in the captured image when the speed of the moving body becomes equal to or higher than a second speed. When the width of the reference extraction region in the direction intersecting with is narrowed to set the extraction region, or when the speed of the moving body becomes smaller than a second speed, the direction away from the moving body in the captured image and the A configuration (seventh configuration) may be employed in which the width of the reference extraction region in the intersecting direction is increased to set the extraction region.

Further, the image processing apparatus having any one of the first to seventh configurations may further include an estimating unit that estimates the orientation of the camera based on the feature points (eighth configuration).

Further, in order to achieve the above object, the image processing method of the present invention includes an acquisition step of acquiring a photographed image from a camera mounted on a moving object, and an extraction step of extracting feature points from an extraction region set in the photographed image. and a setting step of setting and changing the extraction area based on the external environment of the moving body in the photographing direction of the camera (ninth configuration).

According to the present invention, it is possible to extract as many feature points as possible from an image captured by a camera mounted on a mobile object while increasing the accuracy of extracting desired feature points.

A diagram for explaining the configuration of an image processing apparatus. 3 is a flow chart showing an example of camera posture estimation processing executed in an image processing device; Diagram for explaining extraction of feature points Diagram for explaining derivation of optical flow A diagram showing a rectangle formed virtually from optical flow FIG. 4 is a diagram for explaining a technique for identifying two sets of planes whose intersection lines with a predetermined plane are parallel to each other; Diagram for explaining the method of determining the direction of the line of intersection between the faces based on one of the identified pairs of faces. Diagram for explaining the method of determining the direction of the line of intersection between the faces based on the other of the identified faces FIG. 1 for explaining the setting and change of the extraction area by the setting unit FIG. 2 for explaining the setting and change of the extraction area by the setting unit FIG. 3 for explaining the setting and change of the extraction area by the setting unit FIG. 4 for explaining the setting and change of the extraction area by the setting unit 4 is a flowchart showing an example of extraction region setting processing in an image processing apparatus; Diagram for capturing and explaining the flow chart of FIG.

Exemplary embodiments of the invention are described in detail below with reference to the drawings. In the following description, a vehicle is taken as an example of a moving object, but the moving object is not limited to a vehicle. Vehicles broadly include vehicles with wheels, such as automobiles, trains, and automatic guided vehicles. Examples of mobile objects other than vehicles include ships and aircraft.

Further, in the following description, the direction in which the vehicle travels straight ahead, which is the direction from the driver's seat to the steering wheel, is referred to as the "forward direction." In addition, the direction from the steering wheel to the driver's seat, which is the direction in which the vehicle travels straight, is defined as the "rearward direction." In addition, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and from the right side to the left side of the driver who is facing forward is referred to as the "left direction." Also, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and from the left side to the right side of the driver who is facing forward is referred to as the "right direction."

<1. Overview of Configuration of Image Processing Apparatus>
FIG. 1 is a diagram for explaining the configuration of an image processing apparatus 1 according to an embodiment of the invention. FIG. 1 also shows a photographing unit 2 for inputting information to the image processing device 1, a steering angle sensor 3, and a speed sensor 4. FIG. The image processing device 1 is provided for each vehicle in which the photographing unit 2 is mounted. Hereinafter, the vehicle provided with the image processing device 1 may be referred to as the own vehicle.

The photographing unit 2 is provided in the vehicle for the purpose of monitoring the situation outside the vehicle. The imaging unit 2 has at least one camera 21 . The camera 21 is a so-called in-vehicle camera and is fixedly arranged in the vehicle. The camera 21 is connected to the image processing device 1 by wire or wirelessly, and outputs a captured image to the image processing device 1 .

In addition, when the photographing unit 2 is provided for the purpose of monitoring the situation around the vehicle, the photographing unit 2 preferably includes a front camera, a back camera, a left side camera, and a right side camera. A camera 21 is provided. The front camera is a camera that captures the front of the vehicle. A back camera is a camera that photographs the rear of a vehicle. The left side camera is a camera that captures the left side of the vehicle. The right side camera is a camera that captures the right side of the vehicle. These four cameras 21 are configured using, for example, fish-eye lenses, and have a horizontal angle of view θ of 180 degrees or more. Accordingly, the four cameras 21 can photograph the entire circumference of the vehicle in the horizontal direction.

The steering angle sensor 3 is provided in a vehicle in which the image processing device 1 and the photographing unit 2 are mounted, and detects the rotation angle of the steering wheel (steering wheel) of the vehicle. The output of the steering angle sensor 3 is input to the image processing device 1 via a communication bus B1 such as a CAN (Controller Area Network) bus.

A speed sensor 4 is provided in a vehicle in which the image processing device 1 and the photographing unit 2 are mounted, and detects the speed of the vehicle. The output of speed sensor 4 is input to image processing apparatus 1 via communication bus B1.

The image processing device 1 is a posture estimation device that estimates the posture of the camera 21 in this embodiment. The mounting posture of the camera 21 may change due to, for example, deterioration over time or an external impact. The image processing device 1 of this embodiment, which functions as a posture estimation device, can detect a change (abnormality) in the mounting posture of the camera 21 by estimating the posture of the camera 21 .

As shown in FIG. 1 , the image processing apparatus 1 includes an acquisition section 11 , a control section 12 and a storage section 13 .

The acquisition unit 11 acquires a captured image from a camera 21 mounted on a vehicle (moving body). The obtaining unit 11 temporally continuously obtains analog or digital captured images from the camera 21 at a predetermined cycle (for example, a cycle of 1/30 second). That is, an aggregate of captured images acquired by the acquisition unit 11 is the moving image captured by the camera 21 . Then, when the acquired captured image is analog, the acquisition unit 11 converts (A/D converts) the analog captured image into a digital captured image. The acquisition unit 11 outputs the acquired photographed image or the acquired and converted photographed image to the control unit 12 . One captured image output from the acquisition unit 11 becomes one frame image.

The control unit 12 is, for example, a microcomputer, and controls the overall image processing apparatus 1 . The control unit 12 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). The storage unit 13 is, for example, a non-volatile memory such as a flash memory, and stores various kinds of information. The storage unit 13 stores a program as firmware and various data.

The storage unit 13 stores the reference extraction area as data (reference extraction area information 131). The reference extraction area is used for setting an extraction area, which will be described later. The reference extraction region information 131 may be invariable, but it may also have a configuration that can be changed, a configuration that is updated as appropriate, or the like.

The extraction unit 121, the setting unit 122, and the estimation unit 123 shown in FIG. be. In other words, the image processing device 1 includes an extractor 121 , a setter 122 and an estimator 123 .

At least one of the extraction unit 121, the setting unit 122, and the estimation unit 123 of the control unit 12 is configured by hardware such as ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array). good too. Also, the extraction unit 121, the setting unit 122, and the estimation unit 123 are conceptual components. A function executed by one component may be distributed to a plurality of components, or a function possessed by a plurality of components may be integrated into one component. Further, the acquisition unit 11 may be implemented by the CPU of the control unit 12 performing arithmetic processing according to a program.

The extraction unit 121 extracts feature points from the captured image. A feature point is a point that can be conspicuously detected in a captured image, such as an intersection of edges in the captured image. The feature points are, for example, edges of white lines drawn on the road surface, cracks on the road surface, stains on the road surface, gravel on the road surface, and the like. In the present embodiment, as a preferred form, feature points with a high corner degree indicating cornerness are extracted. A corner is where two edges meet. The corner degree can be determined using a known detection method such as Harris operator or KLT (Kanade-Lucas-Tomasi) tracker.

Specifically, the extraction unit 121 extracts feature points from an extraction region set in the captured image. The extraction area is set for the purpose of efficiently extracting a desired feature point from the captured image. In this embodiment, the desired feature point is a feature point on the road surface on which the host vehicle travels. For this reason, the extraction area is set to an area in which the road surface of the photographed image is shown.

The setting unit 122 sets and changes the extraction region. That is, the extraction area is not constant but fluctuates. The setting unit 122 may, for example, determine whether to change the setting of the extraction region for each image. Also, the setting unit 122 may determine whether or not to change the setting of the extraction region for each predetermined number of frames.

Specifically, the setting unit 122 sets and changes the extraction region based on the external environment of the own vehicle (moving body) in the shooting direction of the camera 21 . For example, if the camera 21 is the front camera described above, the setting unit 122 sets and changes the extraction region based on the external environment in front of the vehicle. With such a configuration, it is possible to change the extraction area according to the position of the object existing outside the own vehicle. For example, the extraction area can be changed so that an object existing on the road surface does not enter the extraction area. Since objects on the road surface can be prevented from entering the extraction area, feature points on the road surface can be extracted with high accuracy. In addition, the extraction area can be widened to the extent that objects on the road surface do not fall within the extraction area, making it possible to extract as many feature points as possible.

In this embodiment, the object is a three-dimensional object, and does not include an object such as a sign drawn on the road surface that is considered to have the same height as the road surface. The objects are, for example, other vehicles (including automobiles, motorcycles, bicycles, etc.), pedestrians, animals, or stationary objects such as walls.

The setting unit 122 sets the extraction region by varying the reference extraction region stored as data in the storage unit 13 . According to this, since the extraction area can be determined based on existing data, the burden required for setting the extraction area can be reduced. More specifically, the setting unit 122 sets the extraction area by enlarging or reducing the reference extraction area. Note that the setting unit 122 may be configured to obtain an extraction region without using the reference extraction region. The details of setting and changing the extraction region by the setting unit 122 will be described later.

The estimation unit 123 estimates the orientation of the camera 21 based on the feature points. Since feature points can be extracted with high accuracy by the function of the setting unit 122, the posture of the camera 21 can be estimated with high accuracy. Moreover, the function of the setting unit 122 can increase the number of feature points to be extracted as much as possible, and the result of posture estimation by the estimation unit 123 can be increased. As a result, the reliability of orientation estimation of the camera 21 can be improved.

Note that the estimating unit 123 may be configured to estimate the amount of movement of the own vehicle, for example, based on the temporal change in the positions of the feature points. The posture of the camera 21 may be estimated by comparing the amount of movement obtained by the estimating unit 123 and the amount of movement of the own vehicle obtained from, for example, the speed sensor 4 .

<2. Posture Estimation Processing>
Next, posture estimation processing of the camera 21 executed by the image processing device 1 will be described. FIG. 2 is a flowchart showing an example of posture estimation processing of the camera 21 executed in the image processing device 1. As shown in FIG. Note that when the photographing unit 2 includes a plurality of cameras 21 , the image processing device 1 executes posture estimation processing shown in FIG. 2 for each camera 21 . Here, the attitude estimation processing of the camera 21 executed by the image processing device 1 will be described, taking as an example the case where the camera 21 is the front camera.

As shown in FIG. 2, first, the control unit 12 monitors whether or not the vehicle equipped with the camera 21 is traveling straight (step S1). Whether or not the host vehicle is traveling straight can be determined, for example, based on rotation angle information of the steering wheel obtained from the steering angle sensor 3 . For example, when the steering wheel rotation angle is zero and the vehicle is traveling in a straight line, it will not only be possible when the rotation angle is zero, but also when the rotation angle is within a certain range between positive and negative directions. It may be determined that the own vehicle is going straight including the case where there is. It should be noted that straight running includes both forward straight running and backward straight running.

The control unit 12 repeats the monitoring of step S1 until it detects that the vehicle is running straight. In other words, the control unit 12 does not proceed with the attitude estimation process unless the vehicle is traveling straight. According to this, the attitude is estimated using the positional changes of the feature points during straight movement, and the attitude is not estimated using the information when the traveling direction of the own vehicle is curved. , it is possible to avoid complication of pose estimation processing. However, a configuration may be adopted in which the attitude estimation process is advanced using the positional changes of the feature points when the traveling direction of the host vehicle is curved.

In addition to determining whether the vehicle is traveling straight ahead, the control unit 12 preferably determines whether the speed of the vehicle is within a predetermined speed range. If the speed of the own vehicle is too slow, there is a possibility that the movement of the feature points will be small and the accuracy of deriving the optical flow will decrease. On the other hand, if the speed of the host vehicle is too fast, the feature points may move too fast to follow the feature points. By advancing the attitude estimation process only when the speed of the own vehicle is within a predetermined speed range, it is possible to suppress the occurrence of such a problem. The predetermined speed range may be, for example, 10 km/h or more and 40 km/h or less. The speed of the own vehicle may be acquired using the speed sensor 4 .

When it is determined that the host vehicle is traveling straight (Yes in step S1), the control unit 12 acquires a captured image from the camera 21 via the acquisition unit 11 (step S2). The control unit 12 acquires, for example, several frames to several tens of captured images.

Next, the extraction unit 121 extracts feature points from each frame image (step S3). The extraction unit 121 extracts feature points from the extraction region set by the setting unit 122 for each frame image. In this embodiment, as an example, extraction regions having the same shape and at the same position are set for the plurality of frame images obtained in step S2. The details of setting the extraction area will be described later. An extraction region may be set for each frame image.

FIG. 3 is a diagram for explaining extraction of feature points by the extraction unit 121. As shown in FIG. FIG. 3 schematically shows a photographed image P photographed by the camera 21. As shown in FIG. The captured image P includes an area BO in which the body of the vehicle is reflected. In FIG. 3, an area 5 surrounded by a dashed line is an extraction area. The extraction unit 121 extracts feature points from the extraction region 5 . By setting the extraction area 5, feature points can be efficiently extracted from the road surface RS. Although the extraction region 5 is rectangular in the example shown in FIG. 3, the shape of the extraction region 5 may be circular, elliptical, or other shapes other than rectangular. In the example shown in FIG. 3, a first feature point FP1 and a second feature point FP2 with high corner degrees are present in the portion of the number indicating the speed limit drawn on the road surface RS. For this purpose, the extraction unit 121 extracts the first feature point FP1 and the second feature point FP2 existing on the road surface RS.

In FIG. 3, only two feature points FP1 and FP2 are shown for the sake of convenience. be. Conversely, when there is no feature point whose corner degree exceeds the predetermined feature point threshold value, no feature point is extracted. It is possible that not even one feature point is extracted from each frame image.

Returning to FIG. 3, when the feature point extraction processing by the extraction unit 121 is completed for all frame images, the estimation unit 123 derives optical flows (step S4). Optical flow is a motion vector that indicates the motion of feature points between two captured images captured at different times. In this example, the interval between different times is the same as the frame period of the acquisition unit 11 . However, the interval between different times is not limited to this, and may be, for example, multiple times the frame period of the acquisition unit 11 .

FIG. 4 is a diagram for explaining how the estimation unit 123 derives the optical flow. FIG. 4 is a schematic diagram shown for convenience similar to FIG. FIG. 4 shows a photographed image P′ (referred to as a current frame P′ for convenience) photographed by the camera 21 after one frame cycle has elapsed after photographing the photographed image P (referred to as a previous frame P for convenience) shown in FIG. do). After capturing the captured image P shown in FIG. 3, the own vehicle is traveling straight forward for one frame cycle. A circle FP1P shown in FIG. 4 indicates the position of the first feature point FP1 at the time of capturing the previous frame P shown in FIG. A circle FP2P shown in FIG. 4 indicates the position of the second feature point FP2 at the time of capturing the previous frame P shown in FIG.

As shown in FIG. 4, when the own vehicle goes straight ahead, the first feature point FP1 and the second feature point FP2 existing in front of the own vehicle approach the own vehicle. That is, the first feature point FP1 and the second feature point FP2 appear at different positions between the current frame P' and the previous frame P. FIG. The estimating unit 123 associates the first feature point FP1 of the current frame P′ with the first feature point FP1 of the previous frame P based on pixel values in the vicinity thereof, and calculates each of the associated first feature points FP1. Based on the position, the optical flow OF1 of the first feature point FP1 is derived. Similarly, the estimation unit 123 associates the second feature point FP2 of the current frame P′ with the second feature point FP2 of the previous frame P based on the neighboring pixel values, and , the optical flow OF2 of the second feature point FP2 is derived.

Note that the estimation unit 123 derives the optical flow as described above for each of the acquired frame images. Also, more than two optical flows may be obtained in each frame image. For this reason, the estimation unit 123 may obtain a plurality of optical flow candidates to be used for posture estimation. Conversely, estimation section 123 may not be able to obtain optical flow candidates for use in posture estimation.

Returning to FIG. 2, after deriving the optical flow, the estimating unit 123 selects an optical flow to be used for subsequent processing according to a predetermined selection condition (step S5). A predetermined selection condition is set such that a set of two optical flows suitable for subsequent pose estimation is selected. If there are a plurality of sets that satisfy the predetermined selection condition, all the sets that satisfy the predetermined selection condition are selected. The number of sets satisfying a predetermined selection condition varies depending on the number of feature points extracted from the captured image. It should be noted that if there is no set that satisfies a predetermined selection condition, subsequent processing cannot be performed. Therefore, the orientation estimation process is terminated. If the orientation estimation process ends at this stage, the orientation estimation result cannot be obtained.

Here, the processing after step S6 will be described assuming that the set of used optical flows selected in step S5 includes the set of the first optical flow OF1 and the second optical flow OF2 shown in FIG.

The estimation unit 123 proceeds with processing using the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2. The estimation unit 123 performs coordinate transformation on the feature points using the internal parameters of the camera 21 stored in the storage unit 13 . Aberration correction and distortion correction of the camera 21 are performed in the coordinate transformation. Aberration correction is performed to correct distortion due to aberration of the optical system of the camera 21 . Specifically, it is correction of distortion such as barrel distortion and pincushion distortion. Distortion correction is performed to correct distortion of the optical system itself of the camera 21 . Specifically, it is fisheye correction. Coordinate transformation converts the coordinates of the feature points into coordinates obtained on a two-dimensional image when the subject is photographed by perspective projection.

When the feature point coordinate transformation is performed, the estimation unit 123 sets the starting point of the optical flow OF1 of the first feature point FP1 to the vertex SP1 and the end point of the optical flow OF1 of the first feature point FP1 to the vertex, as shown in FIG. A quadrangle QL is virtually formed with the starting point of the optical flow OF2 of EP1 and the second feature point FP1 as the vertex SP2 and the end point of the optical flow OF2 of the second feature point FP2 as the vertex EP (step S6). Hereinafter, for the sake of explanation, geometric elements such as quadrilaterals and sides of the quadrilaterals are virtually formed and used. However, in actual processing, processing may be based on vector operations such as the coordinates of feature points and the direction of straight lines, and may not be based on geometric elements having equivalent effects.

When the quadrangle QL is virtually formed, the estimation unit 123 uses the quadrangle QL and the internal parameters of the camera 21 stored in the storage unit 13 to obtain the projection plane IMG of the camera 21 in the three-dimensional space (Fig. 6), the quadrangle QL is moved upward to virtually generate a quadrangle QL1 on the projection plane IMG (step S7).

For the sake of explanation, the edges are defined as follows. A first side SD1 of quadrangle QL1 corresponds to a side connecting vertex SP1 and vertex SP2 in quadrangle QL. That is, it corresponds to the side connecting the first feature point FP1 and the second feature point FP2 in the previous frame P. Similarly, the second side SD2 of quadrangle QL1 corresponds to the side connecting vertex SP2 and vertex EP2 in quadrangle QL. That is, it corresponds to the optical flow OF2 of the second feature point FP2. Similarly, the third side SD3 of quadrangle QL1 corresponds to the side connecting vertex EP1 and vertex EP2 in quadrangle QL. That is, it corresponds to the side connecting the first feature point FP1 and the second feature point FP2 in the current frame P'. Similarly, the fourth side SD4 of quadrangle QL1 corresponds to the side connecting vertex SP1 and vertex EP1 in quadrangle QL. That is, it corresponds to the optical flow OF1 of the first feature point FP1.

Also, a surface is defined as follows (see FIG. 6). A plane including the first side SD1 of the quadrangle QL1 and the optical center OC of the camera 21 is defined as a first plane F1. Similarly, the plane including the second side SD2 of the quadrangle QL1 and the optical center OC of the camera 21 is defined as a second plane F2. Similarly, a surface including the third side SD3 of the quadrangle QL1 and the optical center OC of the camera 21 is defined as a third surface F3. Similarly, a plane including the fourth side SD4 of the quadrangle QL1 and the optical center OC of the camera 21 is defined as a fourth plane F4.

Next, the estimating unit 123 identifies two sets of planes whose intersection lines with a predetermined plane are parallel to each other (step S8). A predetermined plane is a plane whose normal is known in advance. Specifically, it is the plane on which the vehicle is moving, that is, the road surface. The predetermined plane does not have to be a strict plane, as long as it can be regarded as a plane when the estimation unit 123 identifies two sets of planes whose intersection lines with the predetermined plane are parallel to each other. good.

In this embodiment, the image processing apparatus 1 extracts feature points from two images taken at different times and calculates the optical flow of the feature points when the vehicle is traveling straight. Also, the feature points are extracted from a stationary object positioned on a predetermined plane such as a road surface. Therefore, in the real world, the calculated optical flow represents the relative position change of the stationary object with respect to the host vehicle. In other words, it is the movement vector of the own vehicle whose direction is reversed.

The second side SD2 and the fourth side SD4 of the quadrangle QL1 both correspond to the optical flow of the feature points, and thus both correspond to the movement vector of the own vehicle in the real world. Therefore, it is assumed that they are parallel to each other on the road surface.

Further, since the first side SD1 and the third side SD3 of the quadrangle QL1 are both in the positional relationship between feature points, they correspond to the positional relationship between stationary objects accompanying the movement of the own vehicle in the real world. The positional relationship before movement corresponds to the first side SD1, and the positional relationship after movement corresponds to the third side SD3. Since the position of the stationary object does not change at this time, the positional relationship does not change before and after the movement. Therefore, it is assumed that they are also parallel to each other on the road surface.

Therefore, the estimating unit 123 determines two sets of planes having parallel lines of intersection with the road surface: the set of the second plane F2 and the fourth plane F4, and the set of the first plane F1 and the third plane F3. identify. In other words, the estimating unit 123 identifies a total of two sets of surfaces including optical flows as one set and surfaces including feature points captured at the same time as another set.

In FIG. 6, a rectangle QL2 represents the position of the first feature point FP1 in the three-dimensional space (real world) at the time of shooting the current frame P', and the position of the second feature point FP2 at the time of shooting the current frame P'. The position in the three-dimensional space, the position in the three-dimensional space of the first feature point FP1 at the time of capturing the previous frame P, and the position in the three-dimensional space of the second feature point FP2 at the time of capturing the previous frame P are It is a rectangle with vertices. The first face F1 includes the first side SD11 of the quadrangle QL2. Similarly, the second surface F2 includes the second side SD12 of the quadrilateral QL2, the third surface F3 includes the third side SD13 of the quadrilateral QL2, and the fourth surface F4 includes the fourth side SD14 of the quadrilateral QL2. At this time, the quadrangle QL2 is assumed to be a parallelogram formed on the road surface as described above.

Next, the estimation unit 123 calculates the normal line of the road surface (step S9). First, the estimating unit 123 obtains the direction of the line of intersection between the surfaces based on the first surface F1 and the third surface F3, which are one of the previously identified sets of surfaces. Specifically, the direction of the line of intersection CL1 between the first surface F1 and the third surface F3 is determined (see FIG. 7). A direction vector V1 of the line of intersection CL1 is a vector perpendicular to each of the normal vector of the first surface F1 and the normal vector of the third surface F3. Therefore, the estimation unit 123 obtains the direction vector V1 of the line of intersection CL1 from the cross product of the normal vector of the first surface F1 and the normal vector of the third surface F3. Since the first plane F1 and the third plane F3 are parallel to the road surface, the direction vector V1 is parallel to the road surface.

Similarly, the estimating unit 123 obtains the direction of the line of intersection between the second surface F2 and the fourth surface F4, which are the other of the set of previously identified surfaces. Specifically, the direction of the intersection line CL2 between the second surface F2 and the fourth surface F4 is obtained (see FIG. 8). A direction vector V2 of the line of intersection CL2 is a vector perpendicular to each of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Therefore, the estimation unit 123 obtains the direction vector V2 of the line of intersection CL2 from the cross product of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Similarly, the second surface F2 and the fourth surface F4 also have parallel lines of intersection with the road surface, so the direction vector V2 is parallel with the road surface.

The estimating unit 123 calculates the normal to the surface of the quadrangle QL2, that is, the normal to the road surface, from the outer product of the direction vector V1 and the direction vector V2. Since the normal line of the road surface calculated by the estimating unit 123 is calculated in the camera coordinate system of the camera 21, the coordinate system of the three-dimensional space is obtained from the difference from the vertical direction, which is the normal line of the actual road surface. pose can be estimated. Based on the estimation result, the estimation unit 123 estimates the posture of the camera 21 with respect to the own vehicle (step S10). Note that the calculation process in step S9 can be executed using, for example, the known ARToolkit.

When the orientation estimation in step S10 is completed, the estimation unit 123 checks whether or not orientation estimation has been completed for all optical flow pairs selected in step S5 (step S11). If posture estimation has been completed for all pairs (Yes in step S11), the flow shown in FIG. 2 ends. On the other hand, if posture estimation has not been completed for all pairs (No in step S11), the process returns to step S6 and the processes after step S6 are repeated.

The image processing apparatus 1 uses the movement of the own vehicle to autonomously specify two pairs of planes whose intersection lines with a predetermined plane are parallel to each other, and thus only the feature point extraction error is the estimation accuracy. can be used to estimate the pose of the camera, which affects the That is, the image processing apparatus 1 can perform camera orientation estimation with few error factors. Therefore, the image processing device 1 can accurately estimate the orientation of the camera.

In the above description, the plane including the optical center OC of the camera 21 and one side of the quadrangle QL1 is specified, but this is not the only option. A plane may be specified by specifying the normal direction of the plane. For example, the normal direction of the plane may be obtained from the outer product of the direction vectors from the optical center OC to each vertex, and the plane may be identified by the normal direction. In this case, the normal direction of the first surface F1 and the normal direction of the third surface F3 are set as one set, and the normal direction of the second surface F2 and the normal direction of the fourth surface F4 are set as another set. Two sets may be identified.

Also, the plane may be translated. For example, instead of the first plane F1, a plane translated from the first plane F1 may be paired with the third plane F3. This is because the direction of the line of intersection with a predetermined plane does not change even if it is translated.

Also, the image processing device 1 preferably determines the posture estimation based on a plurality of estimation results. In this case, the processing in FIG. 2 may be repeated until a predetermined number or more of orientation estimation results are obtained. When a predetermined number or more of posture estimation results are accumulated, for example, an average value or a median value or peak value of a histogram may be obtained, and the obtained result may be used as the fixed result of posture estimation. The determination result of posture estimation may be used for further processing in the image processing device 1 or may be output to an external device.

For example, it may be determined whether or not the mounting of the camera 21 is misaligned based on the determination result of orientation estimation. When it is determined that the mounting of the camera 21 is in a state of deviation, for example, the user of the own vehicle is notified of the abnormal situation. As a result, the user of the own vehicle can take measures such as requesting the dealer to adjust the installation of the camera.

Also, the determined result of pose estimation of the camera 21 may be used to correct the parameters of the camera 21 . As a result, even if the camera 21 is misaligned, the camera is calibrated according to the misalignment, and the adverse effects of the misalignment of the camera 21 on the captured image can be suppressed. Further, when the photographed images are transmitted to a center which is provided so as to be able to communicate with a plurality of vehicles via a network, the posture estimation result may be transmitted to the center together with the photographed images. As a result, it is possible to appropriately handle the photographing information at the center.

<3. Setting and changing the extraction area>
FIG. 9 is a first diagram for explaining setting and changing of the extraction region 5 by the setting unit 122. FIG. FIG. 10 is a second diagram for explaining setting and changing of the extraction region 5 by the setting unit 122. FIG. FIG. 11 is a third diagram for explaining setting and changing of the extraction region 5 by the setting unit 122. FIG. FIG. 12 is a fourth diagram for explaining setting and changing of the extraction region 5 by the setting unit 122. FIG.

9 to 12, reference numeral 200 denotes the own vehicle, and reference numeral 300 denotes the other vehicle. 9 to 12, in order to facilitate understanding of the relationship between own vehicle 200 and other vehicle 300, the extraction area 5 set in the captured image is virtually shown in real space. 9 to 12, the camera 21 is a front camera, and the imaging direction of the camera 21 is forward. 9 to 12, the extraction area 5, which is set in a rectangular shape as an example, is given the same direction as the real space for the sake of convenience. 9 to 12, reference numeral 5r denotes a reference extraction area.

The setting unit 122 sets the extraction region 5 by changing the reference extraction region 5r based on the position of an object that exists away from the own vehicle (moving body) 200 in the photographing direction of the camera 21 . According to this, it is possible to set the extraction area 5 so that the object existing on the road surface RS does not enter the extraction area 5 while reducing the processing load by using the reference extraction area 5r. Since an object existing on the road surface RS can be prevented from entering the extraction region 5, the optical flow can be properly derived.

It should be noted that the position of the object may be obtained from the captured image input from the camera 21, for example. Also, the position of the object may be determined using an in-vehicle sensor other than the camera 21 . Examples of such a sensor include an ultrasonic sensor, radar, LIDAR (Laser Imaging Detection and Ranging), a camera provided separately from the camera 21, and the like.

In the example shown in FIG. 9 , the object is another vehicle 300 . That is, the setting unit 122 sets the extraction region 5 by changing the reference extraction region 5r based on the position of the other vehicle 300 that is present ahead of the own vehicle 200 and is separated from the host vehicle 200 . Specifically, when the other vehicle 300 existing in front is not present within the predetermined distance D from the subject vehicle 200, the setting unit 122 sets the reference extraction region 5r as the extraction region 5, as shown on the left side of FIG. . Note that the predetermined distance D may be the distance from the camera 21 fixed to the own vehicle instead of the own vehicle 200 . The predetermined distance D is the same as the front side FRS of the reference extraction region 5r, or is set ahead of the front side FRS. In the example shown in FIG. 9, the predetermined distance D is the same as the front side FRS of the reference extraction region 5r. Note that the predetermined distance D may be appropriately determined through experiments or the like.

On the other hand, when the other vehicle 300 (object) exists within the predetermined distance D, the setting unit 122 narrows the width of the reference extraction region 5r in the direction toward the host vehicle (moving object) 200 in the captured image, and extracts the extraction region 5r. set. According to this, when the other vehicle 300 does not exist near the own vehicle 200, it becomes possible to extract many feature points by setting the reference extraction area 5r to a wide range in advance. Then, when the other vehicle 300 approaches the own vehicle 200 , the extraction area 5 is made narrower than the reference extraction area 5 r so that the other vehicle 300 can be positioned outside the extraction area 5 . In the example shown on the right side of FIG. 9 , since part of the other vehicle 300 exists within the predetermined distance D, the setting unit 122 brings the front side FRS of the reference extraction region 5r closer to the own vehicle 200 side to make the reference An area obtained by narrowing the extraction area 5r is defined as an extraction area 5. FIG.

Note that the setting unit 122 may be configured to change the extraction region 5 to one region prepared in advance when making the extraction region 5 narrower than the reference extraction region 5r. In addition, the setting unit 122 may be configured to select one of a plurality of regions prepared stepwise when the extraction region 5 is set to be narrower than the reference extraction region 5r. In this case, the setting unit 122 may be configured to select any one area according to the distance from the own vehicle 200 to the other vehicle 300 . Further, when setting the extraction region 5 to be narrower than the reference extraction region 5r, the setting unit 122 may continuously narrow the extraction region 5 according to the distance from the own vehicle 200 to the other vehicle 300 instead of stepwise.

Further, the setting unit 122 may set the reference extraction region 5r as the extraction region 5 when the other vehicle 300 exists within the predetermined distance D. Then, when the other vehicle 300 does not exist within the predetermined distance D, the setting unit 122 may set the extraction area 5 by increasing the width of the reference extraction area 5r in the direction away from the own vehicle 200 in the captured image. good. To explain this modified example using the example shown in FIG. 9, the extraction area 5 on the right side of FIG. As shown, the area obtained by enlarging the reference extraction area is used as the extraction area. When the other vehicle 300 is at a position away from the own vehicle 200, it becomes possible to extract many feature points by making the extraction area 5 wider than the reference extraction area 5r.

Further, the setting unit 122 may set the extraction region 5 by considering the speed (relative speed) of the other vehicle 300 with respect to the own vehicle 200 . The speed of the own vehicle 200 is obtained by the speed sensor 4, for example. The speed of the other vehicle 300 may be obtained using, for example, a radar mounted on the host vehicle 200 or inter-vehicle communication. Here, consider a case where another vehicle 300 exists within a predetermined distance D and it is necessary to set the extraction region 5 by narrowing the reference extraction region 5r. At this time, if it is determined from the relative speed that the other vehicle 300 is approaching the own vehicle 200, the setting unit 122 may increase the amount by which the reference extraction area 5r is narrowed. According to this, when setting the same extraction area 5 for a plurality of frame images that are successively acquired within a short period of time, it is possible to reduce the possibility that other vehicles will enter the extraction area 5 . Conversely, when it is determined from the relative speed that the other vehicle 300 is moving away from the host vehicle 200, the setting unit 122 may reduce the amount by which the reference extraction region 5r is narrowed. As a result, the extraction area 5 can be made as wide as possible.

The setting unit 122 sets and changes the extraction region 5 based on the speed of the own vehicle 200 (moving body). According to this, even when the speed of the own vehicle 200 increases, it is possible to suppress the deterioration of the extraction accuracy of the feature points. The speed of the own vehicle is obtained by a speed sensor 4, for example. Also, the setting unit 122 does not have to set and change the extraction region 5 based on the speed of the own vehicle 200 .

In the example shown in FIG. 10, when the speed of the own vehicle 200 is lower than the first speed, the setting unit 122 sets the reference extraction region 5r as the extraction region 5 as shown on the left side of FIG. The first speed is, for example, 30 km/h. Then, when the speed of the own vehicle 200 (moving body) becomes equal to or higher than the first speed, the setting unit 122 narrows the width of the reference extraction area 5r in the direction away from the own vehicle 200 in the captured image to set the extraction area 5. (See right side of FIG. 10). According to this, the vicinity of the own vehicle 200 that is greatly affected by the speed change in the photographed image is included in the extraction area 5 when the speed of the own vehicle 200 is slow, and is extracted when the speed of the own vehicle 200 becomes faster. For example, it can be excluded from area 5. For this reason, it is possible to improve the extraction accuracy of feature points. Further, when the speed of the own vehicle 200 is slow, it is possible to widen the feature point extraction region 5 and extract many feature points. In the example shown on the right side of FIG. 10 , the speed of the vehicle 200 is equal to or higher than the first speed, so the setting unit 122 separates the rear side BAS of the reference extraction region 5r from the vehicle 200 to form the reference extraction region 5r. is defined as an extraction region 5.

Note that the setting unit 122 may be configured to change the extraction region 5 to one region prepared in advance when making the extraction region 5 narrower than the reference extraction region 5r. In addition, the setting unit 122 may be configured to select one of a plurality of regions prepared stepwise when the extraction region 5 is set to be narrower than the reference extraction region 5r. In this case, the setting unit 122 may be configured to select any one area according to the speed of the own vehicle 200 . Further, when setting the extraction region 5 to be narrower than the reference extraction region 5r, the setting unit 122 may continuously narrow it according to the speed of the vehicle 200 instead of stepwise.

Further, the setting unit 122 may set the reference extraction region 5r as the extraction region 5 when the speed of the own vehicle 200 is equal to or higher than the first speed. Then, when the speed of the vehicle 200 becomes lower than the first speed, the setting unit 122 may set the extraction region 5 by increasing the width of the reference extraction region 5r in the direction toward the vehicle 200 in the captured image. . To explain this modification using the example shown in FIG. 10, the extraction area 5 on the right side of FIG. A region obtained by enlarging the reference extraction region is set as the extraction region. Even with such a configuration, the vicinity of the own vehicle 200 that is greatly affected by the speed change in the captured image is included in the extraction area 5 when the speed of the own vehicle 200 is slow, and is extracted when the speed of the own vehicle 200 is fast. It can be excluded from the extraction area 5 .

In the example shown in FIG. 11, when the speed of the own vehicle 200 is lower than the second speed, the setting unit 122 sets the reference extraction region 5r as the extraction region 5 as shown on the left side of FIG. The second speed is, for example, 30 km/h. Then, when the speed of own vehicle 200 (moving object) becomes equal to or higher than the second speed, setting unit 122 narrows the width of reference extraction region 5r in the direction crossing the direction away from own vehicle 200 (moving object) in the captured image. to set the extraction area 5 (see the right side of FIG. 11). Note that the direction intersecting with the direction away from the moving body may be, for example, a direction orthogonal to the direction away from the moving body. According to this, when the speed of the own vehicle 200 increases, the feature points on the outer edge of the photographed image, whose extraction accuracy tends to decrease due to the influence of the lens distortion, are removed from the extraction target when the speed of the own vehicle 200 increases. can be excluded. For this reason, it is possible to improve the extraction accuracy of feature points. Further, when the speed of the own vehicle 200 is slow, it is possible to expand the extraction range of the feature points and extract many feature points. In the example shown on the right side of FIG. 11, since the speed of the own vehicle 200 is equal to or higher than the second speed, the setting unit 122 shifts the left and right sides LES and RIS of the reference extraction region 5r toward the center to form a reference extraction region. A region obtained by narrowing 5r is defined as an extraction region 5 .

Note that the setting unit 122 may be configured to change the extraction region 5 to one region prepared in advance when making the extraction region 5 narrower than the reference extraction region 5r. In addition, the setting unit 122 may be configured to select one of a plurality of regions prepared stepwise when the extraction region 5 is set to be narrower than the reference extraction region 5r. In this case, the setting unit 122 may be configured to select any one area according to the speed of the own vehicle 200 . Further, when setting the extraction region 5 to be narrower than the reference extraction region 5r, the setting unit 122 may continuously narrow it according to the speed of the vehicle 200 instead of stepwise.

Further, the setting unit 122 may set the reference extraction region 5r as the extraction region 5 when the speed of the own vehicle 200 is equal to or higher than the second speed. When the speed of the vehicle 200 becomes lower than the second speed, the setting unit 122 widens the width of the reference extraction region 5r in the direction crossing the direction away from the vehicle 200 in the captured image, and sets the extraction region 5. It is good also as a structure which carries out. To explain this modification using the example shown in FIG. 11, the extraction area 5 on the right side of FIG. A region obtained by enlarging the reference extraction region is set as the extraction region. Even with such a configuration, when the speed of the own vehicle 200 increases, the feature points at the outer edge of the photographed image, whose extraction accuracy tends to decrease due to the influence of lens distortion, are extracted when the speed of the own vehicle 200 increases. Can be excluded from extraction targets. For this reason, it is possible to improve the extraction accuracy of feature points. Further, when the speed of the own vehicle 200 is slow, it is possible to expand the extraction range of the feature points and extract many feature points.

In this embodiment, when the speed of the vehicle 200 becomes equal to or higher than the first speed, the setting unit 122 separates the rear side BAS of the reference extraction region 5r from the vehicle 200 to narrow the reference extraction region 5r. An extraction area 5 is assumed. Further, when the speed of the own vehicle 200 becomes equal to or higher than the second speed, the setting unit 122 shifts the left and right sides LES and RIS of the reference extraction region 5r toward the center to narrow the reference extraction region 5r to the extraction region. 5. In this case, the first speed and the second speed may be different, but in this embodiment they are the same.

Therefore, as shown in FIG. 12, when the speed of the vehicle 200 becomes equal to or higher than the first speed (=the second speed), the rear side BAS of the reference extraction area 5r is moved away from the vehicle 200. At the same time, the left and right sides LES and RIS are moved toward the center, and an extraction area 5 narrower than the reference extraction area 5r is set.

Note that the setting unit 122 may be configured to change only one of the setting change of the extraction region 5 shown in FIG. 10 and the setting change of the extraction region 5 shown in FIG.

FIG. 13 is a flowchart showing an example of processing for setting the extraction region 5 in the image processing apparatus 1 of this embodiment. FIG. 13 is a flow chart assuming that the camera 21 is a front camera. The setting of the extraction region 5 is performed, for example, after the captured image acquisition process in step S2 of FIG. 2 is performed.

In step S21, the setting unit 122 confirms whether or not the forward vehicle exists within a predetermined distance D. If the forward vehicle is within the predetermined distance D (Yes in step S21), the setting unit 122 proceeds to the next step S22. On the other hand, if the forward vehicle does not exist within the predetermined distance D (No in step S21), the setting unit 122 proceeds to step S23.

In step S<b>22 , the setting unit 122 sets a first area 5 a (see FIG. 14 ) obtained by narrowing the reference extraction area 5 r toward the host vehicle 200 as a first provisional setting. After setting the first region 5a as the first temporary setting, the setting unit 122 advances the process to step S24. Note that FIG. 14 is a diagram for explaining the capturing of the flowchart of FIG. 13 . In FIG. 14, the sides slightly shifted between the regions 5a, 5b, 5c, 5d, 5e, and 5r are originally sides that overlap each other. They are shifted for convenience so that they can be easily distinguished.

In step S23, the setting unit 122 sets the reference extraction region 5r (see FIG. 14) as the first temporary setting. After setting the reference extraction region 5r as the first temporary setting, the setting unit 122 advances the process to step S24.

At step S24, the setting unit 122 confirms whether or not the speed of the own vehicle 200 is equal to or higher than a predetermined speed. The predetermined speed is the same speed as the above-described first speed and second speed, eg, 30 km/h. When the speed of the own vehicle 200 is equal to or higher than the predetermined speed (Yes in step S24), the setting unit 122 advances the process to the next step S25. On the other hand, if the speed of the own vehicle 200 is lower than the predetermined speed (Yes in step S24), the setting unit 122 advances the process to step S27.

In step S<b>25 , the setting unit 122 sets, as the second provisional setting, a region narrowed from the first provisionally set region toward the direction away from the host vehicle 200 . When the first area 5a is set in the first temporary setting, the second area 5b, which is the first area 5a narrowed in the direction away from the host vehicle 200, becomes the second temporary setting area. When the reference extraction area 5r is set in the first temporary setting, the third area 5c, which is the reference extraction area 5r narrowed in the direction away from the host vehicle 200, becomes the second temporary setting area. After performing the second provisional setting, the setting unit 122 advances the process to the next step S26.

In step S<b>26 , the setting unit 122 narrows the width of the second provisionally set area in the direction crossing the direction away from the host vehicle 200 to set the final extraction area 5 . If the second region 5b is set in the second provisional setting, the extraction region 5 is a fourth region 5d that is narrowed in the left-right direction on the captured image of the second region 5b. If the third area 5 c is set in the second provisional setting, the extraction area 5 is the fifth area 5 e obtained by narrowing the horizontal width of the third area 5 c on the image. When step S26 is completed, the setting process of the extraction area 5 is completed.

In step S<b>27 , the setting unit 122 sets the first provisionally set area as the extraction area 5 . If the first region 5 a is set in the first provisional setting, the first region 5 a becomes the extraction region 5 . If the reference extraction region 5r is set in the first temporary setting, the reference extraction region 5r becomes the extraction region 5. FIG. When step S27 is completed, the setting process of the extraction area 5 is completed.

In the example shown in FIG. 13, after confirming whether or not the forward vehicle is within the predetermined distance D, it is confirmed whether or not the speed of the own vehicle 200 is equal to or higher than the predetermined speed. can be vice versa. Further, the order of the processes of steps S25 and S26 shown in FIG. 13 may be changed. That is, the second temporary setting area is obtained by narrowing the width of the first temporary setting area in the left-right direction on the photographed image, and the extraction area 5 is the area obtained by narrowing the second temporary setting area in the direction away from the host vehicle 200 . good.

In the configuration of the present embodiment, when an object (another vehicle 300 or the like) exists within a predetermined distance D in front of the own vehicle 200 or the speed of the own vehicle 200 is equal to or higher than a predetermined speed, reference extraction The extraction area 5 becomes narrower than the area 5r. In this case, the possibility that only one of the first feature point FP1 and the second feature point FP2 can be extracted from a single captured image is more likely than in the case of using the reference extraction region 5r. be. Such a tendency is particularly likely to occur when an object exists within a predetermined distance D in front of own vehicle 200 and the speed of own vehicle 200 is equal to or higher than a predetermined speed.

Considering such a point, the estimation unit 123 calculates the position change (first optical flow OF1) and the change in position of the second feature point (second optical flow OF2) to identify two pairs of planes whose intersection lines with a predetermined plane are parallel to each other (the above configuration). A configuration such as the following may be used as appropriate. That is, the estimation unit 123 selects the first feature points corresponding to each other from the feature points extracted from the two captured images captured at the first time and the second time after the first time, and selects the first feature points. Calculate the position change of the feature point. Further, the estimating unit 123 determines, from the feature points extracted from the two captured images captured at a third time after the first time and a fourth time after the third time, that the first feature point is Second feature points corresponding to each other are selected, and positional changes of the second feature points are calculated. Then, the estimating unit 123 identifies two sets of planes whose lines of intersection with a predetermined plane are parallel to each other, based on the positional change of the first feature point and the positional change of the second feature point. By appropriately using such a configuration, even when the extraction region 5 is made small, the possibility of obtaining the estimation result of the orientation of the camera 21 is increased.

<4. Notes>
The configurations of the embodiments and modifications in this specification are merely examples of the present invention. The configurations of the embodiments and modifications may be changed as appropriate without departing from the technical idea of the present invention. Also, multiple embodiments and modifications may be implemented in combination within a possible range.

INDUSTRIAL APPLICABILITY The present invention can be used for estimating the posture of a camera that assists the driving of a moving object such as automatic parking. The present invention can also be used to estimate the orientation of a camera that records driving information, such as a drive recorder. Further, the present invention can be used for a correction device or the like that corrects a photographed image by using the estimation information of the orientation of the camera. Further, the present invention can be used for a device or the like that operates in cooperation with a center that is provided so as to be able to communicate with a plurality of moving bodies via a network. For example, when transmitting a captured image to the center, the apparatus may be configured to transmit estimation information of the orientation of the camera together with the captured image. Then, at the center, using the camera posture estimation information, various image processing (processing to change the viewpoint and viewing direction of the image considering the camera posture, such as changing the viewpoint and viewing direction to the image in front of the vehicle body) ), correction processing for camera posture in measurement processing using images, statistical processing of changes in camera posture over time (data from many vehicles), etc. to generate useful data to be provided to users. and so on.

1 image processing device 5 extraction region 5r reference extraction region 13 storage unit 21 camera 11 acquisition unit 121 extraction unit 122 setting unit 123 estimation unit 200 host vehicle (moving body)
300 other vehicle (object)
D Predetermined distance

Claims (7)

Captured images are acquired from a camera mounted on a moving body,
extracting feature points from an extraction region set in the captured image;
setting and changing the extraction region based on the external environment of the moving body in the shooting direction of the camera and the speed of the moving body ;
A posture estimation device for estimating a posture of the camera based on the feature points,
narrowing the extraction area when an object existing away from the moving object exists within a predetermined distance in the shooting direction of the camera compared to when the object does not exist within the predetermined distance;
A posture estimating apparatus that narrows the extraction region when the speed of the moving body is equal to or higher than a predetermined speed compared to when the speed of the moving body is less than the predetermined speed. A storage unit that stores the reference extraction region as data,
2. The posture estimation device according to claim 1, wherein said extraction region is set by varying said reference extraction region. setting the extraction area by narrowing the width of the reference extraction area in a direction approaching the moving object in the captured image when the object exists within the predetermined distance, or
3. The posture estimation apparatus according to claim 2 , wherein when the object does not exist within the predetermined distance, the extraction area is set by increasing the width of the reference extraction area in the captured image in a direction away from the moving object. . setting the extraction region by narrowing the width of the reference extraction region in a direction away from the moving object in the photographed image when the speed of the moving object becomes equal to or higher than a first speed, or
4. The posture according to claim 2, wherein when the speed of said moving body becomes lower than a first speed, said extraction region is set by increasing the width of said reference extraction region in a direction toward said moving body in said photographed image. estimation device. when the speed of the moving body becomes equal to or higher than a second speed, setting the extraction region by narrowing the width of the reference extraction region in the direction crossing the direction away from the moving body in the captured image, or
3. From claim 2, wherein when the speed of said moving body becomes smaller than a second speed, said extraction region is set by increasing the width of said reference extraction region in a direction intersecting a direction away from said moving body in said photographed image. 5. The posture estimation device according to any one of 4 . when estimating the orientation of the camera, calculating positional changes of the first feature point and the second feature point;
The first feature point is selected from feature points extracted from the extraction regions of the two captured images captured at a first time and a second time after the first time,
The second feature point is selected from feature points extracted from the extraction regions of the two captured images captured at a third time after the first time and at a fourth time after the third time. The posture estimation device according to any one of claims 1 to 5 . an acquisition step of acquiring a photographed image from a camera mounted on a moving object;
an extraction step of extracting feature points from an extraction region set in the captured image;
a setting step of setting and changing the extraction region based on the external environment of the moving object in the photographing direction of the camera and the speed of the moving object ;
a posture estimation step of estimating a posture of the camera based on the feature points;
is a pose estimation method in which the device executes
narrowing the extraction area when an object existing away from the moving object exists within a predetermined distance in the shooting direction of the camera compared to when the object does not exist within the predetermined distance;
A posture estimation method, wherein when the speed of the moving body is equal to or higher than a predetermined speed, the extraction region is narrowed compared to when the speed of the moving body is less than the predetermined speed.

Images