JP7236352B2 - Posture estimation device and posture estimation method - Google Patents

Description

The present invention relates to a posture estimation device and a posture estimation method.

Conventionally, there is known an on-vehicle camera calibration device that automatically calibrates external parameters such as an attachment position and an attachment angle of an on-vehicle camera (see, for example, Patent Document 1).

The calibration device disclosed in Patent Literature 1 includes a vehicle state determination section, a camera-by-camera processing section, and an operation control section. The vehicle state determination unit determines the vehicle state from vehicle information input from the outside. A per-camera processing unit is arranged for each of the cameras and processes input information from each of the cameras. The operation control unit determines the operation mode of each camera based on the determination result from the vehicle state determination unit and the output from the camera-by-camera processing unit, and determines which camera should be prioritized for calibration. The camera-by-camera processing unit determines how the feature points appear from the image captured by the camera, and notifies the operation control unit.

JP 2018-177004 A

For example, the number of feature points that can be acquired by a plurality of cameras may be biased due to the condition of the road surface on which the vehicle is traveling. In such a case, there may be a large difference in the time required to grasp the mounting state (orientation) of the cameras among the plurality of cameras.

By the way, for example, during automatic parking, a plurality of cameras are normally used at the same time. For this reason, it is desirable that each of the plurality of cameras is aware of its orientation at the start of automatic parking or the like. As a result, automatic driving such as automatic parking can be safely performed using all the cameras to be used. That is, when there are a plurality of cameras that are used simultaneously, it is desirable to reduce the difference in the time required for grasping the postures of the cameras among all the cameras that are used simultaneously.

SUMMARY OF THE INVENTION In view of the above problems, an object of the present invention is to provide a technique capable of reducing the difference in time required to grasp the orientation of a camera among a plurality of cameras mounted on a moving body.

In order to achieve the above object, a posture estimation apparatus of the present invention includes an acquisition unit that acquires a photographed image from each of a plurality of cameras mounted on a moving object, and an estimation process that estimates the posture of the camera. and a setting unit configured to set the priority of the estimation process for each camera based on the number of obtained estimation results obtained by the estimation process performed for each camera. (first configuration).

Further, in the posture estimation device having the first configuration, the setting unit is configured to set the priority based on a comparison result of the cumulative value of the acquired number for each camera (second configuration). is preferred.

Further, the posture estimation apparatus having the first or second configuration determines the posture estimation based on a plurality of the estimation results for the camera for which the cumulative value of the number of acquisitions has reached a predetermined number of times or more. A configuration (third configuration) further including a determination unit is preferable.

Further, in the posture estimation device having the third configuration, it is preferable that the setting unit changes the priority when the camera for which the posture estimation has been confirmed occurs (fourth configuration). .

Further, in the posture estimation device having any one of the first to fourth configurations, the setting unit determines that the difference between the maximum number and the minimum number of the cumulative values of the obtained numbers obtained for each of the cameras is equal to or greater than a predetermined threshold. It is preferable that the configuration (fifth configuration) changes the priority when it becomes.

Further, in the posture estimation apparatus having the fifth configuration, the setting unit sets the priority when the difference between the maximum number and the minimum number becomes smaller than the predetermined threshold after becoming equal to or greater than the predetermined threshold. A changeable configuration (sixth configuration) is preferable.

Further, the posture estimation apparatus having any one of the first to fourth configurations further includes a determination unit that determines whether or not there is a camera having a priority condition other than the acquired number among the plurality of cameras. Further, the setting unit may be configured to set the priority based on the priority condition when there is a camera having the priority condition (seventh configuration).

Further, in the posture estimation device having any one of the first to seventh configurations, the setting unit confirms whether or not to change the priority at a predetermined timing, and the predetermined timing is set to the estimated Each time the order of processing is given to each of the plurality of cameras once, and if there is a camera among the plurality of cameras for which the order is skipped due to the setting of the priority, the camera is given the above-mentioned A configuration (eighth configuration) may be employed in which the predetermined timing is determined by assuming that the order has been given.

Further, in order to achieve the above object, a pose estimation method of the present invention includes an acquisition step of acquiring a photographed image from each of a plurality of cameras mounted on a moving object, and an estimation process of estimating the pose of the camera, a setting step of setting the priority of the estimation process for each camera based on the number of estimation results obtained by the estimation process performed for each camera; It has a configuration (ninth configuration).

According to the present invention, it is possible to reduce the difference in the time required to grasp the postures of the cameras among a plurality of cameras mounted on a moving object.

FIG. 1 is a diagram for explaining the configuration of a posture estimation device according to the first embodiment; FIG. FIG. 2 is a flowchart showing an example of processing executed when estimating the postures of four vehicle-mounted cameras in the posture estimation device of the first embodiment; FIG. 4 is a flowchart showing an example of posture estimation processing by the posture estimation device of the first embodiment; Diagram for explaining extraction of feature points Diagram for explaining derivation of optical flow A diagram showing a rectangle formed virtually from optical flow FIG. 4 is a diagram for explaining a technique for identifying two sets of planes whose intersection lines with a predetermined plane are parallel to each other; Diagram for explaining the method of determining the direction of the line of intersection between the faces based on one of the identified pairs of faces. Diagram for explaining the method of determining the direction of the line of intersection between the faces based on the other of the identified faces Flowchart showing an example of priority change determination processing in the posture estimation device of the first embodiment FIG. 4 is a diagram for explaining priority setting in the posture estimation apparatus of the first embodiment; A diagram for explaining the configuration of a posture estimation apparatus according to a second embodiment. Flowchart showing an example of processing executed when estimating the postures of four vehicle-mounted cameras in the posture estimation device of the second embodiment Flowchart showing an example of priority change determination processing in the posture estimation device of the second embodiment FIG. 5 is a diagram for explaining priority setting in the posture estimation apparatus of the second embodiment;

Exemplary embodiments of the invention are described in detail below with reference to the drawings. In the following description, a vehicle is taken as an example of a moving object, but the moving object is not limited to a vehicle. Vehicles broadly include vehicles with wheels, such as automobiles, trains, and automatic guided vehicles. Examples of mobile objects other than vehicles include ships and aircraft.

Further, in the following description, the direction in which the vehicle travels straight ahead, which is the direction from the driver's seat to the steering wheel, is referred to as the "forward direction." In addition, the direction from the steering wheel to the driver's seat, which is the direction in which the vehicle travels straight, is defined as the "rearward direction." In addition, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and from the right side to the left side of the driver who is facing forward is referred to as the "left direction." Also, the direction perpendicular to the straight traveling direction and the vertical line of the vehicle and from the left side to the right side of the driver who is facing forward is referred to as the "right direction."

<1. First Embodiment>
FIG. 1 is a diagram for explaining the configuration of a posture estimation device 1 according to the first embodiment. FIG. 1 also shows an imaging unit 2 and a steering angle sensor 3 that input information to the attitude estimation device 1 . The attitude estimation device 1 is provided for each vehicle in which the imaging unit 2 is mounted. Hereinafter, the vehicle provided with the attitude estimation device 1 may be referred to as the own vehicle.

The photographing unit 2 is provided in the vehicle for the purpose of monitoring the situation around the vehicle. The photographing unit 2 includes a plurality of cameras 21-24. A plurality of cameras 21 to 24 are in-vehicle cameras. Each vehicle-mounted camera 21-24 is fixedly arranged on the vehicle. The four in-vehicle cameras 21 to 24 are connected to the posture estimation device 1 by wires or wirelessly, respectively, and output captured images to the posture estimation device 1 . The mounting positions and angles of the vehicle-mounted cameras 21 to 24 may deviate due to, for example, deterioration over time or impact from the outside. The attitude estimation device 1 can monitor such changes in the mounting state.

Specifically, the imaging unit 2 includes a front camera 21 , a back camera 22 , a left side camera 23 and a right side camera 24 . The front camera 21 is a camera that photographs the front of the vehicle. The back camera 22 is a camera that photographs the rear of the vehicle. The left side camera 23 is a camera that captures the left side of the vehicle. The right side camera 24 is a camera that captures the right side of the vehicle. Each of the vehicle-mounted cameras 21 to 24 is configured using, for example, a fish-eye lens, and has a horizontal angle of view θ of 180 degrees or more. Therefore, the four on-vehicle cameras 21 to 24 can photograph the entire circumference of the vehicle in the horizontal direction.

Note that the number of cameras included in the photographing unit 2 may be a plurality other than four. For example, when an in-vehicle camera is installed for the purpose of assisting parking in the back, the photographing unit 2 is provided with three in-vehicle cameras: a back camera 22, a left side camera 23, and a right side camera 24. can be

The steering angle sensor 3 is provided in the vehicle on which the attitude estimation device 1 and the imaging unit 2 are mounted, and detects the rotation angle of the steering wheel (steering wheel) of the vehicle. The output of the steering angle sensor 3 is input to the attitude estimation device 1 via a communication bus B1 such as a CAN (Controller Area Network) bus.

As shown in FIG. 1 , posture estimation apparatus 1 includes acquisition section 11 , control section 12 , and storage section 13 .

Acquisition unit 11 acquires a photographed image from each of a plurality of in-vehicle cameras 21-24. The obtaining unit 11 temporally continuously obtains analog or digital captured images from the vehicle-mounted cameras 21 to 24 at a predetermined cycle (for example, 1/30 second cycle). In other words, an aggregate of captured images acquired by the acquisition unit 11 is a moving image captured by the vehicle-mounted cameras 21-24. Then, when the acquired captured image is analog, the acquisition unit 11 converts (A/D converts) the analog captured image into a digital captured image. The acquisition unit 11 outputs the acquired photographed image or the acquired and converted photographed image to the control unit 12 . One captured image output from the acquisition unit 11 becomes one frame image.

The control unit 12 is, for example, a microcomputer, and controls the posture estimation apparatus 1 as a whole. The control unit 12 includes a CPU (Central Processing Unit), a RAM (Random Access Memory), and a ROM (Read Only Memory) (not shown). The storage unit 13 is, for example, a non-volatile memory such as a flash memory, and stores various kinds of information. The storage unit 13 stores a program as firmware and various data.

The estimating unit 121, the determining unit 122, and the setting unit 123 shown in FIG. be. In other words, posture estimation apparatus 1 includes estimation section 121 , determination section 122 , and setting section 123 .

At least one of the estimation unit 121, the determination unit 122, and the setting unit 123 of the control unit 12 is configured by hardware such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). good too. Also, the estimation unit 121, the determination unit 122, and the setting unit 123 are conceptual components. A function executed by one component may be distributed to a plurality of components, or a function possessed by a plurality of components may be integrated into one component. Further, the acquisition unit 11 may be implemented by the CPU of the control unit 12 performing arithmetic processing according to a program.

The estimation unit 121 performs estimation processing for estimating the attitudes of the vehicle-mounted cameras 21 to 24 based on captured images. Specifically, the estimation unit 121 performs posture estimation processing for each of the vehicle-mounted cameras 21 to 24 . The processing for estimating the attitudes of the plurality of vehicle-mounted cameras 21 to 24 is performed in order. The order in which the posture estimation processing is performed will be described later. When estimating the posture of any one of the four in-vehicle cameras 21 to 24, the estimating unit 121 performs estimation processing based on the captured image acquired from the camera that estimates the posture. For example, when estimating the orientation of the front camera 21 , the estimation unit 121 performs estimation processing based on the captured image obtained from the front camera 21 .

In addition, the posture estimation may be configured to obtain an estimated value of a value related to the mounting state, such as the mounting angle of the vehicle-mounted cameras 21 to 24, for example. However, the orientation may be estimated only by estimating whether or not the orientation has changed from a predetermined state, instead of obtaining a specific estimated value regarding the mounting state. The details of the posture estimation processing of this embodiment will be described later.

The determination unit 122 determines the orientation estimation based on a plurality of estimation results for a camera for which the cumulative number of estimation results obtained by the orientation estimation process has reached a predetermined plurality of times. That is, in the present embodiment, the estimation of the orientation of each of the in-vehicle cameras 21 to 24 is not finalized even if only one estimation result of the orientation is obtained. In the present embodiment, when orientation estimation results are obtained for each of the vehicle-mounted cameras 21 to 24 a predetermined number of times or more, the orientation estimation is determined based on the obtained plurality of estimation results. According to this, it is possible to reduce the possibility that posture estimation apparatus 1 outputs an erroneous estimated posture.

Note that the posture estimation result includes estimation information related to the posture state, such as an estimated posture value. In other words, if the estimation information about the attitude state is not obtained, it is not considered that the attitude estimation result has been obtained.

Also, the number of multiple estimation results used to finalize the posture estimation may be the same as or different from the predetermined multiple number of times. Also, the predetermined number of times may be appropriately set based on, for example, experimental results. It is preferable that the predetermined number of times is set so that the finalized result of orientation estimation is as accurate as possible and the time until the finalized result of orientation estimation is obtained is not too long. Further, in the present embodiment, the predetermined number of times is the same for all the vehicle-mounted cameras 21-24. However, the predetermined multiple number of times may be set separately for each vehicle-mounted camera 21-24. That is, the predetermined multiple number of times may differ among the vehicle-mounted cameras 21 to 24 .

The setting unit 123 sets the priority of the orientation estimation process for each of the vehicle-mounted cameras 21 to 24 based on the number of obtained estimation results obtained by the orientation estimation process performed for each of the vehicle-mounted cameras 21 to 24 . Specifically, the setting unit 123 sets the priority of the estimation process for each of the vehicle-mounted cameras 21-24 based on the comparison result of the accumulated number of obtained estimation results for each of the vehicle-mounted cameras 21-24. Settings may include initial settings and subsequent changes to settings.

According to the present embodiment, for example, among the plurality of in-vehicle cameras 21 to 24, if there is a camera that obtains an extremely large number or a small number of posture estimation results compared to the others, the priority is adjusted. By doing so, it is possible to suppress the difference in the number of obtained estimation results from becoming too large. According to the present embodiment, it is possible to bring the timings at which the definite results of posture estimation are obtained among the plurality of vehicle-mounted cameras 21 to 24 as close as possible. For example, when the self-vehicle is set to automatically drive using a plurality of onboard cameras 21 to 24, the attitude estimation device 1 of the present embodiment can grasp the attitudes of all onboard cameras 21 to 24. It is possible to increase the possibility that automatic driving can be started in a safe state.

In this embodiment, when the priority is raised, the frequency of executing the posture estimation process is increased. Also, when the priority is lowered, the frequency of executing the posture estimation process is reduced. Details of the processing related to priority setting will be described later.

The setting unit 123 confirms whether or not to change the priority at a predetermined timing. In the present embodiment, the predetermined timing occurs each time the orientation estimation process is given once to all of the plurality (specifically, four) of the vehicle-mounted cameras 21 to 24 . However, if there is a camera whose order is skipped due to priority setting among the plurality of vehicle-mounted cameras 21 to 24, it is assumed that the order is given to the camera and the predetermined timing is determined. By configuring in this way, it is possible to change the priority at an appropriate timing.

FIG. 2 is a flowchart showing an example of processing executed when estimating the postures of the four vehicle-mounted cameras 21 to 24 in the posture estimation device 1 of the first embodiment. At the start of the processing in FIG. 2, the number of obtained posture estimation results is zero for all of the four vehicle-mounted cameras 21-24, and there is no difference among the four vehicle-mounted cameras 21-24. For this reason, the four vehicle-mounted cameras 21 to 24 have the same orientation estimation processing priority. That is, at the start of the process of FIG. 2, there is no difference in the frequency of estimation of the attitudes of the four on-vehicle cameras 21-24. At the start of the process of FIG. 2, the four vehicle-mounted cameras 21-24 are equally ordered. Posture estimation processing is performed on the four on-vehicle cameras 21 to 24 in a predetermined order.

In step S<b>1 , estimation processing of the attitude of the front camera 21 is performed by the estimation unit 121 . After the posture estimation processing of the front camera 21 is completed, the processing proceeds to step S2. In step S<b>2 , estimation processing of the posture of the back camera 22 is performed by the estimation unit 121 . After the posture estimation processing of the back camera 22 is completed, the processing proceeds to step S3. In step S<b>3 , estimation processing of the orientation of the left side camera 23 is performed by the estimation unit 121 . After the posture estimation processing of the left side camera 23 is completed, the processing proceeds to step S4. In step S<b>4 , estimation processing of the orientation of the right side camera 24 is performed by the estimation unit 121 . After the posture estimation processing of the right side camera 24 is completed, the processing proceeds to step S4.

In this example, the posture estimation processing is performed in the order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24, but this is an example. The posture estimation processing may be performed in another order.

Here, the processing for estimating the attitudes of the vehicle-mounted cameras 21 to 24 will be described in detail. FIG. 3 is a flowchart showing an example of posture estimation processing by the posture estimation device 1 of the first embodiment. FIG. 3 is a diagram showing the details of the posture estimation processing in steps S1 to S4 in FIG. Since the posture estimation processing is the same for each of the vehicle-mounted cameras 21 to 24, the posture estimation processing for the front camera 21 will be described below as an example.

As shown in FIG. 3, first, the estimation unit 121 monitors whether or not the vehicle equipped with the front camera 21 is traveling straight (step S11). Whether or not the host vehicle is traveling straight can be determined, for example, based on rotation angle information of the steering wheel obtained from the steering angle sensor 3 . For example, when the steering wheel rotation angle is zero and the vehicle is traveling in a straight line, it will not only be possible when the rotation angle is zero, but also when the rotation angle is within a certain range between positive and negative directions. It may be determined that the own vehicle is going straight including the case where there is. It should be noted that straight running includes both forward straight running and backward straight running.

The estimation unit 121 repeats the monitoring of step S11 until it detects that the host vehicle is traveling straight. In other words, the estimating unit 121 does not proceed with the attitude estimation process unless the host vehicle is traveling straight. According to this, the posture is estimated using the positional changes of feature points (details will be described later) during straight movement, and the posture is estimated using information when the traveling direction of the own vehicle is curved. Since no estimation is performed, it is possible to avoid complicating the posture estimation process. However, a configuration may be adopted in which the attitude estimation process is advanced using the positional changes of the feature points when the traveling direction of the host vehicle is curved.

In this embodiment, when it is determined that the host vehicle is traveling straight (Yes in step S11), the estimation unit 121 acquires a predetermined number of frames of captured images from the front camera 21 via the acquisition unit 11 (step S12). The estimating unit 121 acquires, for example, several frames to several tens of captured images.

Next, the estimation unit 121 extracts feature points from each frame image (step S13). A feature point is a point that can be conspicuously detected in a captured image, such as an intersection of edges in the captured image. The feature points are, for example, edges of white lines drawn on the road surface, cracks on the road surface, stains on the road surface, gravel on the road surface, and the like. In the present embodiment, as a preferred form, feature points with a high corner degree indicating cornerness are extracted. A corner is where two edges meet. The corner degree can be determined using a known detection method such as Harris operator or KLT (Kanade-Lucas-Tomasi) tracker. In this embodiment, the feature point extraction conditions are the same among the four vehicle-mounted cameras 21 to 24 . However, the feature point extraction conditions may be different among the four vehicle-mounted cameras 21 to 24 .

FIG. 4 is a diagram for explaining extraction of feature points by the estimation unit 121. As shown in FIG. FIG. 4 schematically shows a photographed image P photographed by the front camera 21. As shown in FIG. The captured image P includes an area BO in which the body of the vehicle is reflected. In the example shown in FIG. 4, a first feature point FP1 and a second feature point FP2 having a high corner degree are present in the portion of the number indicating the speed limit drawn on the road surface RS. For this purpose, the estimation unit 121 extracts a first feature point FP1 and a second feature point FP2 existing on the road surface RS.

In FIG. 4, only two feature points FP1 and FP2 are shown for the sake of convenience. be. Conversely, if there is no feature point whose corner degree exceeds the predetermined feature point threshold value, no feature point is extracted. It is possible that not even one feature point is extracted from each frame image.

Returning to FIG. 3, the estimating unit 121 derives an optical flow when the process of extracting feature points for all frame images is completed (step S14). Optical flow is a motion vector that indicates the motion of feature points between two captured images captured at different times. In this example, the interval between different times is the same as the frame period of the acquisition unit 11 . However, the interval between different times is not limited to this, and may be, for example, multiple times the frame period of the acquisition unit 11 .

FIG. 5 is a diagram for explaining how the estimation unit 121 derives the optical flow. FIG. 5 is a schematic diagram shown for convenience similar to FIG. FIG. 5 shows a photographed image P′ (current frame P′ for convenience) photographed by the front camera 21 after one frame cycle has elapsed after photographing the photographed image P (previous frame P for convenience) shown in FIG. 4 . ). After photographing the photographed image P shown in FIG. 4, the own vehicle is traveling straight forward for one frame period. A circle FP1P shown in FIG. 5 indicates the position of the first feature point FP1 at the time of capturing the previous frame P shown in FIG. A circle FP2P shown in FIG. 5 indicates the position of the second feature point FP2 at the time of capturing the previous frame P shown in FIG.

As shown in FIG. 5, when the own vehicle goes straight forward, the first feature point FP1 and the second feature point FP2 existing in front of the own vehicle approach the own vehicle. That is, the first feature point FP1 and the second feature point FP2 appear at different positions between the current frame P' and the previous frame P. FIG. The estimating unit 121 associates the first feature point FP1 of the current frame P′ with the first feature point FP1 of the previous frame P based on pixel values in the vicinity thereof, and calculates each of the associated first feature points FP1. Based on the position, the optical flow OF1 of the first feature point FP1 is derived. Similarly, the estimation unit 121 associates the second feature point FP2 of the current frame P′ with the second feature point FP2 of the previous frame P based on the neighboring pixel values, and , the optical flow OF2 of the second feature point FP2 is derived.

Note that the estimation unit 121 derives the optical flow for each of the acquired frame images as described above. Also, more than two optical flows may be obtained in each frame image. For this reason, the estimation unit 121 may obtain a plurality of optical flow candidates for use in posture estimation. Conversely, the estimation unit 121 may not be able to obtain optical flow candidates for use in posture estimation.

Returning to FIG. 3, after deriving the optical flow, the estimating unit 121 selects an optical flow to be used for subsequent processing according to a predetermined selection condition (step S15). A predetermined selection condition is set such that a set of two optical flows suitable for subsequent pose estimation is selected. If there are a plurality of sets that satisfy the predetermined selection condition, all the sets that satisfy the predetermined selection condition are selected. The number of sets satisfying a predetermined selection condition varies depending on the number of feature points extracted from the captured image. It should be noted that if there is no set that satisfies a predetermined selection condition, subsequent processing cannot be performed. Therefore, the orientation estimation process is terminated. If the orientation estimation process ends at this stage, the orientation estimation result will not be obtained. In this embodiment, the conditions for selecting the optical flow to be used are the same among the four vehicle-mounted cameras 21-24. However, the selection conditions may be different among the four vehicle-mounted cameras 21-24.

Here, the processing after step S16 will be described on the assumption that the set of optical flows selected in step S15 includes the set of the first optical flow OF1 and the second optical flow OF2 shown in FIG.

The estimation unit 121 advances the processing using the optical flow OF1 of the first feature point FP1 and the optical flow OF2 of the second feature point FP2. The estimation unit 121 performs coordinate transformation on the feature points using the internal parameters of the front camera 21 stored in the storage unit 13 . In the coordinate conversion, aberration correction and distortion correction of the front camera 21 are performed. Aberration correction is performed to correct distortion due to aberration of the optical system of the front camera 21 . Specifically, it is correction of distortion such as barrel distortion and pincushion distortion. Distortion correction is performed to correct distortion of the optical system itself of the front camera 21 . Specifically, it is fisheye correction. Coordinate transformation converts the coordinates of the feature points into coordinates obtained on a two-dimensional image when the subject is photographed by perspective projection.

When the feature point coordinate transformation is performed, the estimation unit 121 sets the starting point of the optical flow OF1 of the first feature point FP1 to the vertex SP1 and the end point of the optical flow OF1 of the first feature point FP1 to the vertex, as shown in FIG. A quadrangle QL is virtually formed having the vertex SP2 as the starting point of the optical flow OF2 of the second feature point FP1 and the vertex EP as the ending point of the optical flow OF2 of the second feature point FP2 (step S16). Hereinafter, for the sake of explanation, geometric elements such as quadrilaterals and sides of the quadrilaterals are virtually formed and used. However, in actual processing, processing may be based on vector operations such as the coordinates of feature points and the direction of straight lines, and may not be based on geometric elements having equivalent effects.

When the quadrangle QL is virtually formed, the estimation unit 121 uses the quadrangle QL and the internal parameters of the front camera 21 stored in the storage unit 13 to determine the projection plane IMG of the front camera 21 in the three-dimensional space. The quadrangle QL is moved upward (see FIG. 7) to virtually generate a quadrangle QL1 on the projection plane IMG (step S17).

For the sake of explanation, the edges are defined as follows. A first side SD1 of quadrangle QL1 corresponds to a side connecting vertex SP1 and vertex SP2 in quadrangle QL. That is, it corresponds to the side connecting the first feature point FP1 and the second feature point FP2 in the previous frame P. Similarly, the second side SD2 of quadrangle QL1 corresponds to the side connecting vertex SP2 and vertex EP2 in quadrangle QL. That is, it corresponds to the optical flow OF2 of the second feature point FP2. Similarly, the third side SD3 of quadrangle QL1 corresponds to the side connecting vertex EP1 and vertex EP2 in quadrangle QL. That is, it corresponds to the side connecting the first feature point FP1 and the second feature point FP2 in the current frame P'. Similarly, the fourth side SD4 of quadrangle QL1 corresponds to the side connecting vertex SP1 and vertex EP1 in quadrangle QL. That is, it corresponds to the optical flow OF1 of the first feature point FP1.

Also, a surface is defined as follows (see FIG. 7). A plane including the first side SD1 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a first plane F1. Similarly, a plane including the second side SD2 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a second plane F2. Similarly, a surface including the third side SD3 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a third surface F3. Similarly, a plane including the fourth side SD4 of the quadrangle QL1 and the optical center OC of the front camera 21 is defined as a fourth plane F4.

Next, the estimating unit 121 identifies two sets of planes whose intersection lines with a predetermined plane are parallel to each other (Step S18). A predetermined plane is a plane whose normal is known in advance. Specifically, it is the plane on which the vehicle is moving, that is, the road surface. The predetermined plane does not have to be a strict plane, as long as it can be regarded as a plane when the estimation unit 121 identifies two sets of planes whose intersection lines with the predetermined plane are parallel to each other. good.

In this embodiment, the posture estimation apparatus 1 extracts feature points from two images taken at different times and calculates the optical flow of the feature points when the host vehicle is traveling straight. Also, the feature points are extracted from a stationary object positioned on a predetermined plane such as a road surface. Therefore, in the real world, the calculated optical flow represents the relative position change of the stationary object with respect to the host vehicle. In other words, it is the movement vector of the own vehicle whose direction is reversed.

The second side SD2 and the fourth side SD4 of the quadrangle QL1 both correspond to the optical flow of the feature points, and thus both correspond to the movement vector of the own vehicle in the real world. Therefore, it is assumed that they are parallel to each other on the road surface.

Further, since the first side SD1 and the third side SD3 of the quadrangle QL1 are both in the positional relationship between feature points, they correspond to the positional relationship between stationary objects accompanying the movement of the own vehicle in the real world. The positional relationship before movement corresponds to the first side SD1, and the positional relationship after movement corresponds to the third side SD3. Since the position of the stationary object does not change at this time, the positional relationship does not change before and after the movement. Therefore, it is assumed that they are also parallel to each other on the road surface.

Therefore, the estimating unit 121 determines two sets of planes having parallel lines of intersection with the road surface: the set of the second plane F2 and the fourth plane F4, and the set of the first plane F1 and the third plane F3. identify. In other words, the estimating unit 121 identifies a total of two sets of surfaces including optical flows as one set and surfaces including feature points captured at the same time as another set.

In FIG. 7, a rectangle QL2 represents the position of the first feature point FP1 in the three-dimensional space (real world) at the time of shooting the current frame P', and the position of the second feature point FP2 at the time of shooting the current frame P'. The position in the three-dimensional space, the position in the three-dimensional space of the first feature point FP1 at the time of capturing the previous frame P, and the position in the three-dimensional space of the second feature point FP2 at the time of capturing the previous frame P are It is a rectangle with vertices. The first face F1 includes the first side SD11 of the quadrangle QL2. Similarly, the second surface F2 includes the second side SD12 of the quadrilateral QL2, the third surface F3 includes the third side SD13 of the quadrilateral QL2, and the fourth surface F4 includes the fourth side SD14 of the quadrilateral QL2. At this time, the quadrangle QL2 is assumed to be a parallelogram formed on the road surface as described above.

Next, the estimation unit 121 calculates the normal line of the road surface (step S19). First, the estimating unit 121 obtains the direction of the line of intersection between the surfaces based on the first surface F1 and the third surface F3, which are one of the previously identified sets of surfaces. Specifically, the direction of the line of intersection CL1 between the first surface F1 and the third surface F3 is obtained (see FIG. 8). A direction vector V1 of the line of intersection CL1 is a vector perpendicular to each of the normal vector of the first surface F1 and the normal vector of the third surface F3. Therefore, the estimation unit 121 obtains the direction vector V1 of the line of intersection CL1 by the outer product of the normal vector of the first surface F1 and the normal vector of the third surface F3. Since the first plane F1 and the third plane F3 are parallel to the road surface, the direction vector V1 is parallel to the road surface.

Similarly, the estimating unit 121 obtains the direction of the line of intersection between the second surface F2 and the fourth surface F4, which are the other of the set of previously identified surfaces. Specifically, the direction of the intersection line CL2 between the second surface F2 and the fourth surface F4 is obtained (see FIG. 9). A direction vector V2 of the line of intersection CL2 is a vector perpendicular to each of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Therefore, the estimation unit 121 obtains the direction vector V2 of the line of intersection CL2 by the cross product of the normal vector of the second surface F2 and the normal vector of the fourth surface F4. Similarly, the second surface F2 and the fourth surface F4 also have parallel lines of intersection with the road surface, so the direction vector V2 is parallel with the road surface.

The estimating unit 121 calculates the normal to the surface of the quadrangle QL2, that is, the normal to the road surface, from the outer product of the direction vector V1 and the direction vector V2. Since the normal line of the road surface calculated by the estimating unit 121 is calculated in the camera coordinate system of the front camera 21, the coordinate system of the three-dimensional space is obtained from the difference from the vertical direction, which is the normal line of the actual road surface. The pose of the camera 21 can be estimated. Based on the estimation result, the estimation unit 121 estimates the posture of the front camera 21 with respect to the vehicle (step S20). Note that the calculation process in step S19 can be executed using, for example, the known ARToolkit.

When the orientation estimation in step S20 is completed, the estimation unit 121 checks whether or not orientation estimation has been completed for all optical flow pairs selected in step S15 (step S21). If posture estimation has been completed for all pairs (Yes in step S21), the flow shown in FIG. 3 ends. On the other hand, if posture estimation has not been completed for all pairs (No in step S21), the process returns to step S16 and the processes after step S16 are repeated.

The attitude estimation apparatus 1 uses the movement of the own vehicle to autonomously identify two sets of planes whose intersection lines with a predetermined plane are parallel to each other, so that only the feature point extraction error is the estimation accuracy. can be used to estimate the pose of the camera, which affects the That is, the posture estimation device 1 can perform posture estimation of the camera with few error factors. Therefore, the posture estimation device 1 can accurately estimate the posture of the camera.

In the above description, the plane including the optical center OC of the front camera 21 and one side of the quadrangle QL1 is specified, but this is not the only option. A plane may be specified by specifying the normal direction of the plane. For example, the normal direction of the plane may be obtained from the outer product of the direction vectors from the optical center OC to each vertex, and the plane may be identified by the normal direction. In this case, the normal direction of the first surface F1 and the normal direction of the third surface F3 are set as one set, and the normal direction of the second surface F2 and the normal direction of the fourth surface F4 are set as another set. Two sets may be identified.

Also, the plane may be translated. For example, instead of the first plane F1, a plane translated from the first plane F1 may be paired with the third plane F3. This is because the direction of the line of intersection with a predetermined plane does not change even if it is translated.

Returning to FIG. 2, in step S5, the determination unit 122 performs a process of determining the estimated orientation according to predetermined conditions. Specifically, the determination unit 122 obtains the cumulative number of obtained posture estimation results for each of the four vehicle-mounted cameras 21 to 24 . After the first posture estimation process, the acquired number becomes the accumulated value as it is. From the second time onwards, the cumulative value of the acquired number is obtained by adding the acquired number obtained this time to the cumulative value acquired last time.

The determination unit 122 compares the cumulative number of acquisitions with a predetermined multiple number of times for each of the four vehicle-mounted cameras 21 to 24 . For a camera for which the cumulative number of acquisitions has reached a predetermined number of times or more, the determination unit 122 determines the orientation estimation based on the multiple estimation results obtained at that time. The determining unit 122 uses, for example, an average value of a plurality of estimation results, a median value or a peak value of a histogram created from a plurality of estimation results, or the like as a determined posture estimation value. For a camera for which the cumulative number of acquisitions does not reach a predetermined number of times or more, it is determined that the orientation estimation is not confirmed, and the orientation estimation is not confirmed. When the estimation confirmation process is completed, the confirmation unit 122 advances the process to the next step S6.

In step S6, the confirming unit 122 confirms whether or not the estimation of the orientations of all the vehicle-mounted cameras 21-24 has been confirmed. If the orientations of all the vehicle-mounted cameras 21 to 24 have been estimated (Yes in step S6), the process shown in FIG. 2 ends. In this case, the estimation of the orientation of all the on-vehicle cameras 21 to 24 is confirmed, and the confirmation result of the orientation estimation can be output from the orientation estimation device 1 . It should be noted that each time a camera for which pose estimation has been determined occurs, the determination result of pose estimation may be output to the outside. For example, it is determined whether or not the on-vehicle cameras 21 to 24 are in a state of misalignment based on the determination result of posture estimation. When it is determined that the on-vehicle cameras 21 to 24 are dislocated, for example, the user of the vehicle is notified of the abnormal situation. As a result, the user of the own vehicle can take measures such as requesting the dealer to adjust the mounting of the camera.

Further, the confirmed result of the posture estimation of each of the vehicle-mounted cameras 21-24 may be used for correcting the parameters of each of the vehicle-mounted cameras 21-24. As a result, even if there is a misalignment in the mounting of the vehicle-mounted cameras 21-24, camera calibration is executed according to the misalignment, and the adverse effect of the mounting misalignment of the vehicle-mounted cameras 21-24 on the captured image is eliminated. can be suppressed. Further, when the photographed images are transmitted to a center which is provided so as to be able to communicate with a plurality of vehicles via a network, the posture estimation result may be transmitted to the center together with the photographed images. As a result, it is possible to appropriately handle the photographing information at the center.

If there are onboard cameras 21 to 24 whose orientations have not been estimated (No in step S6), the process proceeds to step S7.

In step S7, the setting unit 123 determines whether or not it is necessary to change the priority of each of the vehicle-mounted cameras 21-24. FIG. 10 is a flowchart showing an example of priority change determination processing in the posture estimation apparatus 1 of the first embodiment. FIG. 10 is a flow chart showing details of the process of step S7 in FIG.

As shown in FIG. 10, first, the setting unit 123 confirms whether or not there are any vehicle-mounted cameras 21 to 24 whose orientations have been estimated (step S71). More specifically, when there is a new camera whose posture estimation has been confirmed in the estimation confirmation processing performed immediately before, the setting unit 123 determines that there is an in-vehicle camera 21 to 24 whose orientation has been confirmed. Then, if there are any of the vehicle-mounted cameras 21 to 24 whose orientations have been estimated (Yes in step S71), the setting unit 123 determines that the priority needs to be changed (step S72).

That is, in the present embodiment, the setting unit 123 changes the priority when there is a camera whose orientation has been estimated. According to this, it is possible to preferentially perform orientation estimation processing for a camera that needs to accumulate data in order to confirm orientation estimation.

On the other hand, if there is no new camera whose orientation has been estimated in the immediately preceding estimation confirmation processing, the setting unit 123 determines that there is no vehicle-mounted camera 21 to 24 whose orientation has been estimated ( If No in step S71), the following processing is performed. Specifically, the setting unit 123 extracts the maximum number and the minimum number from the accumulated values of the number of estimation results obtained for each of the four vehicle-mounted cameras 21 to 24 . Then, the setting unit 123 confirms whether or not the difference between the maximum number and the minimum number is equal to or greater than a predetermined threshold (step S73).

If the difference between the maximum number and the minimum number is greater than or equal to the predetermined threshold (Yes in step S73), the setting unit 123 advances the process to step S72. That is, in the present embodiment, when the difference between the maximum number and the minimum number of accumulated values of acquisition numbers obtained for each of the cameras 21 to 24 becomes equal to or greater than a predetermined threshold, the priority is changed. According to this, it is possible to suppress the occurrence of bias in the accumulated number of obtained estimation results among the plurality of cameras 21 to 24 .

On the other hand, if the difference between the maximum number and the minimum number has not reached the predetermined threshold value or more (No in step S73), the setting unit 123 determines the difference between the maximum number and the minimum number during the previous priority change determination process. It is checked whether the difference is greater than or equal to a predetermined threshold (step S74). If the difference between the maximum number and the minimum number is greater than or equal to the predetermined threshold value in the previous priority change determination process (Yes in step S74), the setting unit 123 advances the process to step S72.

That is, in the present embodiment, the setting unit 123 changes the priority when the difference between the maximum number and the minimum number becomes smaller than the predetermined threshold after becoming equal to or greater than the predetermined threshold. According to this, it is possible to appropriately review the setting of the priority and suppress the occurrence of bias in the accumulated value of the acquisition number of the estimation result.

When the difference between the maximum number and the minimum number is smaller than the predetermined threshold value in the previous priority change determination process (No in step S74), the setting unit 123 determines that the priority change is unnecessary (step S75).

Returning to FIG. 2, when it is determined in step S7 that the priority is to be changed (Yes in step S7), the process proceeds to step S8. If it is determined in step S7 that the priority need not be changed (No in step S7), the process proceeds to step S10.

In step S8, the setting unit 123 changes the priority setting. If there are any of the vehicle-mounted cameras 21 to 24 whose orientations have been estimated, the setting unit 123 lowers the priority of the cameras. When the difference between the maximum number and the minimum number of the cumulative number of estimation results obtained for each of the vehicle-mounted cameras 21 to 24 is equal to or greater than a predetermined threshold, the setting unit 123 sets the cumulative value of the number of estimation results to be acquired. Decrease the priority of the camera with the most If there is a camera whose difference between the maximum number and the minimum number has become smaller than the predetermined threshold after the difference between the maximum number and the minimum number has become equal to or greater than the predetermined threshold, the setting unit 123 restores the priority lowered in the previous round. In this embodiment, when the priority is lowered, the order of posture estimation processing is skipped. By skipping the order of cameras with low priority, the processing of cameras with high priority can be given priority in terms of time. After setting the priority by the setting unit 123, the process proceeds to the next step S9.

In step S<b>9 , the setting unit 123 sequentially performs posture estimation processing for the cameras to be subjected to estimation processing according to the changed priority. When the estimation process is completed for all the cameras targeted for the estimation process, the process returns to step S5.

In step S<b>10 , the setting unit 123 maintains the same priority as the previous time, and sequentially performs posture estimation processing for the cameras to be estimated. When the estimation process is completed for all the cameras targeted for the estimation process, the process returns to step S5.

FIG. 11 is a diagram for explaining priority setting in the posture estimation apparatus 1 of the first embodiment. The priority setting in the posture estimation device 1 will be further described based on the specific example shown in FIG. 11 . In the example shown in FIG. 11, there is no priority difference between the four vehicle-mounted cameras 21 to 24 by default. By default, each of the vehicle-mounted cameras 21 to 24 is given an order of attitude estimation processing once. In FIG. 11, "implemented" means that the posture estimation process is executed, and "not implemented" means that the posture estimation process is not executed. The cumulative values shown in FIG. 11 are cumulative values of the acquisition number of posture estimation results. In the example shown in FIG. 11, when the cumulative value reaches 80 (=predetermined multiple times) or more, the posture estimation is finalized. A predetermined threshold value for determining the difference between the maximum number and the minimum number of accumulated values is 20.

In the first time, there is no difference in the priority of the estimation process, and the estimation process is performed in the order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24. FIG. The accumulated values of the vehicle-mounted cameras 21 to 24 after the estimation process are all 10, and the posture estimation is not confirmed for any of the cameras. Also, the difference between the maximum number and the minimum number of cumulative values is zero, which is smaller than the predetermined threshold (=20). Furthermore, since it is the first time, there is no data for the previous time, and there is no fact that the difference between the maximum number and the minimum number of accumulated values was equal to or greater than a predetermined threshold value the previous time. Therefore, in the priority change determination process performed after the estimation process for the four vehicle-mounted cameras 21 to 24, it is determined that the priority setting need not be changed.

In the second time, estimation processing is performed in the order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24, as in the first time. None of the accumulated values of the vehicle-mounted cameras 21 to 24 after the estimation process reaches 80 or more, and the estimation of the posture is not confirmed for any of the cameras. On the other hand, the difference between the maximum number (=50) and the minimum number (=20) of cumulative values is 30, which is larger than the predetermined threshold (=20). For this reason, in the priority change determination process that is performed after the estimation process for the four vehicle-mounted cameras 21 to 24, it is determined that the priority setting needs to be changed. In this example, the priority is lowered for cameras whose cumulative value is greater than the minimum number by 20 or more (this setting may be changed as appropriate). That is, the priority of the front camera 21 and the back camera 22 is lowered, and the next (third) order is skipped.

In the third time, estimation processing is performed in order of the left side camera 23 and the right side camera 24 . None of the accumulated values of the vehicle-mounted cameras 21 to 24 after the estimation process reaches 80 or more, and the estimation of the posture is not confirmed for any of the cameras. The difference between the maximum number (=50) and the minimum number (=40) of cumulative values is smaller than the predetermined number (=20). However, since the difference between the maximum and minimum cumulative values was larger than the predetermined threshold at the time of the previous confirmation, it corresponds to the case where the difference becomes smaller than the predetermined threshold after the difference becomes equal to or greater than the predetermined threshold. do. Therefore, in the priority change determination process performed after the estimation process of the two vehicle-mounted cameras 23 and 24, it is determined that the priority setting needs to be changed. Specifically, the priority setting of the front camera 21 and the back camera 22 whose priority was lowered last time is restored. That is, there is no difference in the priority of the estimation processing of the four vehicle-mounted cameras 21-24, and the estimation processing of the four vehicle-mounted cameras 21-24 is performed next time (fourth time).

In the fourth time, estimation processing is performed in order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24. FIG. After the estimation process, the cumulative value reaches 80 or more for the front camera 21, and the estimation of the orientation is confirmed. For this reason, in the priority change determination process that is performed after the estimation process for the four vehicle-mounted cameras 21 to 24, it is determined that the priority setting needs to be changed. Specifically, the priority setting of the front camera 21 whose orientation has been estimated is lowered, and the next (fifth) order is skipped.

In this example, the maximum number of accumulated values is 80 and the minimum number is 65 at the fourth time. For this reason, the difference between the maximum number and the minimum number is smaller than a predetermined threshold (=20), and the priority setting is not changed based on this point of view. However, it is assumed that the cumulative value of the right side camera 24 is 50. In this case, since the difference between the maximum number (=80) and the minimum number (=50) is equal to or greater than the predetermined threshold (=20), the priority is set in consideration of the difference between the maximum number and the minimum number. is done. For example, the priority of the back camera 22 and the left side camera 23 whose cumulative value is equal to or greater than a predetermined number (20 in this example) from the minimum number is lowered.

In the fifth time, estimation processing is performed in the order of the back camera 22, the left side camera 23, and the right side camera 24. FIG. The cumulative value reaches 80 or more for all the three cameras 22 to 24 that have undergone the estimation process, and the pose estimation is finalized for these cameras 22 to 24 . In this case, the posture estimation for all four on-vehicle cameras 21 to 24 has been determined, so the series of posture estimation processing is temporarily terminated.

In the above description, all the four onboard cameras 21 to 24 are always used when determining the maximum number and minimum number of cumulative values, but this configuration is not necessary. For example, when a camera whose orientation has been estimated is generated, the camera may be excluded and the maximum number and the minimum number of cumulative values are compared. That is, a configuration may be adopted in which the comparison between the maximum number and the minimum number of accumulated values is performed by paying attention only to the camera for which the orientation estimation process has not been determined.

<2. Second Embodiment>
Next, a posture estimation device according to the second embodiment will be described. In the explanation of the second embodiment, the explanation of the contents overlapping with the first embodiment will be omitted if there is no particular need for explanation.

FIG. 12 is a diagram for explaining the configuration of a posture estimation device 1A according to the second embodiment. As shown in FIG. 12 , in the second embodiment, the function realized by the CPU of the control unit 12A executing arithmetic processing according to the program stored in the storage unit 13 includes the judgment unit 124 . That is, posture estimation apparatus 1A includes determination section 124 . This point is different from the first embodiment.

The determination unit 124 determines whether or not there is a camera having a priority condition other than the number of obtained estimation results among the plurality of in-vehicle cameras 21 to 24 . The priority condition based on factors other than the number of estimation results obtained may be, for example, a priority condition based on the driver's field of view, a priority condition based on the operation status of the own vehicle, a priority condition based on information from the navigation device, or the like.

A priority condition based on the driver's field of view is stored in the storage unit 13 in advance, for example. For example, the front camera 21 and the right side camera 24 (assuming that the driver's seat is on the right side) that capture images in a direction in which the driver's field of view is easily secured are replaced by the rear camera 22 and the left side camera that captures a direction in which the driver's field of vision is difficult to secure. The priority of estimation processing is lowered compared to the camera 23 .

The priority condition based on the operation status of the host vehicle may be set by the determination unit 124 based on, for example, information obtained from the steering angle sensor 3 via the communication bus B1, information obtained from a shift sensor (not shown), or the like. Alternatively, it may be input from the outside. For example, when the shift sensor is in reverse, the priority of the estimation process of the back camera 22 is raised.

The priority condition based on the information from the navigation device may be set by the determination unit 124 based on information obtained from a navigation device (not shown) connected to the posture estimation device 1A, or may be input from the outside. For example, when a right turn or a left turn is assumed ahead of the route set in the navigation device, the priority of the estimation process of the camera corresponding to the turning direction is raised.

The setting unit 123 sets the priority of the estimation process based on the priority condition when the vehicle-mounted cameras 21 to 24 having the priority condition exist. According to this, it is possible to change the method of setting the priority of the estimation process in accordance with the driving scene of the vehicle, etc., and perform the posture estimation of the camera suitable for the driving scene.

FIG. 13 is a flow chart showing an example of processing executed when estimating the postures of the four vehicle-mounted cameras 21 to 24 in the posture estimation device 1A of the second embodiment. At the start of the process of FIG. 13, the priority of the posture estimation process for each of the four vehicle-mounted cameras 21 to 24 is the same. That is, at the start of the processing of FIG. 13, there is no difference in the frequency of estimation processing of the postures of the four vehicle-mounted cameras 21-24. Note that in this example, when a new priority condition occurs, the old priority condition is updated. Also, when the orientation of the camera to be prioritized is estimated according to the priority condition, it is treated as if there is no priority condition unless a new priority condition occurs. However, the handling of this priority condition is merely an example, and may be changed as appropriate.

In step S31, the determination unit 124 confirms whether or not there is a camera having a priority condition (hereinafter sometimes simply referred to as a priority condition) based on factors other than the number of acquisitions in the estimation process. The determination unit 124 checks the information stored in the storage unit 13, the CAN information obtained using the communication bus B1, the route information obtained from the navigation device, and the like to determine whether or not there is a camera having a priority condition. to judge whether If the determining unit 124 determines that there is a camera having a priority condition (Yes in step S31), the process proceeds to step S32. On the other hand, if the determination unit 124 determines that there is no camera having a priority condition (No in step S31), the process proceeds to step S34.

In step S32, the setting unit 123 sets the priority of each vehicle-mounted camera 21-24 according to the priority condition. As a result, the priority setting is changed from the setting at the start of processing. After the priority is set by the setting unit 123, the process proceeds to step S33.

In step S33, the estimation process is sequentially performed for the cameras to be subjected to the attitude estimation process according to the changed priority. It should be noted that there may be a plurality of cameras or a single camera whose orientation is to be estimated. When the estimation process is completed for all the cameras to be estimated, the process proceeds to step S35.

In step S34, the priority is not changed because there is no camera with a priority condition, and the camera to be the target of the estimation process is estimated with the priority set at the start of the process (same for all cameras 21 to 24). Processing is done in order. In this example, posture estimation processing is performed in the order of the front camera 21, the back camera 22, the left side camera 23, and the right side camera 24. FIG. When the posture estimation processing for all the four vehicle-mounted cameras 21 to 24 is completed, the processing proceeds to step S35.

In step S35, the determination unit 122 performs a process of determining the estimated posture according to a predetermined condition. Since this process is the same as step S5 in FIG. 2 described above, description thereof is omitted. When the estimation confirmation process is completed, the confirmation unit 122 advances the process to the next step S36.

In step S36, the determination unit 122 confirms whether or not the estimation of the orientations of all the vehicle-mounted cameras 21 to 24 has been determined. If the orientations of all the vehicle-mounted cameras 21 to 24 have been estimated (Yes in step S36), the process shown in FIG. 13 ends. If there are onboard cameras 21 to 24 whose orientations have not been estimated (No in step S36), the process proceeds to step S37.

In step S37, the setting unit 123 determines whether or not it is necessary to change the priority of each of the vehicle-mounted cameras 21-24. FIG. 14 is a flowchart showing an example of priority change determination processing in the posture estimation apparatus 1A of the second embodiment. FIG. 14 is a diagram showing details of the process of step S37 in FIG.

The setting unit 123 first confirms whether or not a new priority condition has occurred (step S371). A new priority condition is a priority condition that is not taken into consideration in the priority for each of the cameras 21 to 24 set at that time. If a new priority condition occurs (Yes in step S371), the setting unit 123 determines that the priority needs to be changed (step S372).

If a new priority condition has not occurred (No in step S371), the setting unit 123 confirms whether or not there are any vehicle-mounted cameras 21 to 24 whose orientations have been estimated (step S373). The confirmation process is the same as step S71 in FIG. If there are any vehicle-mounted cameras 21 to 24 whose orientations have been estimated (Yes in step S373), the setting unit 123 determines that the priority needs to be changed (step S372).

If there is no in-vehicle camera 21 to 24 whose orientation has been estimated (No in step S373), the setting unit 123 checks whether the current priority is set according to the priority condition (step S374). If the current priority is set according to the priority condition (Yes in step S374), it is determined that the priority setting need not be changed (step S377).

If the current priority is not set according to the priority condition (No in step S374), the setting unit 123 confirms whether or not the difference between the maximum number and the minimum number of cumulative values is equal to or greater than a predetermined threshold (step S375). The processing is the same as step S73 shown in FIG. Step S376 performed after this is the same as step S74 shown in FIG. Therefore, description of these will be omitted.

Returning to FIG. 13, when it is determined in step S37 that the priority is to be changed (Yes in step S37), the process proceeds to step S38. If it is determined in step S37 that the priority need not be changed (No in step S37), the process proceeds to step S40.

The processing of steps S38 to S40 is the same as the processing of steps 8 to S10 shown in FIG. For this reason, the description is omitted. In the case of changing the priority setting, if a new priority condition occurs, the previous priority condition is discarded and the priority is set according to the new priority condition. Different from the form.

FIG. 15 is a diagram for explaining priority setting in the posture estimation apparatus 1A of the second embodiment. The priority setting in the posture estimation apparatus 1A will be further described based on the specific example shown in FIG. In the example shown in FIG. 15, a priority condition is stored in the storage unit 13 so that the estimation processing of the back camera 22 and the left side camera 23 is prioritized. In FIG. 15, "implemented" means that the posture estimation process is executed, and "not implemented" means that the posture estimation process is not executed. The cumulative values shown in FIG. 15 are the cumulative values of the acquired number of posture estimation results. In the example shown in FIG. 15, posture estimation is determined when the cumulative value reaches 80 (=predetermined multiple times) or more.

At the first time, the priority of the posture estimation processing of the back camera 22 and the left side camera 23 is raised according to the priority condition stored in the storage unit 13 . As a result, in the first time, the order of the front camera 21 and the right side camera 24 is skipped, and the estimation process is performed in the order of the back camera 22 and the left side camera 23 . Neither the cumulative value (=30) of the back camera 22 nor the cumulative value (=10) of the left side camera 23 after the estimation process has reached 80 or more, and the posture estimation is not confirmed for either camera. . Also, the current priority settings are in accordance with the priority conditions. Therefore, in the priority change determination process performed after the estimation process of the back camera 22 and the left side camera 23, if no new priority condition is found, it is determined that the priority setting need not be changed. In this example, no new priority condition occurs at this stage, and the priority setting is not changed.

In this example, at the end of the first time, the difference in accumulated value between the back camera 22 and the left side camera 23 is large. In order to eliminate this difference, a configuration may be adopted in which the estimation processing of the left side camera 23 is prioritized (the order of the back camera 22 is skipped) in the second time. That is, the priority may be changed when the difference between the maximum number and the minimum number of cumulative values of the number of estimation results obtained for each camera to be prioritized is equal to or greater than a predetermined threshold.

In the second time, estimation processing is performed in the order of the back camera 22 and the left side camera 23, as in the first time. Here also, the accumulated values of the back camera 22 and the left side camera 23 after the estimation process have not reached 80 or more, and the estimation of the posture is not confirmed for either camera. Also, the current priority settings are in accordance with the priority conditions. Therefore, in the priority change determination process performed after the estimation process of the back camera 22 and the left side camera 23, if no new priority condition is found, it is determined that the priority setting need not be changed. In this example, a new priority condition arises at this stage. Therefore, it is determined that the priority setting needs to be changed. The new priority condition is a condition generated by obtaining information from the navigation device that there is an upcoming left turn. The priority setting is changed due to the occurrence of a new priority condition. As a result, only the estimation processing of the left side camera 23 is performed next time (third time).

At the third time, the estimation process is performed only for the left side camera 23 . The accumulated value of the left side camera 23 after the estimation process has not reached 80 or more, and the estimation of the orientation is not confirmed. Also, the current priority settings are in accordance with the priority conditions. Therefore, in the priority change determination process performed after the estimation process of the left side camera 23, if no new priority condition is found, it is determined that the priority setting need not be changed. In this example, no new priority condition occurs at this stage, and the priority setting is not changed.

In the fourth time, estimation processing is performed only for the left side camera 23 . For the left side camera 23 after the estimation process, the cumulative value reaches 80 or more, and the estimation of the orientation is confirmed. Therefore, in the priority change determination process performed after the left side camera 23 estimation process, it is determined that the priority setting needs to be changed. The left side camera 23 whose orientation has been estimated has its priority set lower, and the next (fifth) order is skipped. Further, since the orientation estimation of the left side camera 23 is confirmed, there is no priority condition. As a result, the result of whether or not the difference between the maximum number (=85) and the minimum number (=0) of the cumulative values obtained for each of the vehicle-mounted cameras 21 to 24 is equal to or greater than a predetermined threshold is considered. Priority can be determined. As a result, the priority of the back camera 22 with a large accumulated value is lowered, and the next order is skipped.

In the fifth time, the estimation process is performed in the order of the front camera 21 and the right side camera 24 . Neither of the accumulated values of the front camera 21 and the right side camera 24 after the estimation process has reached 80 or more, and the orientation estimation has not been confirmed except for the previously confirmed left side camera 23 . Therefore, even after this, the estimation process for determining the posture estimation is continued according to the above-described flowchart shown in FIG. 13 and the like.

<3. Notes>
The configurations of the embodiments and modifications in this specification are merely examples of the present invention. The configurations of the embodiments and modifications may be changed as appropriate without departing from the technical idea of the present invention. Also, multiple embodiments and modifications may be implemented in combination within a possible range.

The method of estimating the orientation of the camera described above is merely an example. The method of estimating the orientation of the camera may be changed as appropriate. The estimation processing of the posture of the camera may be performed by, for example, comparing the movement of the feature points extracted from the captured image and the movement information (movement amount and speed) of the own vehicle.

INDUSTRIAL APPLICABILITY The present invention can be used for estimating the posture of a camera that assists the driving of a moving object such as automatic parking. The present invention can also be used to estimate the orientation of a camera that records driving information, such as a drive recorder. Further, the present invention can be used for a correction device or the like that corrects a photographed image by using the estimation information of the orientation of the camera. Further, the present invention can be used for a device or the like that operates in cooperation with a center that is provided so as to be able to communicate with a plurality of moving bodies via a network. For example, when transmitting a captured image to the center, the apparatus may be configured to transmit estimation information of the orientation of the camera together with the captured image. Then, at the center, using the camera posture estimation information, various image processing (processing to change the viewpoint and viewing direction of the image considering the camera posture, such as changing the viewpoint and viewing direction to the image in front of the vehicle body) ), correction processing for camera posture in measurement processing using images, statistical processing of changes in camera posture over time (data from many vehicles), etc. to generate useful data to be provided to users. and so on.

1, 1A posture estimation device 21 to 24 in-vehicle camera 11 acquisition unit 121 estimation unit 122 determination unit 123 setting unit 124 determination unit

Claims (9)

an acquisition unit that acquires a captured image from each of a plurality of cameras mounted on a moving object;
an estimation unit that performs estimation processing for estimating the orientation of the camera based on the captured image;
a setting unit that sets the priority of the estimation process for each camera based on the number of estimation results obtained by the estimation process performed for each camera;
A posture estimation device comprising: 2. The posture estimation apparatus according to claim 1, wherein said setting unit sets said priority based on a comparison result of said accumulated number of acquisitions for said cameras. 3. The posture estimation according to claim 1, further comprising a determination unit that determines the estimation of the posture based on a plurality of the estimation results for the camera for which the cumulative value of the obtained number has reached a predetermined number of times or more. Device. 4. The posture estimation apparatus according to claim 3, wherein said setting unit changes said priority when said camera for which said posture estimation has been confirmed occurs. 5. The setting unit according to any one of claims 1 to 4, wherein the setting unit changes the priority when a difference between a maximum number and a minimum number of cumulative values obtained for each camera becomes equal to or greater than a predetermined threshold. The attitude estimation device according to . The posture estimation apparatus according to claim 5, wherein the setting unit changes the priority when the difference between the maximum number and the minimum number becomes smaller than the predetermined threshold after becoming equal to or greater than the predetermined threshold. further comprising a determination unit that determines whether or not there is a camera having a priority condition based on a condition other than the number of acquisitions among the plurality of cameras;
The posture estimation apparatus according to any one of claims 1 to 4, wherein when there is a camera having the priority condition, the setting unit sets the priority based on the priority condition. The setting unit confirms whether or not to change the priority at a predetermined timing,
The predetermined timing occurs each time the order of the estimation process is given to each of the plurality of cameras,
8. When there is a camera among the plurality of cameras whose order is skipped due to the setting of the priority, the predetermined timing is determined on the assumption that the order is given to the camera. The posture estimation device according to any one of . an acquisition step of acquiring a photographed image from each of a plurality of cameras mounted on a moving object;
an estimation step of performing an estimation process for estimating the orientation of the camera based on the captured image;
a setting step of setting the priority of the estimation process for each camera based on the number of estimation results obtained by the estimation process performed for each camera;
A pose estimation method comprising:

Images