JP7358933B2 - camera calibration device - Google Patents

Description

Embodiments of the present invention relate to a camera calibration device.

Detects two feature points where two marking lines intersect and the direction of the two marking lines from images taken around the moving object by multiple cameras mounted on a moving object such as a vehicle. Based on the results, a bird's-eye view image of the moving object and its surroundings is generated, and the camera is calibrated so that the two feature points have a horizontal or vertical positional relationship and the two division lines intersect at right angles in the bird's-eye view image. Techniques have been developed to carry out this process.

Patent No. 5491235

However, with the above technique, it is difficult to calibrate the camera unless the road surface on which the moving object travels is flat and there are no marking lines on the road surface.

Therefore, one of the problems of the embodiment is to provide a camera calibration device that can perform camera calibration even when the road surface on which a moving object runs is not flat or when there are no marking lines on the road surface. That's true.

As an example, the camera calibration device of the embodiment acquires an image of the surroundings of the moving object by a first imaging section and a second imaging section different from the first imaging section, which are mounted on the moving object. an acquisition unit; a KF selection unit that selects a first key frame that is the frame at a predetermined timing from among the frames imaged by each of the first imaging unit and the second imaging unit; a camera position and orientation estimation unit that estimates position and orientation information indicating the respective positions and orientations of the first imaging unit and the second imaging unit based on the image capturing unit; and a camera position and orientation estimation unit that extracts feature points existing in the frame, a map creation unit that creates a surrounding map of each of the first imaging unit and the second imaging unit and including the position and orientation information based on the surrounding map of the first imaging unit; , the surrounding map of the second imaging unit, a map integrating unit that creates an integrated map that integrates the surrounding map of the second imaging unit, the position and orientation information of the first imaging unit in the integrated map, and the position of the second imaging unit and a camera parameter calculation unit that calculates a camera parameter indicating a relationship in position and orientation between the first imaging unit and the second imaging unit based on the orientation information. Therefore, as an example, even when a specific object such as a flat road surface or a lane marking does not exist, the relationship between the position and orientation information between the first and second imaging units can be calculated.

Further, in the camera calibration device of the embodiment, for example, the predetermined timing is a timing at which a predetermined period of time has elapsed since the previous first key frame was captured, or a timing when the previous first key frame was captured. This is the timing at which the moving body moves a predetermined distance after being imaged. Therefore, as an example, it is possible to improve the calculation accuracy of the relationship between position and orientation information of a plurality of imaging units.

Further, the camera calibration device of the embodiment further includes, as an example, a scene search unit that searches for a similar portion between the frame of the first imaging unit and the frame of the second imaging unit, and the map integration The part includes correspondence of the similar parts between the frame of the first imaging unit and the frame of the second imaging unit, and feature points of the surrounding maps of each of the first imaging unit and the second imaging unit. The integrated map is created based on the correspondence between the and the similar locations. Therefore, as an example, a highly accurate integrated map can be created.

Further, in the camera calibration device of the embodiment, as an example, the map creation unit may set a predetermined value to a predetermined number or ratio of matching feature points of a predetermined frame and feature points of the frame before the predetermined frame. In the following cases, the predetermined frame is selected as a second key frame, and the surrounding map is created based on the feature points existing in the second key frame. Therefore, as an example, the accuracy of the surrounding map can be improved.

Further, the camera calibration device of the embodiment further includes, as an example, an overall optimization unit that entirely optimizes the integrated map, and the camera parameter calculation unit is configured to perform the overall optimization of the integrated map. The camera parameters are calculated based on the position and orientation information of the first imaging unit and the position and orientation information of the second imaging unit. Therefore, as an example, the relationship between the position and orientation information between the first and second imaging units can be determined with higher accuracy.

Further, in the camera calibration device of the embodiment, as an example, the moving object is a vehicle, and the scene search unit is configured to retrieve the frame of the first imaging unit and the second imaging unit when the vehicle is parked. The similar portion of the frame is searched for.

Further, in the camera calibration device of the embodiment, as an example, the scene search unit may search the frame of the first imaging unit before the vehicle turns back, and the frame of the second imaging unit after the vehicle turns back. The similar portion of the frame is searched. Therefore, as an example, the processing load due to searching for similar locations can be reduced.

Further, in the camera calibration device of the embodiment, as an example, the scene search unit may search the frame of the first imaging unit in an intermediate area between the parking start position and the turning position of the vehicle, and The similar portion to the frame of the second imaging unit is searched for in an intermediate region from the turning position to the parking target position. Therefore, as an example, the processing load due to searching for similar locations can be further reduced.

FIG. 1 is a perspective view showing an example of a state in which a part of the cabin of a vehicle in which the camera calibration device according to the present embodiment is mounted is seen through. FIG. 2 is a plan view of an example of a vehicle according to this embodiment. FIG. 3 is a block diagram showing an example of the functional configuration of the vehicle according to this embodiment. FIG. 4 is a block diagram showing an example of the functional configuration of an ECU included in the vehicle according to the present embodiment. FIG. 5 is a diagram for explaining an example of a surrounding map creation process by the ECU included in the vehicle according to the present embodiment. FIG. 6 is a diagram for explaining an example of an integrated map creation process by the ECU included in the vehicle according to the present embodiment. FIG. 7 is a flowchart illustrating an example of the flow of camera parameter calculation processing performed by the ECU included in the vehicle according to the present embodiment.

Exemplary embodiments of the invention are disclosed below. The configuration of the embodiment shown below, and the actions, results, and effects brought about by the configuration are examples. The present invention can be realized by configurations other than those disclosed in the embodiments below, and at least one of various effects and derivative effects based on the basic configuration can be obtained.

The vehicle equipped with the camera calibration device according to the present embodiment may be a vehicle that uses an internal combustion engine as a drive source (internal combustion engine vehicle), or a vehicle that uses an electric motor as a drive source (electric vehicle). The vehicle may be an automobile, a fuel cell vehicle, etc.), or a vehicle using both of these as drive sources (hybrid vehicle). Further, the vehicle can be equipped with various transmission devices, and various devices (systems, parts, etc.) necessary for driving the internal combustion engine and the electric motor. Furthermore, the system, number, layout, etc. of devices related to driving the wheels of the vehicle can be set in various ways.

FIG. 1 is a perspective view showing an example of a state in which a part of the cabin of a vehicle in which the camera calibration device according to the present embodiment is mounted is seen through. As shown in FIG. 1, the vehicle 1 includes a vehicle body 2, a steering section 4, an acceleration operation section 5, a brake operation section 6, a speed change operation section 7, and a monitor device 11. The vehicle body 2 has a compartment 2a in which a passenger rides. A steering section 4, an acceleration operation section 5, a brake operation section 6, a speed change operation section 7, etc. are provided in the vehicle interior 2a, with the driver as a passenger facing the seat 2b. The steering unit 4 is, for example, a steering wheel protruding from the dashboard 24. The acceleration operation unit 5 is, for example, an accelerator pedal located under the driver's foot. The brake operation unit 6 is, for example, a brake pedal located under the driver's feet. The speed change operation unit 7 is, for example, a shift lever protruding from the center console.

The monitor device 11 is provided, for example, at the center of the dashboard 24 in the vehicle width direction (that is, in the left-right direction). The monitor device 11 may have a function such as a navigation system or an audio system, for example. The monitor device 11 includes a display device 8, an audio output device 9, and an operation input section 10. Furthermore, the monitor device 11 may have various operation input units such as switches, dials, joysticks, and push buttons.

The display device 8 is configured with an LCD (Liquid Crystal Display), an OELD (Organic Electroluminescent Display), or the like, and is capable of displaying various images based on image data. The audio output device 9 is composed of a speaker and the like, and outputs various sounds based on audio data. The audio output device 9 may be provided at a different position other than the monitor device 11 within the vehicle compartment 2a.

The operation input unit 10 is configured with a touch panel or the like, and allows the occupant to input various information. Further, the operation input unit 10 is provided on the display screen of the display device 8 and can transmit images displayed on the display device 8 . Thereby, the operation input unit 10 allows the occupant to view the image displayed on the display screen of the display device 8. The operation input unit 10 receives input of various information by the occupant by detecting a touch operation by the occupant on the display screen of the display device 8 .

FIG. 2 is a plan view of an example of a vehicle according to this embodiment. As shown in FIGS. 1 and 2, the vehicle 1 is a four-wheel vehicle or the like, and has two left and right front wheels 3F and two left and right rear wheels 3R. All or some of the four wheels 3 are steerable.

The vehicle 1 is equipped with a plurality of imaging units 15 (vehicle cameras). In this embodiment, the vehicle 1 is equipped with, for example, four imaging units 15a to 15d. The imaging unit 15 is a digital camera having an imaging element such as a CCD (Charge Coupled Device) or a CIS (CMOS Image Sensor). The imaging unit 15 is capable of imaging the surroundings of the vehicle 1 at a predetermined frame rate. The imaging unit 15 then outputs a captured image obtained by capturing the surroundings of the vehicle 1. The imaging units 15 each have a wide-angle lens or a fisheye lens, and can image a range of, for example, 140° to 220° in the horizontal direction. Further, the optical axis of the imaging unit 15 may be set diagonally downward.

Specifically, the imaging unit 15a is located, for example, at the rear end 2e of the vehicle body 2, and is provided on a wall below the rear window of the rear hatch door 2h. The imaging unit 15a can image an area behind the vehicle 1 among the surroundings of the vehicle 1. The imaging unit 15b is located, for example, at the right end 2f of the vehicle body 2, and is provided on the right side door mirror 2g. The imaging unit 15b can image an area around the vehicle 1 to the side of the vehicle. The imaging unit 15c is located, for example, on the front side of the vehicle body 2, that is, on the front end 2c of the vehicle 1 in the longitudinal direction, and is provided on a front bumper, a front grill, or the like. The imaging unit 15c can image an area in front of the vehicle 1 among the surroundings of the vehicle 1. The imaging unit 15d is located, for example, on the left side of the vehicle body 2, that is, on the left end portion 2d in the vehicle width direction, and is provided on the left side door mirror 2g. The imaging unit 15d can image an area on the side of the vehicle 1 among the surroundings of the vehicle 1.

FIG. 3 is a block diagram showing an example of the functional configuration of the vehicle according to this embodiment. Next, an example of the functional configuration of the vehicle 1 according to the present embodiment will be described using FIG. 3.

As shown in FIG. 3, the vehicle 1 includes a steering system 13, a brake system 18, a steering angle sensor 19, an accelerator sensor 20, a shift sensor 21, a wheel speed sensor 22, an in-vehicle network 23, an ECU ( Electronic Control Unit) 14.

The monitor device 11, steering system 13, brake system 18, steering angle sensor 19, accelerator sensor 20, shift sensor 21, wheel speed sensor 22, and ECU 14 are electrically connected via an in-vehicle network 23, which is a telecommunications line. ing. The in-vehicle network 23 is configured by a CAN (Controller Area Network) or the like.

The steering system 13 is an electric power steering system, an SBW (Steer By Wire) system, or the like. Steering system 13 includes an actuator 13a and a torque sensor 13b. The steering system 13 is electrically controlled by the ECU 14 and the like, and steers the wheels 3 by operating the actuator 13a and adding torque to the steering unit 4 to supplement the steering force. The torque sensor 13b detects the torque that the driver applies to the steering section 4, and transmits the detection result to the ECU 14.

The brake system 18 includes an ABS (Anti-lock Brake System) that controls the locking of the brakes of the vehicle 1, an electronic stability control (ESC) that suppresses skidding of the vehicle 1 during cornering, and an electronic stability control (ESC) that increases the braking force. It includes an electric brake system that assists the brakes and BBW (Brake By Wire).

Brake system 18 includes an actuator 18a and a brake sensor 18b. The brake system 18 is electrically controlled by the ECU 14 and the like, and applies braking force to the wheels 3 via the actuator 18a. The brake system 18 detects signs of brake locking, idling of the wheels 3, skidding, etc. from the difference in rotation between the left and right wheels 3, and performs control to suppress locking of the brakes, idling of the wheels 3, and skidding. Execute. The brake sensor 18b is a displacement sensor that detects the position of a brake pedal as a movable part of the brake operation unit 6, and transmits the detection result of the position of the brake pedal to the ECU 14.

The steering angle sensor 19 is a sensor that detects the amount of steering of the steering unit 4 such as a steering wheel. In this embodiment, the steering angle sensor 19 is configured with a Hall element or the like, detects the rotation angle of the rotating portion of the steering section 4 as a steering amount, and transmits the detection result to the ECU 14.

The accelerator sensor 20 is a displacement sensor that detects the position of an accelerator pedal as a movable part of the acceleration operation section 5, and transmits the detection result to the ECU 14.

The shift sensor 21 is a sensor that detects the position of a movable part (bar, arm, button, etc.) of the speed change operation section 7, and transmits the detection result to the ECU 14.

The wheel speed sensor 22 is a sensor that includes a Hall element and the like and detects the amount of rotation of the wheel 3 and the number of rotations of the wheel 3 per unit time, and transmits the detection result to the ECU 14.

The ECU 14 is composed of a computer or the like, and controls overall control of the vehicle 1 through cooperation of hardware and software. Specifically, the ECU 14 includes a CPU (Central Processing Unit) 14a, a ROM (Read Only Memory) 14b, a RAM (Random Access Memory) 14c, a display control section 14d, an audio control section 14e, and an SSD (Solid State Drive) 14f. Equipped with The CPU 14a, ROM 14b, and RAM 14c may be provided within the same circuit board.

The CPU 14a reads a program stored in a nonvolatile storage device such as a ROM 14b, and executes various arithmetic operations according to the program. For example, the CPU 14a executes image processing on image data to be displayed on the display device 8, control of traveling of the vehicle 1 along a target route to a target position such as a parking position, processing related to calibration of the imaging unit 15, etc. .

The ROM 14b stores various programs and parameters necessary for executing the programs.

The RAM 14c temporarily stores various data used in calculations by the CPU 14a.

Among the arithmetic processing performed by the ECU 14, the display control unit 14d mainly processes image data acquired from the imaging unit 15 and output to the CPU 14a, and performs image processing for displaying image data acquired from the CPU 14a on the display device 8. Executes conversion to data, etc.

Among the calculation processing performed by the ECU 14, the audio control unit 14e mainly executes audio processing that is acquired from the CPU 14a and output to the audio output device 9.

The SSD 14f is a rewritable nonvolatile storage unit that continues to store data acquired from the CPU 14a even when the ECU 14 is powered off.

FIG. 4 is a block diagram showing an example of the functional configuration of an ECU included in the vehicle according to the present embodiment.

Next, an example of the functional configuration of the ECU 14 included in the vehicle 1 according to the present embodiment will be described using FIG. 4.

As shown in FIG. 4, the ECU 14 includes an image acquisition unit 401, a camera position and orientation estimation unit 402, a map creation unit 403, a completion determination unit 404, a KF selection unit 405, a scene search unit 406, a map integration unit 407, and an overall optimization unit. section 408 and camera parameter calculation section 409.

For example, when a processor such as the CPU 14a mounted on the circuit board executes a camera calibration program stored in a storage medium such as the ROM 14b or the SSD 14f, the ECU 14 can control the image acquisition unit 401, camera position and orientation estimation unit 402, etc. , a map creation section 403, a completion determination section 404, a KF selection section 405, a scene search section 406, a map integration section 407, an overall optimization section 408, and a camera parameter calculation section 409. Image acquisition unit 401, camera position and orientation estimation unit 402, map creation unit 403, completion determination unit 404, KF selection unit 405, scene search unit 406, map integration unit 407, overall optimization unit 408, and camera parameter calculation unit 409. Part or all of it may be configured by hardware such as a circuit.

The image acquisition unit 401 acquires a captured image (frame) obtained by capturing the surroundings of the vehicle 1 (an example of a moving object) using the imaging unit 15. In this embodiment, the image acquisition unit 401 acquires frames from the imaging unit 15 when the vehicle 1 is parked. As a result, it is possible to calculate camera parameters indicating the relationship between the positions and postures of the plurality of imaging units 15 using frames obtained by imaging the same area by the plurality of imaging units 15. Calculation accuracy can be improved. For example, the image acquisition unit 401 acquires frames from the imaging unit 15 from when the automatic parking of the vehicle 1 is instructed via the operation input unit 10 until the parking of the vehicle 1 is completed.

The KF selection unit 405 is an example of a KF selection unit that selects a first key frame, which is a frame at a predetermined timing, from among frames imaged by each of the plurality of imaging units 15. Here, the predetermined timing is the timing when a predetermined time has elapsed since the previous first key frame was captured, or the timing when the vehicle 1 has moved a predetermined distance after the previous first key frame was captured. be. Note that the predetermined time is a preset time, and may or may not be a fixed time. Further, the predetermined distance is a distance set in advance, and may or may not be a fixed distance. Furthermore, the predetermined timing between the plurality of imaging units 15 may be the same timing, or may be a timing at which the difference is less than or equal to a preset threshold.

Thereby, the relationship between the position and orientation information between the plurality of imaging units 15 can be determined using the position and orientation information estimated based on frames captured at the same timing. As a result, the accuracy of calculating the relationship between the position and orientation information of the plurality of imaging units 15 can be improved.

The camera position and orientation estimation unit 402 is an example of an estimation unit that estimates the position and orientation information of each imaging unit 15 based on the first key frame selected by the KF selection unit 405. Here, the position and orientation information is information indicating the position (hereinafter referred to as self-position) and orientation of the imaging unit 15. For example, the camera position and orientation estimating unit 402 estimates the position and orientation information of the imaging unit 15 based on the first key frame using SFM (Structure from Motion), Visual Slam, or the like. In this embodiment, the camera position and orientation estimating unit 402 estimates the position and orientation information of each imaging unit 15 based on the first key frame, but it also estimates the position and orientation information of each imaging unit 15 based on the second key frame, which will be described later. 15 position and orientation information may be estimated.

The map creation unit 403 is a creation unit that extracts feature points present in the frame imaged by the imaging unit 15 and creates a map of each imaging unit 15 (hereinafter referred to as a surrounding map) based on the feature points. This is an example. Here, the surrounding map is a map around the vehicle 1 (for example, a three-dimensional map). Further, the surrounding map includes the self-position of the imaging unit 15 and the attitude of the imaging unit 15 (namely, position and orientation information).

For example, the map creation unit 403 creates a surrounding map using Visual SLAM (Simultaneous Localization And Mapping), SFM, or the like. Specifically, the map creation unit 403 extracts feature points from the frame captured by the imaging unit 15. Next, the map creation unit 403 creates a surrounding map by determining the distance between the imaging unit 15 and the feature point by triangulation based on the position and orientation information of the imaging unit 15 and the extracted feature points.

In this embodiment, the map creation unit 403 selects the second key frame from the frames imaged by the imaging unit 15. Here, the second key frame is a frame imaged at an important position among the frames imaged by the imaging unit 15 in creating a surrounding map by the map creation unit 403, which will be described later. For example, if the number or ratio of matching feature points of a predetermined frame and feature points of a frame before the predetermined frame is less than or equal to a predetermined value, the map creation unit 403 may select the predetermined frame using the second key. Select as frame. Here, the frame selected as the first key frame may be selected as the second key frame. Then, the map creation unit 403 creates a surrounding map based on the feature points present in the selected second key frame among the frames imaged by the imaging unit 15.

This prevents the interval between frames used to create a surrounding map from becoming too long and insufficient feature points to create a highly accurate surrounding map, thereby improving the accuracy of the surrounding map. . Further, since it is possible to prevent the number of frames used for creating the surrounding map from increasing and the number of feature points extracted from the frames to become too large, it is possible to reduce the processing load due to creating the surrounding map.

In the present embodiment, the map creation unit 403 also calculates the rotational speed of the wheels 3 detected by the wheel speed sensor 22, the angular velocity and acceleration acting on the vehicle 1 detected by an IMU (Inertial Measurement Unit) not shown, etc. Based on this, by correcting the position and orientation information of the imaging unit 15, etc., the influence of scale drift included in the surrounding map is reduced.

The completion determination unit 404 determines whether the movement of the imaging unit 15 has stopped and the creation of the surrounding map by the map creation unit 403 has been completed. In the present embodiment, the completion determination unit 404 determines whether the image capturing unit 15 Determine whether movement has stopped. When the completion determination unit 404 determines that the movement of the imaging unit 15 has stopped, it determines that the creation of the surrounding map by the map creation unit 403 has been completed.

The scene search unit 406 retrieves a frame from one of the plurality of imaging units 15 (hereinafter referred to as a first imaging unit) 15 and an imaging unit different from the first imaging unit 15 among the plurality of imaging units 15. This is an example of a search unit that searches for similar locations (for example, feature points; hereinafter referred to as similar locations) with frame No. 15 (hereinafter referred to as the second imaging unit). In this embodiment, the scene search unit 406 searches for similar locations from the frames used to create the respective surrounding maps of the first and second imaging units 15 . This makes it possible to improve the accuracy of searching for similar locations between the frames of the first and second imaging units 15, so that a highly accurate integrated map can be created by the map integration unit 407, which will be described later.

For example, the scene search unit 406 may express each frame of the first and second imaging units 15 with a feature amount using Bag-Of-Visual Words, Fisher Vectors, Deep Learning, etc. The similarity of local features between each frame is determined. Thereby, the scene search unit 406 searches for feature points captured in the same region among the feature points included in each frame of the first and second imaging units 15 as similar locations.

In the present embodiment, when the completion determination unit 404 determines that the creation of the surrounding map by the map creation unit 403 has been completed, the scene search unit 406 detects similar points in each frame of the first and second imaging units 15. shall be searched.

The map integration unit 407 is an example of an integration unit that creates a map that integrates the surrounding map of the first imaging unit 15 and the surrounding map of the second imaging unit 15 (hereinafter referred to as an integrated map). Specifically, the map integration unit 407 determines the correspondence between similar locations between the respective frames of the first and second imaging units 15 and feature points (hereinafter referred to as , map point group) and similar locations, rotation, translation, scaling, etc. are performed on the respective surrounding maps of the first and second imaging units 15. Thereby, the map integration unit 407 creates an integrated map that integrates the surrounding map of the first imaging unit 15 and the surrounding map of the second imaging unit 15.

In this embodiment, the map integration unit 407 executes scaling of the respective surrounding maps of the first and second imaging units 15, but the rotation speed of the wheels 3 detected by the wheel speed sensor 22, When the absolute scale is determined using the angular velocity, acceleration, etc. acting on the vehicle 1 detected by the IMU (not shown), the rotation An integrated map may be created by performing only translation and translation.

The overall optimization unit 408 executes overall optimization of the integrated map created by the map integration unit 407. Specifically, the overall optimization unit 408 uses the scene search unit 406 to determine the feature points of each frame of the first and second imaging units 15 (in this embodiment, the first key frame and the second key frame). The integrated map is adjusted so that the map point groups of the surrounding maps corresponding to feature points other than the searched similar locations match between the respective surrounding maps of the first and second imaging units 15. As a result, a more accurate integrated map is created, so that the relationship between the position and orientation information between the first and second imaging units 15 can be determined with higher accuracy.

The camera parameter calculation unit 409 calculates camera parameters indicating the relationship between the position and orientation information between the first and second imaging units 15 based on the position and orientation information of each of the first and second imaging units 15 at a predetermined timing of the integrated map. do. As a result, it is possible to create an integrated map without using specific objects such as flat road surfaces or lane markings, so even if specific objects such as flat road surfaces or lane markings do not exist, the first and second imaging The relationship between position and orientation information between the parts 15 can be calculated.

In the present embodiment, the camera parameter calculation unit 409 calculates camera parameters based on the position and orientation information of each of the first and second imaging units 15 at a predetermined timing of the integrated map that has been globally optimized by the global optimization unit 408. do. Thereby, camera parameters can be calculated using a more accurate integrated map, so it is possible to improve the calculation accuracy of the relationship between the position and orientation information between the first and second imaging units 15.

In the present embodiment, the camera parameter calculation unit 409 calculates camera parameters for each imaging unit 15 for an imaging unit 15 different from the imaging unit 15 concerned. Then, the camera parameter calculation unit 409 calculates the camera parameters calculated for each imaging unit 15 based on the camera parameters calculated for each of the plurality of imaging units 15, and applies the camera parameters calculated for each imaging unit 15 to one of the imaging units 15 among the plurality of imaging units 15. The camera parameters are corrected based on the position and orientation information of the camera.

FIG. 5 is a diagram for explaining an example of a surrounding map creation process by the ECU included in the vehicle according to the present embodiment.

Next, an example of the process of creating a surrounding map by the ECU 14 according to the present embodiment will be described using FIG. 5.

Here, as shown in FIG. 5, surrounding maps M1 and M2 of the imaging units 15b and 15c when the vehicle 1 is parked side by side from the parking start position S to the parking target position E via the turning position T. The creation process will be explained. It is assumed that surrounding maps are created in the same manner for the other imaging units 15a and 15d.

When parking the vehicle 1 side by side with respect to the parking target position E, the map creation unit 403 generates, for example, the second key frame of the imaging unit 15c capable of imaging the front of the vehicle 1, as shown in FIG. A surrounding map M1 is created based on the position and orientation information I1-1 to I1-7 estimated based on the first key frame of the section 15c. Here, as shown in FIG. 5, the surrounding map M1 includes position and orientation information I1-1 to I1-7, and map point groups P1 to P8 corresponding to the feature points included in the second key frame of the imaging unit 15c. is included.

Furthermore, as shown in FIG. 5, the map creation unit 403 estimates the image based on the second key frame of the imaging unit 15b that can image the right side of the vehicle 1 and the first key frame of the imaging unit 15b. A surrounding map M2 is created based on the position and orientation information I2-1 to I2-7. As shown in FIG. 5, the surrounding map M2 includes position and orientation information I2-1 to I2-7 and map point groups P1 to P8 corresponding to the feature points included in the second key frame of the imaging unit 15b. .

Furthermore, as shown in FIG. 5, the map creation unit 403 includes position and orientation information I1'-1 to I1'-7 estimated based on the second key frame of the imaging unit 15c in the surrounding map M1. Similarly, as shown in FIG. 5, the map creation unit 403 includes position and orientation information I2'-1 to I2'-7 estimated based on the second key frame of the imaging unit 15b in the surrounding map M2. .

Here, the position and orientation information I1-1 to I1-7, I1'-1 to I1'-7, I2-1 to I2-7, I2'-1 to I2'-7 are as shown in FIG. It is represented by a pyramid such as a triangular pyramid or a square pyramid. The positions of the pyramids in the surrounding maps M1 and M2 represent the self-positions of the imaging units 15c and 15b where the first key frame or the second key frame was imaged. Further, the orientation of the bottom surface BA of the cone represents the attitude of the imaging units 15c and 15b.

FIG. 6 is a diagram for explaining an example of an integrated map creation process by the ECU included in the vehicle according to the present embodiment.

Next, an example of an integrated map creation process by the ECU 14 included in the vehicle 1 according to the present embodiment will be described using FIGS. 5 and 6. Here, an example will be described in which an integrated map M is created by integrating the surrounding map M1 of the imaging unit 15c and the surrounding map M2 of the imaging unit 15b, and camera parameters of the imaging unit 15c and the imaging unit 15 are calculated using the integrated map M. An example will be described in which the integrated map is created and camera parameters are calculated in the same manner for other imaging units 15 as well.

When the completion determination unit 404 determines that the map creation unit 403 has completed creating the surrounding map of each imaging unit 15a to 15d (see FIG. 5), the scene search unit 406 is an example of the first imaging unit 15. Similar points between a frame of a certain imaging unit 15c (in this embodiment, the second key frame) and a frame of the imaging unit 15b, which is an example of the second imaging unit 15 (in this embodiment, the second key frame) search for.

That is, the scene search unit 406 searches for similar locations between the frame of the first imaging unit 15c and the frame of the second imaging unit 15b when the vehicle 1 is parked. This increases the possibility that similar locations can be searched from frames that capture images of the same area, so it is possible to improve the accuracy of searching for similar locations.

In the present embodiment, the scene search unit 406 searches for similar locations between the second key frame of the first imaging unit 15c before the vehicle 1 turns back and the second key frame of the second imaging unit 15b after the vehicle 1 turns back. Search for. As a result, when searching for similar points among the second key frames of the plurality of imaging units 15a to 15d, it is possible to search for similar points among the second key frames of the plurality of imaging units 15a to 15d. Since it is sufficient to search for similar locations, the processing load due to the search for similar locations can be reduced.

At this time, as shown in FIG. It is preferable to search for a similar location to the second key frame of the second imaging unit 15b in the intermediate region R2 up to the target position E.

As a result, when searching for similar points among the second key frames of the plurality of imaging units 15a to 15d, the number of second key frames for searching for similar points can be reduced, so the processing load due to the search for similar points can be reduced. can be further reduced.

Next, as shown in FIG. 6, the map integration unit 407 calculates the correspondence of similar locations between the second key frames of the first and second imaging units 15c and 15b, and the surrounding maps M1 and M2, as described above. Rotation, translation, scaling, etc. are performed on the surrounding maps M1 and M2 based on the correspondence between the map point groups P1 to P8 included in the map and similar locations. In the example shown in FIG. 5, the map integration unit 407 rotates the surrounding map M2 clockwise by 90 degrees, and then rotates the surrounding map M2 so that the map point groups P1 to P8 included in each of the surrounding maps M1 and M2 match. Translate and scale the surrounding maps M1 and M2. Thereby, the map integration unit 407 creates an integrated map M that integrates the surrounding maps M1 and M2, as shown in FIG.

When the integrated map M is created, the camera parameter calculation unit 409 calculates the position and orientation information I1-1 to I1-7, I1'-1 to I1'-7 of the first imaging unit 15c, as shown in FIG. , extracts position and orientation information (for example, position and orientation information I1'-7) at a preset timing (for example, a predetermined timing). Further, as shown in FIG. 6, the camera parameter calculation unit 409 calculates the position and orientation information at a predetermined timing from the position and orientation information I2-1 to I2-7, I2'-1 to I2'-7 of the second imaging unit 15b. (For example, position/orientation information I2'-7) is extracted. Then, the camera parameter calculation unit 409 calculates the camera parameter calculation unit 409, which indicates the relationship between the position and orientation information between the first and second imaging units 15c and 15b, based on the position and orientation information I1′-7 and the position and orientation information I2′-7. Calculate parameters.

FIG. 7 is a flowchart illustrating an example of the flow of camera parameter calculation processing performed by the ECU included in the vehicle according to the present embodiment.

Next, an example of the flow of camera parameter calculation processing by the ECU 14 included in the vehicle 1 according to the present embodiment will be described using FIG. 7 .

When automatic parking of the vehicle 1 is instructed via the operation input unit 10, each time the image acquisition unit 401 acquires a frame, the map creation unit 403 determines the position of the imaging unit 15 for each imaging unit 15. Based on the posture information and the second key frame imaged by the imaging unit 15, a surrounding map is created using Visual SLAM, SFM, etc. (step S701).

The KF selection unit 405 selects a first key frame from each frame of the plurality of imaging units 15 (step S702). Then, the map creation unit 403 includes the position and orientation information estimated based on the first key frame in each surrounding map of the imaging unit 15.

Next, the completion determination unit 404 determines whether the movement of the imaging unit 15 has stopped and creation of the surrounding map of each of the plurality of imaging units 15 has been completed (step S703). If the creation of surrounding maps for each of the plurality of imaging units 15 is not completed (step S703: No), the process returns to step S701, and the map creation unit 403 continues to create surrounding maps.

On the other hand, when the creation of the surrounding map of each of the plurality of imaging units 15 is completed (step S703: Yes), the scene search unit 406 retrieves the second key frame of the first imaging unit 15 among the plurality of imaging units 15, A similar location is searched for with the second key frame of the second imaging unit 15 among the plurality of imaging units 15 (step S704).

Then, the map integration unit 407 integrates the surrounding map of the first imaging unit 15 and the surrounding map of the second imaging unit 15 based on the similar locations searched by the scene search unit 406 to create an integrated map. Create (step S705). Furthermore, the overall optimization unit 408 executes overall optimization of the created integrated map (step S706).

Next, the camera parameter calculation unit 409 calculates the position and orientation information of the first imaging unit 15 and the second imaging unit 15 at a predetermined timing in the overall optimized integrated map. Camera parameters indicating the relationship of position and orientation information between the imaging units 15 are calculated (step S707).

In this way, according to the vehicle 1 according to the present embodiment, it is possible to create an integrated map without using specific objects such as flat road surfaces or lane markings. Even when no object exists, the relationship between the position and orientation information between the first and second imaging units 15 can be calculated.

1... Vehicle, 14... ECU, 14a... CPU, 14b... ROM, 14c... RAM, 14f... SSD, 15... Imaging unit, 401... Image acquisition unit, 402... Camera position and orientation estimation unit, 403... Map creation unit, 404 ... Completion determination section, 405... KF selection section, 406... Scene search section, 407... Map integration section, 408... Overall optimization section, 409... Camera parameter calculation section.

Claims (8)

an image acquisition unit that acquires a frame captured around the moving body by a first imaging unit and a second imaging unit different from the first imaging unit, which is mounted on the moving body;
a KF selection unit that selects a first key frame that is the frame at a predetermined timing from among the frames imaged by each of the first imaging unit and the second imaging unit;
a camera position and orientation estimation unit that estimates position and orientation information indicating positions and orientations of each of the first imaging unit and the second imaging unit, based on the first key frame;
Extracting feature points existing in the frame, and creating a surrounding map that is a map around each of the first imaging unit and the second imaging unit and including the position and orientation information based on the feature points. Cartography department and
a map integration unit that creates an integrated map that integrates the surrounding map of the first imaging unit and the surrounding map of the second imaging unit;
The position and orientation between the first imaging unit and the second imaging unit are determined based on the position and orientation information of the first imaging unit and the position and orientation information of the second imaging unit in the integrated map. a camera parameter calculation unit that calculates camera parameters indicating a relationship;
A camera calibration device comprising: The predetermined timing is a timing at which a predetermined time has elapsed since the previous first key frame was captured, or a timing at which the moving body moves a predetermined distance after the previous first key frame was captured. A camera calibration device according to claim 1. further comprising a scene search unit that searches for a similar portion between the frame of the first imaging unit and the frame of the second imaging unit,
The map integration unit is configured to determine the correspondence of the similar locations between the frame of the first imaging unit and the frame of the second imaging unit, and the surrounding maps of each of the first imaging unit and the second imaging unit. The camera calibration device according to claim 1 or 2, wherein the integrated map is created based on the correspondence between the feature points and the similar locations. The map creation unit sets the predetermined frame as a second key frame when the number or proportion of matching feature points of the predetermined frame and the feature points of the frame before the predetermined frame is equal to or less than a predetermined value. The camera calibration device according to any one of claims 1 to 3, wherein the surrounding map is created based on the selected feature points existing in the second key frame. further comprising an overall optimization unit that entirely optimizes the integrated map,
The camera parameter calculation unit calculates the camera parameters based on the position and orientation information of the first imaging unit and the position and orientation information of the second imaging unit in the overall optimized integrated map. The camera calibration device according to any one of claims 1 to 4. The mobile object is a vehicle,
The camera calibration device according to claim 3, wherein the scene search unit searches for the similar portion between the frame of the first imaging unit and the frame of the second imaging unit when the vehicle is parked. . The scene search unit searches for the similar portion between the frame of the first imaging unit before the vehicle turns back and the frame of the second imaging unit after the vehicle turns back. camera calibration device. The scene search unit includes the frame of the first imaging unit in an intermediate area between a parking start position and a turning position of the vehicle, and the second frame in an intermediate area from the turning position of the vehicle to a parking target position. The camera calibration device according to claim 7, wherein the camera calibration device searches for the similar portion to the frame of the imaging unit.

Images