Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T7/00—Image analysis
  • G06T7/70—Determining position or orientation of objects or cameras
  • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
  • B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
  • B60R2300/40—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the details of the power supply or the coupling to vehicle components
  • B60R2300/402—Image calibration
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10016—Video; Image sequence
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30244—Camera pose
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/30—Subject of image; Context of image processing
  • G06T2207/30248—Vehicle exterior or interior
  • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

• the invention relates to the calibration of camera systems of vehicles.
• the invention relates to a device for calibrating a camera system for a
• the calibration may include the estimation of both intrinsic and extrinsic parameters of the camera system.
• the intrinsic parameters of a camera describe the light paths of the optical system and may include the focal length, the geometry of the image sensor and the focal point.
• the extrinsic parameters of a camera describe the
• Optical, camera-based and vehicle-based calibrations are usually used as successive process steps.
• the intrinsic parameters are determined and checked in a - possibly partially installed - optics. This step usually takes place at the tape end of the optical module lines of the
• the camera-based calibration requires a fully assembled camera, which except the
• Target-based methods are state of the art. More specifically, easily recognizable patterns such as chess boards with a precisely manufactured geometry are used. For reasons of effort and costs, only part of the later sight cone in front of the camera is usually covered by calibration targets.
• the camera installed in a vehicle is measured.
• the aim of this process step is to check the installation position of the camera in the vehicle.
• expensive target-based methods are state of the art. Specifically, the vehicle must be aligned exactly opposite the target. For this purpose, usually a chassis dynamo is used.
• the invention aims to dispense as far as possible with the module-based, camera-based and vehicle-based calibration steps or to reduce them to simple tests. This is accomplished by dynamically calibrating the camera system of the driver assistance system at the vehicle manufacturer's plant without special targets as the vehicle is moved through the factory floor.
• the central one is accomplished by dynamically calibrating the camera system of the driver assistance system at the vehicle manufacturer's plant without special targets as the vehicle is moved through the factory floor.
• Invention aspect is that no highly accurately manufactured and / or measured targets are necessary for the calibration.
• the latter is achieved by the fact that the rigid part of Factory environment measured once and used as a calibration target.
• a first aspect of the invention relates to a device for calibrating a camera system for a vehicle.
• the apparatus has a first receiving unit for receiving first image data, the first image data comprising a plurality of images of a factory environment, the images of the first image data corresponding to a plurality of first recording positions or a plurality of first recording directions, and wherein the first
• Image data was recorded with the camera system to be calibrated. Furthermore, the device has a first
• Extraction unit for extracting features from the first image data and a first association unit for associating features extracted from the first image data with features of the work environment of a database.
• the database includes features of the plant environment and related
• the device has a calibration unit for estimating parameters of the camera system using the first image data and the
• the device according to the invention can be designed for calibrating mono- as well as multi-camera systems.
• the following assumes that the camera system is fixed in the
• the first image data provided by the camera system to be calibrated comprises a plurality of images of a work environment from a plurality of first shot positions or from several first shooting directions.
• the first recording positions and the first recording directions may be unknown.
• the cameras preferably record the images in a time-synchronized manner.
• the device according to the invention for calibrating a camera system can also be designed to support asynchronous image recordings.
• the device for calibrating the camera system has a first extraction unit for extracting features from the first image data. In the process, features are searched in the images of the first image data. In particular, the images of the first image data are searched for punctiform features.
• the first association unit of the device for calibrating the camera system is designed to associate the features extracted from the first image data with features of the work environment of the database.
• the database can be part of the device according to the invention for calibrating a
• the database is stored for example on a server, with which the inventive device for calibration of a
• the database contains
• Position information about the features of the plant environment is the positions of the features relative to a fixed predetermined coordinate system. In certain cases, however, it is sufficient if the positions of the features are limited to
• the intrinsic parameters of the camera system can then be estimated be, if the positions of the features only up to
• the first association unit examines whether the features extracted from the images of the first image data are stored in the database to determine positional information of the features extracted from the first image data.
• Association unit is thus designed to check the similarity of extracted from the images of the first image data features with features of the database. If a feature extracted from the first image data is sufficiently similar to a feature stored in the database, then
• the calibration unit is designed to receive the first image data as well as the position information from the first one
• the calibration unit can be designed to estimate both extrinsic and intrinsic parameters of the camera system.
• the calibration unit may be designed to estimate only a portion of these parameters.
• Camera system can be designed as a separate component in the vehicle.
• the receiving unit the processing unit
• Calibration unit be designed as separate components. However, it is also possible and in many cases advantageous that the functionalities of the receiving unit, the extraction unit, the association unit and the calibration unit are completely or partially realized by means of a single arithmetic unit. Furthermore, the inventive device for
• Calibration of a camera system include a test unit that is designed to check the calibration of the camera system. If the calibration is unsuccessful, the test unit may re-calibrate.
• the device for calibrating a camera system comprises the database with the features of the factory environment and the associated position information.
• An association unit adapted to associate features extracted from a plurality of images of the first image data, and to add the features extracted and associated with a plurality of images of the first image data to the database.
• the first association unit may also include a tracking algorithm that tracks the movement of a feature between temporally successive images of a camera. Furthermore, the first association unit may be configured to add a feature to the database if the feature has been extracted from multiple images of the first image data and is not yet included in the database. The first association unit adds a new feature to the database, but does not store position information for that feature in the database.
• Association units of the newly added feature database may be determined by the calibration unit.
• Calibration unit can therefore be designed to estimate not only extrinsic and intrinsic parameters of the camera system but also position information of features of the plant environment.
• the first association unit is designed to detect moving objects of the factory environment, and to remove and discard features of the moving objects of the factory environment from the database.
• Association unit to be detected to detect moving objects of the plant environment and remove features of the moving objects of the plant environment from the database.
• the position information of the features of the moving objects of the plant environment is also removed from the database.
• the features of the moving objects of the factory environment are discarded, i. not further used for the calibration of the camera system.
• the calibration unit is adapted to update the features and position information of the database.
• the calibration unit can be designed to estimate, in addition to the parameters of the camera system, also position information of features of the factory environment.
• the parameters of the camera system and the position information of features of the factory environment can be estimated together.
• the parameters of the camera system and the position information of features of the factory environment can be estimated sequentially.
• the calibration unit may be adapted to a feature of the factory environment and its associated
• the calibration unit may be configured to update position information of features stored in the database. This can be done, for example, if the estimated by the calibration unit
• Position information of the feature by less than a predetermined threshold differ from the position information stored in the database feature of the feature.
• the calibration unit may therefore be designed to estimate the position information of the not yet localized features of the factory environment in addition to the parameters of the camera system and the estimated
• the calibration unit may be configured to update the database to determine the accuracy of the position information of the features of the data stored therein
• Receiving unit for sending and receiving the database include.
• the transmit and receive unit allows the database updated by the calibration unit to be sent to others
• the calibration unit is designed to provide information about the first
• the calibration unit may be designed to use information about the first recording positions and the first recording directions or position information of the database in order to estimate the orientation of the camera system relative to the coordinate system of the vehicle or the mounting height of the camera in the vehicle.
• the calibration unit may be configured to determine the orientation of the camera system relative to the coordinate system of the vehicle from unaccelerated straight ahead travel when the database has at least three non-collinear features that span a plane parallel to the ground plane and the at least three are non-collinear Features were associated by the first association unit in multiple images of the first image data. This is also possible if the position information stored in the database of the features of the plant environment are unique only to similarity mappings.
• the database therefore preferably includes, among other things, position information of features on the ground level.
• the calibration unit can be designed to, by means of
• the calibration unit may be configured to determine the pitch and yaw angles of the camera system relative to the coordinate system of the vehicle via the rotation map representing the normalized optical axis of the camera
• the calibration unit can also be designed to the roll angle of the camera system to the coordinate system of the vehicle as the angle between the ground plane and the plane passing through the optical axis and the horizontal axis of the vehicle
• the calibration unit may be designed to
• the distance may be, for example, the distance between two features of the plant environment in the database. In this case, scaling ambiguities of the position information of the database can be resolved. However, it is also possible that the distance is a distance provided by an odometer between two of the first pickup positions. From the features extracted from two or more images of the first image data and associated by the first association unit
• a second aspect of the invention relates to a system for calibrating a camera system for a driver assistance system for a vehicle.
• the system has a device according to the invention for calibrating a camera system and a measuring device for determining or updating the database.
• the measuring device comprises a
• Receiving unit for receiving second image data of
• the second image data comprises a plurality of images of the factory environment, wherein the images of the second image data correspond to a plurality of second recording positions or a plurality of second recording directions and wherein the second image data were recorded with the calibrated reference camera system.
• the surveying device comprises a second
• Extraction unit for extracting features from the second image data
• a second association unit for associating extracted from a plurality of images of the second image data
• Characteristics a reconstruction unit for determining position information of the features associated with the second association unit and an input / output unit for
• the measuring device is designed to, the
• Factory environment to measure i. an initial database containing features of the factory environment and related
• the surveying device can also be designed to update the database. This can be relevant if larger
• the measuring device comprises a calibrated
• Reference camera system and a second receiving unit for receiving second image data of the factory environment, wherein the second image data more with the calibrated Reference camera system recorded images of the factory environment, which were taken from several second shooting positions or multiple second shooting directions.
• the knowledge of the second pick-up positions or the second pick-up directions is not required for the measuring device.
• the second extraction unit of the surveying device differs from the first extraction unit of the device for calibrating the camera system substantially in that it operates on the second image data.
• the second extraction unit thus searches in the second image data for features suitable for the calibration and extracts them.
• the second association unit of the surveying device is designed to search for features of the factory environment extracted from a plurality of images of the second image data.
• the second association unit can use a tracking algorithm that tracks the displacement of features between temporally successive images of a camera.
• the measuring device comprises a
• Position information of the features associated with the second association unit can in particular use the known from the prior art bundle adjustment method according to Hartley and Zisserman.
• the reconstruction unit of the surveying device may be configured to use in advance certain positional information of features of the factory environment or distances between features to avoid ambiguities in determining the
• the input / output unit for storing the features associated with the second association unit and the position information determined by the reconstruction unit in a database is preferably also configured to Database to send to an inventive device for calibration of a camera system.
• the input / output unit of the surveying device can be designed to check the suitability of the database for calibrating the camera system.
• the second receiving unit, the second extraction unit, the second association unit, the reconstruction unit and the input / output unit can be embodied as separate components. Alternatively, these units can be the
• Measuring device or a part of these units by means of a single arithmetic unit to be implemented.
• Association unit of the surveying device designed to detect moving objects of the factory environment and to discard features of the moving objects of the plant environment.
• the second one can also be used
• Association unit of the surveying device to be designed to detect moving objects of the plant environment and to discard features of the moving objects of the plant environment.
• the second association unit does not transfer features of moving objects to the reconstruction unit of the surveying device, since consideration of these
• Characteristics in determining the position information can lead to a deterioration in accuracy.
• Reconstruction unit of the surveying executed to information about the second pickup positions, the second pickup directions or previously known
• Resolve positional information of the features associated with the second association unit can be measured and / or special targets can be installed in the plant environment.
• the input / output unit of the surveying device may be configured to input predetermined in advance
• a third aspect of the invention relates to a method for
• the method comprises receiving first image data, the first image data comprising a plurality of images of a factory environment, wherein the images of the first image data correspond to a plurality of first shot positions or a plurality of first shot directions, and wherein the first image data has been captured with the camera system to be calibrated. Further, features are extracted from the first image data and the extracted features associated with features of a database, the database comprising features of the work environment and associated position information. Using the first image data and the
• Position information of the associated features is then calibrated by the camera system.
• the inventive method both for
• Recording positions and the first recording directions may be unknown.
• the cameras take preferably the pictures at the same times.
• inventive method for calibrating a camera system can also be designed to support asynchronous image recordings.
• extraction of features from the images of the first image data is in particular after
• the inventive method uses a database with features of the plant environment and associated
• Position information is the unique positions of the features relative to a fixed coordinate system. In certain cases, however, it may be sufficient if the positions of the features are known only to similarity mappings. This is the case in particular if only intrinsic parameters of the camera system are to be estimated.
• the features extracted from the first image data are associated with features of the work environment stored in the database to determine positional information of the features extracted from the images of the first image data. In this case, the similarity of features extracted from the images of the first image data with features of the database is determined. If a feature extracted from the first image data is sufficiently similar to a feature stored in the database, the extracted feature may be assigned the position information stored in the database.
• Factory environment are used to calibrate the camera system. Both extrinsic and intrinsic parameters of the camera system can be estimated. Alternatively, only a subset of these parameters can be estimated.
• the method of calibrating a camera system comprises associating features extracted from a plurality of images of the first image data, and adding features extracted and associated with a plurality of images of the first image data to the database.
• a feature has been extracted from multiple images of the first image data. If a feature has been extracted from multiple images of the first image data, it may be inserted into the database with no associated ones
• Position information are stored, since these are not known at first.
• Camera system also position information of features of the plant environment can be estimated. It is thus possible to estimate the initially unknown position information of newly added features in the calibration step and to store them in the database.
• the method for calibrating a camera system comprises detecting movable objects of the factory environment from the first image data
• the inventive method for calibrating a camera system can be designed to detect moving objects of the factory environment and Remove features of the moving objects of the plant environment together with the corresponding position information from the database.
• the method for calibrating a camera system comprises updating the features and position information of the database.
• the calibration step of the method according to the invention for calibrating a camera system can be carried out in addition to the parameters of the camera system as well
• Estimate position information of features of the plant environment can be used to update the database. For example, a feature of the plant environment and its associated position information may be removed from the database if the estimated
• Positional information of the feature deviate by more than a predetermined threshold from the position information of the feature stored in the database.
• position information of features stored in the database may be updated if the last estimated position information of the feature deviates less than a predetermined threshold from the position information of the feature stored in the database.
• the inventive method for calibrating a camera system can also be a transmission and
• Receiving the database include. This allows the updated database to be shared so that it becomes available for other camera systems to be calibrated.
• the method for calibrating a camera system comprises receiving information about the first recording positions or the first
• Information about the first recording positions or the first recording directions can be obtained, for example, by means of odometry and acceleration sensors.
• the information about the first recording positions, the first recording directions or the position information of features of the database can in particular be used to estimate the orientation of the camera system relative to the coordinate system of the vehicle and the mounting height of the camera system in the vehicle.
• the method for calibrating a camera system comprises a method for determining or updating the database, wherein the method for determining or updating the database comprises the following steps: receiving second image data of the database
• the second image data comprises a plurality of images of the factory environment, wherein the images of the second image data correspond to a plurality of second shot positions or a plurality of second shot directions, and wherein the second image data has been captured with a calibrated reference camera system, extracting features from the second image data;
• a calibrated reference camera system is used to determine or update the database.
• the associated features are stored together with position information in a database.
• the method for determining or updating the database further comprises detecting moving objects of the factory environment and the
• Fig. 2 shows the measurement of the factory environment by means of a calibrated reference camera system.
• FIG 3 illustrates the method for calibrating a camera system for a driver assistance system.
• Fig. 4 illustrates the method for measuring the
• FIG. 1 illustrates the dynamic calibration of a camera system 2 for a driver assistance system of a vehicle 1.
• the camera system 2 is permanently installed in the vehicle 1.
• the camera system may be a monocamera system or a multi-camera system.
• Figure 1 for the sake of simplicity, only one
• Monocamera system 2 shown.
• the vehicle 1 moves along a trajectory 8 through the factory environment 9.
• the trajectory 8 may be unknown for the calibration of the camera system 2. While the vehicle 1 through the
• Viewing cone 6 of the camera system 2 features 3-1, 3-2 and 3-3, so that these features can be extracted from a recorded image.
• the factory environment does not have to be for calibration the vehicle have specially attached targets.
• both immobile objects 5 and movable objects 4-1 and 4-2 can be located.
• a database is used which stores features of the plant environment and associated position information. During calibration, both intrinsic and extrinsic parameters of the
• Camera system 2 can be estimated. But it is also possible that only a part of these parameters is estimated. Both
• intrinsic parameters of a camera of the camera system 2 are in particular the focal length, the geometry of the image sensor and the position of the focal point.
• extrinsic parameters of a camera of the camera system 2 describe the position and orientation of the camera relative to a coordinate system 7 of the vehicle 1.
• Camera system is designed to enable a dynamic targetless calibration of the camera system.
• the device according to the invention makes it possible to calibrate a camera system of a vehicle without having to bring the vehicle into a predetermined position or orientation relative to a target.
• FIG. 2 schematically shows the measurement of the factory environment 9 by means of a reference camera system 11.
• the reference camera system 11 is mounted on a stand 10.
• the reference camera system 11 is mounted on a stand 10.
• Reference camera system 11 is carried by hand. With the
• the factory environment 9 includes both immovable objects 5 and moving objects 4-1 and 4-2.
• position information of features of the factory environment is calculated from the images taken with the calibrated reference camera system 11.
• the calculated position information is in a database stored together with the associated features.
• the database can then be used by a device according to the invention for calibrating a camera system. Therefore, the measurement of the factory environment serves for the initial determination of the database required for the device according to the invention for calibrating a camera system.
• FIG. 3 schematically illustrates the method according to the invention for calibrating a camera system for a
• step S 1 a plurality of images of a factory environment of different recording positions and recording directions are recorded with the camera system to be calibrated.
• step S2 features of the with the
• calibrated camera system extracted images Since the factory environment can include both moving and immovable objects, features of moving and immovable objects are extracted from the captured images.
• the extracted features are associated with features of the factory environment stored in a database.
• the database includes not only features of the plant environment but also position information of the characteristics.
• the database may include the locations of features of the factory environment relative to a fixed coordinate system.
• the database it is also possible for the database to contain certain position information of the features of the plant environment only up to similarity maps.
• the database does not include mobile features Objects, as these are usually not suitable for the calibration of the camera system.
• the extracted features for which position information could be determined by association can subsequently be used for calibration S4.
• the image data recorded with the camera system to be calibrated is used for the calibration.
• both intrinsic and extrinsic parameters of the camera system 2 can be estimated.
• step S5 the estimated parameters of the
• FIG. 4 illustrates the method according to the invention for determining or updating the database, which is used by the method according to the invention for calibrating a database
• Camera system is used.
• images of the factory environment of a plurality of recording positions and / or multiple recording directions are recorded in step S10 with a reference camera system.
• the shooting positions and shooting directions need not be known.
• step S features of the factory environment are extracted from the captured images.
• step S12 features extracted from different images are compared with each other to determine which features have been extracted from multiple images. In step S12, therefore, an association takes place from
• step S12 it can also be detected which features belong to moving objects. Characteristics of moving objects can be discarded for the following steps of determining or updating the database, since these features become one Deteriorate the accuracy of the positioning of the characteristics of the factory environment can lead.
• Factory environment stored in a database Furthermore, it can be checked whether the database is suitable for calibrating a camera system, in particular whether the database has sufficiently many features scattered over a wide range of the factory environment.

Landscapes

• Engineering & Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Theoretical Computer Science (AREA)
• Studio Devices (AREA)
• Image Analysis (AREA)
• Length Measuring Devices By Optical Means (AREA)
• Measurement Of Optical Distance (AREA)
• Navigation (AREA)
• Traffic Control Systems (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=62630878

Family Applications (1)

Country Status (5)

Families Citing this family (8)

Citations (5)

Family Cites Families (2)

• 2017
  • 2017-05-05 DE DE102017207614.3A patent/DE102017207614A1/de not_active Withdrawn
• 2018
  • 2018-04-27 JP JP2019548293A patent/JP7332475B2/ja active Active
  • 2018-04-27 KR KR1020197026371A patent/KR102522228B1/ko active Active
  • 2018-04-27 DE DE112018000805.3T patent/DE112018000805A5/de active Pending
  • 2018-04-27 US US16/609,609 patent/US11062478B2/en active Active
  • 2018-04-27 WO PCT/DE2018/200040 patent/WO2018202258A1/de not_active Ceased

Patent Citations (5)

Non-Patent Citations (2)

Also Published As

Similar Documents

Legal Events