KR20150089677A - Camera system, calibration device and calibration method - Google Patents

Abstract

The purpose of the present invention is to provide a camera system, a calibration device and a calibration method, which is capable of performing real-time calibration without spatial and temporal constraints. According to an embodiment of the present invention, the calibration device comprises: an optical flow tracking unit analyzing image data acquired through a camera to extract optical flow information; a distortion detection unit acquiring information on the distortion of the image data based on the optical flow information; a calibration data acquisition unit deriving calibration data of the image data from the distortion information; and a calibration unit calibrating the distortion of the image data based on the calibration data.

Description

[0001] CAMERA SYSTEM, CALIBRATION DEVICE, AND CALIBRATION METHOD [0002]

The present invention relates to a camera system, a calibration apparatus, and a calibration method, and more particularly, to a camera system, a calibration apparatus, and a calibration method applied to an ambient view monitor system.

Recently, there is a growing demand for a car camera for convenience and safe operation of a driver.

For example, an Around View Monitor (AVM) system may be used that takes images of the surroundings of the vehicle with several cameras mounted on the vehicle, synthesizes them, and displays a surrounding image of the driver looking down from the vehicle.

The AVM system combines images acquired through several cameras to provide a single screen. In AVM system, each camera 's position and direction are important information for image synthesis. Therefore, when the AVM system is initially installed in a vehicle, it is necessary to perform a calibration process according to the position of each camera.

In general, the calibration process of the AVM system is performed using a large chart with patterns for calibration, which is time and space constrained. In addition, when the position or direction of the camera is changed due to an impact occurring during vehicle operation, real-time calibration is difficult, which may cause inconvenience to the driver.

SUMMARY OF THE INVENTION It is an object of the present invention to provide a camera system, a calibration apparatus and a calibration method capable of real-time calibration without restriction of time and space.

According to an embodiment of the present invention, a calibration apparatus includes an optical flow tracking unit for analyzing image data obtained through a camera and extracting optical flow information, and a distortion detector for detecting distortion information of the image data based on the optical flow information A correction data obtaining unit for calculating correction data of the image data on the basis of the distortion information, and a correction unit for correcting the distortion of the image data based on the correction data.

According to an embodiment of the present invention, a camera system includes a plurality of cameras disposed at different positions, an image receiving unit for acquiring a plurality of image data using the plurality of cameras, A calibration unit that acquires distortion information of each of the plurality of image data based on the optical flow information and corrects the plurality of image data based on the distortion information, And an image synthesizer for generating the image.

According to an embodiment of the present invention, there is provided a method of calibrating a camera system, comprising the steps of extracting, from image data obtained through a camera, optical flow information including each pixel constituting the image data or a motion vector of each block, Calculating distortion data of the image data based on the flow information, calculating correction data of the image data based on the distortion information, and correcting the distortion of the image data based on the correction data .

According to the embodiment of the present invention, it is possible to acquire distortion information due to movement or rotation of each camera by analyzing image data or synthesized image obtained in real time without using a specific chart, have.

In addition, distortion information can be acquired by analyzing image data obtained through each camera. Even if there is no information on the camera specification, distortion of the image data due to distortion of the camera lens can be corrected, Correction is possible for the cameras. This makes it possible to correct the distortion even if the lens characteristic deviates between the cameras in the manufacturing process.

1 is a schematic view illustrating an AVM system according to an embodiment of the present invention.
2 is a detailed block diagram illustrating a calibration unit according to an embodiment of the present invention.
FIG. 3 illustrates an example of a motion vector detected by optical flow tracking in a calibration unit according to an embodiment of the present invention.
4 is a flowchart illustrating a method of calibrating a camera system according to an exemplary embodiment of the present invention.
5 is a view for explaining a calibration method according to an embodiment of the present invention.
6 is a detailed block diagram illustrating a calibration unit according to another embodiment of the present invention.
7 is a view for explaining a method of acquiring distortion information in a calibration unit according to another embodiment of the present invention.
8 is a flowchart illustrating a method of calibrating a camera system according to another embodiment of the present invention.
9 is a view for explaining a calibration method according to another embodiment of the present invention.

The present invention is capable of various modifications and various embodiments, and specific embodiments are illustrated and described in the drawings. It should be understood, however, that the invention is not intended to be limited to the particular embodiments, but includes all modifications, equivalents, and alternatives falling within the spirit and scope of the invention.

The terms including ordinal, such as second, first, etc., may be used to describe various elements, but the elements are not limited to these terms. The terms are used only for the purpose of distinguishing one component from another. For example, without departing from the scope of the present invention, the second component may be referred to as a first component, and similarly, the first component may also be referred to as a second component. And / or < / RTI > includes any combination of a plurality of related listed items or any of a plurality of related listed items.

In addition, the suffix "module" and " part "for constituent elements used in the following description are given or mixed in consideration of ease of specification, and do not have their own meaning or role.

It is to be understood that when an element is referred to as being "connected" or "connected" to another element, it may be directly connected or connected to the other element, . On the other hand, when an element is referred to as being "directly connected" or "directly connected" to another element, it should be understood that there are no other elements in between.

The terminology used in this application is used only to describe a specific embodiment and is not intended to limit the invention. The singular expressions include plural expressions unless the context clearly dictates otherwise. In the present application, the terms "comprises" or "having" and the like are used to specify that there is a feature, a number, a step, an operation, an element, a component or a combination thereof described in the specification, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, or combinations thereof.

Unless defined otherwise, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. Terms such as those defined in commonly used dictionaries are to be interpreted as having a meaning consistent with the contextual meaning of the related art and are to be interpreted as either ideal or overly formal in the sense of the present application Do not.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings, wherein like or corresponding elements are denoted by the same reference numerals, and redundant description thereof will be omitted.

Hereinafter, a calibration method according to an embodiment of the present invention, a calibration device and a camera system for performing the calibration method will be described in detail with reference to FIG. 1 to FIG.

BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a schematic view showing an AV view system according to an embodiment of the present invention, showing an AVM system mounted on a vehicle. FIG.

Referring to FIG. 1, an AVM system according to an embodiment of the present invention may include a camera system 10 and a display device 20.

The camera system 10 may include a camera unit 11, a camera driving unit 12, an image receiving unit 13, a storage unit 14, an image synthesizing unit 15, a calibration unit 16, and the like. The components shown in FIG. 1 are not essential, and a camera system according to an embodiment of the present invention may include more or fewer components.

The camera unit 11 includes a plurality of cameras (not shown).

Each camera of the camera unit 11 is mounted at a different position of the vehicle and acquires a peripheral image of the vehicle at different positions and directions.

The camera unit 11 includes, but is not limited to, four or more cameras. The camera unit 11 can acquire the forward / backward and left / right images of the vehicle using four or more cameras.

The camera driving unit 12 controls the driving of each camera included in the camera unit 11. For example, it can control the On / Off of each camera.

The image receiving unit 13 performs signal processing such as sampling on the image signal received from each camera of the camera unit 11 to acquire image data. The image data acquired corresponding to each camera can be stored in the storage unit 14 for each camera.

The storage unit 14 may store a program for the operation of the camera system 10 and data input to / output from the camera system 10. [

For example, the storage unit 14 may store a plurality of image data acquired from a plurality of cameras included in the camera unit 11, or a composite image obtained by combining the plurality of image data.

Also, for example, the storage unit 14 may store the coordinate mapping information between the image data acquired from each camera and the composite image generated by combining them, into a database. Here, the coordinate mapping information is information indicating to which pixel of the composite image each pixel of the image data acquired through each camera is mapped.

Further, for example, the storage unit 14 may store correction data for calibration of the camera system 10. [

The image synthesis unit 15 synthesizes a plurality of image data obtained from a plurality of cameras included in the camera unit 11 based on the coordinate mapping information stored in the storage unit 14 to generate one composite image. Here, the composite image may include a top view image corresponding to a 360-degree view of the vehicle on the vehicle.

The image synthesizing unit (15) delivers the composite image obtained by synthesizing a plurality of image data photographed around the vehicle to the display device (20). The display device 20 displays the composite image received from the image composition unit 15 on the screen.

The calibration unit 16 performs calibration of the camera system 10 based on an Obtical Flow trace.

2 is a detailed block diagram illustrating a calibration unit according to an embodiment of the present invention. FIG. 3 illustrates an example of a motion vector detected by optical flow tracking in a calibration unit according to an embodiment of the present invention.

2, the calibration unit 16 according to an exemplary embodiment of the present invention includes an optical flow tracking unit 110, a distortion detection unit 120, a correction data acquisition unit 130, a correction unit 140, can do.

The optical flow tracking unit 110 may extract the optical flow information by analyzing image data obtained through each camera of the camera unit 11. [

Optical flow tracking is a method for detecting motion in an image, and it is a method of tracking motion by searching for a corresponding pixel among consecutive frames based on pixel information of each pixel. Here, the pixel information may include color intensity and texture of each pixel. FIG. 3 illustrates an example of tracking motion vectors in the optical flow tracking unit 110. In FIG. 3, lines drawn in an image represent motion vectors obtained through optical flow tracking.

On the other hand, when optical flow tracking is performed for each pixel, the amount of computation increases and real-time processing may be difficult. Accordingly, the optical flow tracking unit 110 may divide the image data into blocks each consisting of a plurality of pixels to track the optical flow in real time, and may perform optical flow tracking on a block-by-block basis. In addition, the optical flow tracking unit 110 may perform an optical flow tracking process for a plurality of image data obtained through a plurality of cameras in parallel.

The distortion detector 120 may analyze the optical flow information of each camera acquired by the optical flow tracking unit 110 to detect distortion information of each camera.

When the camera moves or the direction of the line of sight of the camera is changed, the direction of the motion vector in the video data of the camera may change rapidly.

Accordingly, the distortion detector 120 may analyze the direction and magnitude of the motion vectors included in the optical flow information to extract a motion vector that is distorted beyond a predetermined level. In addition, the distortion information can be obtained based on the distortion direction and distortion degree of the distorted motion vector. Here, the distortion information is a parameter indicating how much the camera has translated in which direction and how much rotation has been performed with respect to the optical axis.

The correction data obtaining unit 130 may generate correction data for correcting the image data for each camera based on the distortion information of each camera. Here, the correction data is data for correcting the image data obtained from each camera by reflecting the changed position and the viewing direction due to the movement or rotation of each camera, and may include movement information for each pixel or each block for distortion correction, . ≪ / RTI >

The correction data acquisition unit 130 may store the correction data acquired for each camera in the storage unit 14. [

The correction unit 140 can correct the image data acquired through each camera based on the correction data acquired by the correction data acquisition unit 130. [ The correction unit 140 can correct the image data by moving the pixels in the image data based on the correction data.

The image data for each camera, which is corrected by the correction unit 140, is transmitted to the image synthesis unit 15. The image synthesizing unit 15 can generate a synthesized image by synthesizing the distortion-corrected image data based on the coordinate mapping information.

FIG. 4 is a flowchart illustrating a method of calibrating a camera system according to an embodiment of the present invention. FIG. 5 is a diagram illustrating a calibration method according to an embodiment of the present invention.

Referring to FIG. 4, the camera system 10 continuously receives video signals of the respective cameras through the camera unit 11. FIG. Then, the image signal received through each camera is processed to obtain image data for each camera (S101).

The calibration unit 16 performs an optical flow tracking process on the camera-specific image data acquired in step S101. In addition, optical flow information corresponding to each camera is obtained through an optical flow tracking process (S102).

5A, for example, the calibration unit 16 may perform optical flow tracking on image data to obtain a motion vector such as an arrow displayed on a road in an image.

The calibration unit 16 acquires the optical flow information, and obtains the distortion information for each camera from the obtained optical flow information (S103). Then, the calibration unit 16 calculates correction data for distortion correction for each camera based on the distortion information acquired for each camera, and corrects the distortion of the image data acquired through each camera using the correction data (S104).

5 (b) and 5 (c) illustrate a distortion correction process. FIG. 5 illustrates an example of correcting distortion of an area included in a composite image in the image data shown in FIG. 5 FIG. If the composite image is a top view image, the area included in the composite image in FIG. 5A may be a road area. 5 (b) and 5 (c) show the road area of the image data as mesh data in order to express the degree of distortion. FIG. 5 (b) shows data before distortion correction, Represents data after distortion correction.

The image data having been subjected to the distortion correction for each camera is transmitted to the image synthesis unit 15.

When the image data whose distortion has been corrected for each camera is inputted, the image synthesizing unit 15 synthesizes the image data for each camera based on the coordinate mapping information to generate a synthesized image (S105).

In step S105, when the image data whose distortion has been corrected is inputted, the image synthesizing unit 15 may process the synthesized image data into a top view image. 5, the image synthesizing unit 15, when receiving an image whose distortion is corrected as shown in (c) of FIG. 5, The image data can be processed so as to correspond to the view point. The image synthesizer 15 generates a synthesized image using the visual line change.

Hereinafter, a calibration method according to another embodiment of the present invention and a configuration of a calibration device for performing the same will be described in detail with reference to FIG. 6 to FIG.

The AVM system and the camera system for performing the calibration method according to another embodiment of the present invention are the same as the AVM system and the camera system described with reference to FIG. 1 in the above embodiment, Will be omitted.

6 is a detailed block diagram illustrating a calibration unit according to another embodiment of the present invention. 7 is a view for explaining a method of acquiring distortion information in a calibration unit according to another embodiment of the present invention.

6, the calibration unit 16 according to another embodiment of the present invention may include an optical flow tracking unit 210, a distortion detection unit 220, a correction data acquisition unit 230, and the like.

The optical flow tracking unit 210 may analyze the composite image generated by the image combining unit 15 and extract optical flow information for each pixel or each block constituting the composite image.

The distortion detection unit 220 may detect the distortion information of the composite image by analyzing the optical flow information of the composite image acquired by the optical flow tracking unit 110.

When one of the cameras moves or the direction of the line of sight changes, when the image data obtained through the camera is synthesized according to the existing coordinate mapping information, the motion vector of the region corresponding to the corresponding camera in the composite image is The direction of the motion vector is reversed. 7, it can be seen that distortion of the image photographed on the left side of the vehicle has occurred.

Accordingly, the distortion detector 220 can detect a region where a motion vector of a predetermined level or higher is distorted by analyzing the direction and magnitude of the motion vectors included in the optical flow information. In addition, the distortion information can be obtained based on the distortion direction and the degree of distortion of the motion vector in the region where the motion vector is distorted. Here, the distortion information may include position information of a pixel or a block whose motion vector is distorted.

The correction data obtaining unit 230 may generate correction data for correcting the composite image based on the distortion information. Here, the correction data is used for correcting that each pixel or block constituting each image data moves or rotates due to movement or rotation of each camera, thereby causing each pixel or block constituting the composite image to move or rotate to be distorted As the data, motion information, rotation information, and the like for each pixel or block of the synthesized image for distortion correction may be included.

The correction data acquisition unit 230 may store the correction data acquired for the composite image in the storage unit 14. [

The correction data acquired by the correction data acquiring unit 230 is transmitted to the image synthesizing unit 15 and can be used to correct the synthesized image in the image synthesizing unit 15. [ That is, the image synthesizing unit 15 can correct the coordinate mapping information based on the correction data, and generate the composite image based on the corrected coordinate mapping information.

FIG. 8 is a flowchart illustrating a method of calibrating a camera system according to another embodiment of the present invention, and FIG. 9 is a diagram illustrating a calibration method according to another embodiment of the present invention.

Referring to FIG. 8, the camera system 10 continuously receives image signals of the respective cameras through the camera unit 11. FIG. Then, the video signal received through each camera is processed to obtain video data for each camera (S201).

The synthesized image is generated by synthesizing the image data obtained through the cameras through the image synthesizing unit 15 (S202).

When the synthesized image is generated through the image synthesizing unit 15, the calibration unit 16 performs an optical flow tracking process on the synthesized image. Also, optical flow information for the composite image is obtained through an optical flow tracing process (S203).

For example, in FIG. 9A, the calibration unit 16 may perform optical flow tracking for each block constituting a composite image, and obtain a motion vector corresponding to each block.

When the calibration unit 16 acquires the light flow information, distortion information on each pixel or block of the synthesized image is obtained from the calibration information (S204). The calibration unit 16 calculates correction data for distortion correction of each pixel or block constituting the composite image based on the distortion information acquired from the composite image, and corrects the distortion of the combined image using the calculated correction data (S205) .

For example, FIG. 9A and FIG. 9B show an example of correcting the distortion of the synthesized image, which is the top view image, in the distortion correction process. 9 (a) shows a synthesized image before distortion correction, and FIG. 9 (b) shows a synthesized image after distortion correction.

In step S205, the calibration unit 16 may correct the distortion of the composite image by correcting the coordinate mapping information used to generate the composite image by combining the image data obtained through each camera. That is, in order to correct the distortion of the coordinates in the composite image of the image data acquired through the camera due to the movement or rotation of the camera, the coordinate mapping information is corrected, and the composite projection is generated based on the coordinate mapping information. .

According to the embodiments of the present invention described above, it is possible to acquire distortion information due to movement or rotation of each camera by analyzing image data or synthesized image acquired in real time without using a specific chart, It is effective.

In addition, as described in the embodiment of the present invention, distortion information can be obtained by analyzing image data obtained through each camera. Even if there is no information on the camera specification, Distortion can be corrected, and correction is possible for a camera having an arbitrary distortion. This makes it possible to correct the distortion even if the lens characteristic deviates between the cameras in the manufacturing process.

As used in this embodiment, the term " portion " refers to a hardware component such as software or an FPGA (field-programmable gate array) or ASIC, and 'part' performs certain roles. However, 'part' is not meant to be limited to software or hardware. &Quot; to " may be configured to reside on an addressable recording medium and may be configured to play back one or more processors. Thus, by way of example, 'parts' may refer to components such as software components, object-oriented software components, class components and task components, and processes, functions, , Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays, and variables. The functions provided in the components and components may be further combined with a smaller number of components and components or further components and components. In addition, the components and components may be implemented to play back one or more CPUs in a device or a secure multimedia card.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the present invention as defined by the following claims It can be understood that

Claims (9)

An optical flow tracking unit for analyzing image data obtained through a camera and extracting optical flow information,
A distortion detector for obtaining distortion information of the image data based on the optical flow information,
A correction data obtaining unit for calculating correction data of the image data based on the distortion information, and
And a correction unit for correcting distortion of the image data based on the correction data,
≪ / RTI >
The method according to claim 1,
Wherein the optical flow information includes each pixel constituting the image data or a motion vector for each block.
3. The method of claim 2,
Wherein the distortion detector extracts a motion vector distorted by a predetermined level or more from the optical flow information and obtains the distortion information based on distortion direction and distortion information of the distorted motion vector.
3. The method of claim 2,
Wherein the correction data includes movement information and rotation information for distortion compensation for each pixel or each block.
A plurality of cameras disposed at different positions,
An image receiving unit for acquiring a plurality of image data using the plurality of cameras,
The image processing apparatus includes a calibration section for extracting optical flow information by analyzing each of the plurality of image data, acquiring distortion information of each of the plurality of image data based on the optical flow information, and correcting the plurality of image data based on the distortion information. , And
An image synthesis unit for synthesizing the plurality of image data and generating a synthesized image,
.
6. The method of claim 5,
Wherein the optical flow information includes each pixel constituting each of the plurality of image data or a motion vector of each block.
The method according to claim 6,
Wherein the calibration unit extracts a motion vector that is distorted more than a predetermined level from the optical flow information, and obtains the distortion information based on distortion direction and distortion information of the distorted motion vector.
The method according to claim 6,
Wherein the calibration unit calculates correction data including movement information and rotation information for each pixel or each block on the basis of the distortion information, and corrects the plurality of image data using the correction data.
Extracting light flow information including each pixel constituting the image data or a motion vector of each block from image data obtained through a camera,
Obtaining distortion information of the image data based on the optical flow information,
Calculating correction data of the image data based on the distortion information, and
Correcting the distortion of the image data based on the correction data
Gt; a < / RTI > camera system.

Images