KR102012568B1 - Multi sensor calibration system and calibration method for multi sensor - Google Patents

Abstract

The present invention relates to a multi-sensor calibration system and a multi-sensor calibration method, comprising: a color pattern plate having a grid pattern formed by alternating a black square of a predetermined size and a white square of a predetermined size on one surface; A hot wire plate having a checkered pattern formed thereon and a hot wire installed at an edge of the checkered pattern; A photographing sensor unit for photographing the color pattern plate and the hot wire plate; A depth pattern plate having a preset depth and having a square pattern formed of at least one quadrangle having a predetermined size; And a depth sensor unit for photographing the depth pattern plate as an image. And generating calibration image information by correcting the image information photographed by the photographing sensor unit, generating depth correction information by correcting the image information photographed by the depth sensor unit, and integrating the depth correction information and the correction image information. And a correction unit for generating final image information.

Description

Multi sensor calibration system and calibration method for multi sensor

The present invention relates to a multi-sensor calibration system and a multi-sensor calibration method, and more specifically, each plate formed with a pattern having a heating wire, a color pattern, and a depth with various sensors to integrate the photographed image. It relates to a multi-sensor calibration system and a multi-sensor calibration method implemented to calibrate.

In general, various sensors are used to detect the surrounding environment in robots or autonomous vehicles. Among them, lidar, radar, cameras, etc. are used to detect the environment around robots or autonomous vehicles. At this time, rather than using a single sensor, the sensor information using multiple multiple sensors are fused and utilized. In order to fuse different sensor information, it is necessary to know the relationship between each sensor. The relationship between the sensors here is the rotation and translation of the sensor teeth. In order to know the relationship between the sensors, the sensors must be calibrated. In order to increase the accuracy of information convergence, it is necessary to minimize calibration errors.

In order to perform calibration in a system equipped with a general multiple sensor, a combination of two sensors capable of calibration is first performed, followed by calibration of other combinations. In order to calibrate one after another, the first calibration error affects the second calibration, which in turn affects the next calibration, leading to a greater error between the first sensor and the last sensor relationship. Has the problem.

Republic of Korea Patent Registration No. 10-1473736 Republic of Korea Patent Publication No. 10-2013-0098040

Therefore, the present invention has been made to solve the above problems, the problem to be solved of the present invention is to shoot each image plate formed with a pattern having a heating wire, color pattern and depth with various sensors, It is to provide a multisensor calibration system and multisensor calibration method implemented to integrate and calibrate.

However, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned above are clearly apparent to those skilled in the art from the following description. It can be understood.

As a means for achieving the above technical problem, a color pattern plate having a grid pattern formed by alternating a black square of a certain size and a white square of a predetermined size are formed on one surface; A hot wire plate having a checkered pattern formed thereon and a hot wire installed at an edge of the checkered pattern; A photographing sensor unit for photographing the color pattern plate and the hot wire plate; A depth pattern plate having a square pattern formed of at least one quadrangle having a predetermined depth and having a predetermined depth; And a depth sensor unit for photographing the depth pattern plate as an image. And calibrated image information taken by the photographing sensor unit to generate calibrated image information, calibrate the image information taken by the depth sensor unit to generate depth calibration information, and integrate depth calibration information and calibrated image information to obtain final image information. Provides a multi-sensor calibration system comprising a; calibration unit for generating a.

Also, in one embodiment, the photographing sensor unit photographs the color pattern plate to generate color image information, and photographs the hot wire plate to generate thermal image information.

Further, in one embodiment, the depth sensor unit is characterized in that to produce the depth image information by photographing the depth pattern plate.

In addition, in one embodiment, the correction unit is characterized in that for generating the corrected image information by correcting the distortion portion of the color pattern plate and the hot wire plate in the image information photographed by the image sensor.

Another means for achieving the above technical problem, the first step of photographing the color pattern plate and the hot wire plate in the image sensor unit; A second step of correcting the image information generated by the photographing sensor unit to generate the corrected image information; A third step of photographing the depth pattern plate as an image by the depth sensor unit; And a fourth step of generating a depth correction information by correcting the image information photographed by the photographing sensor unit, and generating final image information by integrating the depth correction information and the corrected image information. To provide.

In an embodiment, the second step may include image information generated by the photographing sensor unit being thermal image information or color image information.

According to an embodiment of the present invention, by providing a multi-sensor calibration system and a multi-sensor calibration method, the multi-sensor can be calibrated simultaneously.

However, effects obtained in the present invention are not limited to the above-mentioned effects, and other effects not mentioned above will be clearly understood by those skilled in the art from the following description. Could be.

The following drawings, which are attached in this specification, illustrate preferred embodiments of the present invention, and together with the detailed description of the present invention, serve to further understand the technical spirit of the present invention. It should not be construed as limited to.
1 is a block diagram of a multi-sensor calibration system according to a first embodiment of the present invention.
FIG. 2 is a front view of the color pattern plate shown in FIG. 1.
3 is a front view of the hot wire plate in FIG. 1.
4 is a front view of the depth pattern plate in FIG. 1.
5 is a cross-sectional view taken along line AA ′ of the depth pattern plate of FIG. 1.
6 is an actual picture of the hot wire plate in FIG.
7 is a flowchart showing a multi-sensor calibration method of the present invention as a first embodiment.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art may easily implement the present invention. However, since the description of the present invention is only an embodiment for structural or functional description, the scope of the present invention should not be construed as limited by the embodiments described in the text. That is, since the embodiments may be variously modified and may have various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical idea. In addition, the objects or effects presented in the present invention does not mean that a specific embodiment should include all or only such effects, the scope of the present invention should not be understood as being limited thereby.

The meaning of the terms described in the present invention will be understood as follows.

Terms such as "first" and "second" are intended to distinguish one component from another component, and the scope of rights should not be limited by these terms. For example, the first component may be named a second component, and similarly, the second component may also be named a first component. When a component is referred to as being "connected" to another component, it should be understood that there may be other components in between, although it may be directly connected to the other component. On the other hand, when a component is said to be "directly connected" to another component, it should be understood that there is no other component in between. On the other hand, other expressions describing the relationship between the components, such as "between" and "immediately between" or "neighboring to" and "directly neighboring to", should be interpreted as well.

Singular expressions should be understood to include plural expressions unless the context clearly indicates otherwise, and terms such as "include" or "have" refer to features, numbers, steps, operations, components, parts, or parts thereof described. It is to be understood that the combination is intended to be present and does not exclude in advance the possibility of the presence or addition of one or more other features or numbers, steps, operations, components, parts or combinations thereof.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. Generally, the terms defined in the dictionary used are to be interpreted as being consistent with the meanings in the context of the related art, and should not be interpreted as having ideal or excessively formal meanings unless clearly defined in the present invention.

<Configuration according to the first embodiment>

1 is a block diagram of a multi-sensor calibration system according to a first embodiment of the present invention, FIG. 2 is a front view of the color pattern plate of FIG. 1, and FIG. 3 is a front view of the hot wire plate of FIG. 4 is a front view of the depth pattern plate in FIG. 1, FIG. 5 is a sectional view taken along line AA ′ of the depth pattern plate in FIG. 1, and FIG. 6 is an actual photograph of the hot wire plate in FIG.

1 to 2, The multi-sensor calibration system 10 includes a color pattern plate 100, a heating wire plate 200, a photographing sensor unit 300, a depth pattern plate 400, a depth sensor unit 500, and a calibration unit 600. Include.

The color pattern plate 100 has a grid pattern formed by alternating black squares 11 and white squares 12 having a predetermined size on one surface thereof (see FIG. 2).

The black rectangle 11 and the white rectangle 12 may be formed in a rectangle (preferably square). The black rectangle 11 and the white rectangle 12 may be formed in the same size and shape.

The heating wire plate 200 is formed with a checkered pattern on one surface, the heating wire 13 is installed in the corner portion of the checkered pattern (see Fig. 3). The checkerboard pattern may be formed in the same size and shape as the grid pattern of the color pattern plate 100.

The photographing sensor unit 300 photographs the color pattern plate 100 and the hot wire plate 200 as an image. The photographing sensor unit 300 may generate color image information by photographing the color pattern plate 100, and generate thermal image information by photographing the hot wire plate 200. The photographing sensor unit 300 may transmit the color image information and the thermal image information to the correction unit 600. The photographing sensor unit 300 may be a camera (for example, a color camera) for photographing the color pattern plate 100 or a camera (for example, a thermal imaging camera, a night vision camera, a 3D Lidar) for sensing and photographing a heat ray. Etc.), and may be formed of multiple types of multi-sensors.

Thermal imaging cameras are devices that track and detect heat and show it on the screen at a glance. The thermal camera displays the screen according to how much heat it emits, so that objects can be identified regardless of the presence or absence of obstacles such as smoke or light.

Night vision cameras use the infrared region to secure the field of view and detect objects in front of the field of view. These night vision cameras can be divided into near-infrared and far-infrared. Here, the near-infrared method is a method of detecting the near-infrared (wavelength: 800 ~ 1,000nm) reflected by the infrared radiation emitted through the IR generator after reaching the object through the camera. In addition, the far infrared method refers to a method of detecting heat emitted by an object (wavelength: 1,000 nm or more) using a far infrared camera.

3D Lidar is a sensor that detects the surroundings of objects regardless of their movement.

The depth pattern plate 400 is formed with a square pattern 14 formed of at least one rectangle having a predetermined depth (see FIG. 4). In the rectangular pattern 14, the rectangle may be formed to have the same size as the black rectangle 11 or the white rectangle 12. In addition, the square pattern 14 may be installed at positions spaced apart by the length of the white rectangle 12 in the horizontal direction, and may be installed at positions spaced apart by the length of the white rectangle 12 in the vertical direction.

The depth sensor unit 500 photographs the depth pattern plate 400 as an image. The depth sensor unit 500 may photograph the depth pattern plate 400 to generate depth image information. Here, the depth image information may store depth information of the photographed depth pattern plate 400. The depth sensor unit 500 may transmit the depth image information to the correction unit 600. The depth sensor unit 500 may be formed of various types of depth sensors, and may be used as an active depth sensor method and a passive depth sensor method.

The active depth sensor method is a method of directly acquiring three-dimensional depth information of an object using a laser sensor, an infrared sensor, and a pattern sensor. In this case, a low resolution depth image may be provided and a high cost may be obtained, but accurate depth information may be obtained.

The passive depth sensor method is a method of obtaining 3D depth information indirectly from a multiview and a stereo image. In this case, it takes a long time to acquire the depth information, and has a disadvantage of providing inaccurate depth information in a relatively occlusion area or a texture-free area, but provides a high resolution depth image and low cost for hardware configuration. There is an advantage.

The calibration unit 600 generates calibration image information by calibrating the image information photographed by the photographing sensor unit 300, and generates depth calibration information by calibrating the image information photographed by the depth sensor unit 500, and depth calibration. The final image information is generated by integrating the information with the calibration image information. The calibrator 600 may generate calibrated image information by calibrating a distortion portion of the color pattern plate 100 and the hot wire plate 200 from the image information photographed by the imaging sensor unit 300. The corrector 600 receives color image information from the photographing sensor unit 300 to correct a distorted portion (for example, a peripheral portion of the color pattern plate 100) of the color pattern plate 100 photographed. Can be. In other words, the correction unit 600 may correct the distortion portion in the image coordinate system of the color pattern plate 100 in the color image information. The calibrator 600 may receive a thermal image information from the photographing sensor unit 300 to correct a distorted portion (for example, a peripheral portion of the hot wire plate 200) of the shape of the hot wire plate 200. . In other words, the correction unit 600 may correct the distortion portion in the image coordinate system of the hot wire plate 200 in the thermal image information. The calibrator 600 may generate the calibrated image information by comparing the color image information with the thermal image information and correcting the distorted portion. The calibration unit 600 receives the depth image information from the depth sensor unit 500 and checks the depth of the photographed depth pattern plate 400, and the depth of the photographed depth pattern plate 400 and the actual depth pattern plate 400. ) Can be corrected by comparing the depth of For example, when the depth of the depth pattern plate 400 stored in the depth image information is 5 mm and the depth of the actual depth pattern plate 400 is 10 mm, the correction unit 600 may correct the difference of 5 mm. . The calibration unit 600 may store in advance the actual pattern information of the color pattern plate 100, the actual pattern information of the hot wire plate 200, and the actual depth information of the depth pattern plate 400.

In the multiple sensor calibration system 10 having the configuration as described above, the photographing sensor unit 300 and the depth sensor unit 500 include a color pattern plate 100, a hot wire plate 200, and a depth pattern plate 400. By photographing at a time, a plurality of multi-sensors (eg, thermal imaging cameras, night vision cameras, 3D Lidar, etc.) can be simultaneously calibrated.

<Method according to the first embodiment>

7 is a flowchart illustrating a multi-sensor calibration method according to an embodiment of the present invention.

Referring to FIG. 7, the multi-sensor calibration method includes a first step S100, a second step S200, a third step S300, and a fourth step S400.

First, the photographing sensor unit 300 photographs the color pattern plate 100 and the hot wire plate 200 as an image (S100).

After the above-described step S100, the photographing sensor unit 300 corrects the photographed image information to generate the corrected image information (S200).

After the above-described step S200, the depth pattern plate 400 is photographed by the depth sensor unit 500 (S300).

After the above-described step S300, the calibration image information and the image information photographed by the depth sensor unit 500 is corrected (S400).

<Configuration according to the second embodiment>

The multi-sensor calibration system includes a color pattern and a heating wire plate (not shown in the drawings for convenience of description), the photographing sensor unit 300, the depth pattern plate 400, the depth sensor unit 500, and the correction unit 600. Include. Here, the photographing sensor unit 300, the depth pattern plate 400, the depth sensor unit 500, and the correction unit 600 will not be described below for parts similar to those of FIG. 1, and only for other parts. Explain.

The color pattern and the hot wire plate have a grid pattern formed by alternating black squares of a certain size and white squares of a constant size on one surface thereof, and a hot wire is installed at an edge of the grid pattern.

The photographing sensor unit 300 photographs one surface of the color pattern and the heating wire plate as an image.

The detailed description of the preferred embodiments of the invention disclosed as described above is provided to enable those skilled in the art to implement and practice the invention. Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will understand that various modifications and changes can be made without departing from the scope of the present invention. For example, those skilled in the art can use each of the configurations described in the above-described embodiments in combination with each other. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The invention can be embodied in other specific forms without departing from the spirit and essential features of the invention. Accordingly, the above detailed description should not be interpreted as limiting in all aspects and should be considered as illustrative. The scope of the invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the invention are included in the scope of the invention. The present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In addition, the claims may be combined to form an embodiment by combining claims that do not have an explicit citation relationship or may be incorporated as new claims by post-application correction.

10: multi-sensor calibration system
11: black square
12: white square
13: heating wire
14: square pattern
100: color pattern plate
200: heated wire plate
300: photographing sensor unit
400: depth pattern plate
500: depth sensor
600: correction part

Claims (6)

A color pattern plate having a grid pattern formed by alternating black squares of a predetermined size and white squares of a predetermined size formed on one surface thereof;
A hot wire plate having a checkered pattern formed thereon and a hot wire installed at an edge of the checkered pattern;
A photographing sensor unit for photographing the color pattern plate and the hot wire plate;
A plurality of square patterns are formed in a rectangular shape having a predetermined depth having a predetermined depth, the square pattern is installed in a position spaced apart by a horizontal length in the horizontal direction, depth patterns are respectively installed in a position spaced by a vertical length in the vertical direction plate;
A depth sensor unit for photographing the depth pattern plate as an image; And
By generating the corrected image information by correcting the image information photographed by the photographing sensor unit, by generating the depth correction information by correcting the image information photographed by the depth sensor unit by integrating the depth correction information and the corrected image information Includes; correction unit for generating the final image information,
The photographing sensor unit photographs the color pattern plate to generate color image information, and photographs the hot wire plate to generate thermal image information.
The depth sensor unit photographs the depth pattern plate to generate depth image information,
The corrector may generate corrected image information by correcting distortions of the color pattern plate and the hot wire plate from the image information photographed by the photographing sensor unit.
The calibration unit may simultaneously calibrate the image information photographed by the photographing sensor unit, the image information photographed by the depth sensor unit, and the image information photographed by the hot wire plate,
The depth sensor unit, a multi-sensor calibration system, characterized in that using any one of a laser sensor, an infrared sensor and a pattern sensor to detect the surrounding environment. delete delete delete As a calibration method of a multi-sensor calibration system according to claim 1,
A first step of photographing the color pattern plate and the hot wire plate by an image sensor;
A second step of generating corrected image information by correcting the image information generated by the photographing sensor unit;
A third step of photographing the depth pattern plate as an image by the depth sensor unit; And
And a fourth step of calibrating the image information photographed by the photographing sensor unit to generate depth correction information, and generating final image information by integrating the depth correction information and the corrected image information.
In the second step, the image information generated by the photographing sensor unit is thermal image information or color image information.
The calibration unit may simultaneously calibrate the image information photographed by the photographing sensor unit, the image information photographed by the depth sensor unit, and the image information photographed by the hot wire plate,
The depth sensor unit, a multi-sensor calibration method, characterized in that using any one of a laser sensor, an infrared sensor, and a pattern sensor to detect the surrounding environment. delete

Images