JP2022130981A - Method, device, and system for calibration - Google Patents

Abstract

To provide a method, a device, and a system for calibration that can increase the freedom of a location.SOLUTION: The method for calibration includes the steps of: creating corrected drawing data corrected according to a projection surface by a calibration device; outputting the corrected drawing data and projecting a projection drawing to the projection surface by a projection device; imaging the projection drawing by the imaging device; and performing a calibration in a monitoring system on the basis of the taken projection drawing by the calibration device.SELECTED DRAWING: Figure 2

Description

The present invention relates to a calibration method, a calibration device, and a calibration system in a monitoring system.

2. Description of the Related Art Conventionally, a monitoring system has been known in which fisheye cameras are arranged on the front, rear, left, and right of a vehicle, and stitching of the obtained images creates an image as if the vehicle were viewed from above. The created video is displayed on a vehicle monitor, for example, and utilized for driving support and the like. Patent documents 1 and 2 disclose stitching methods.

By the way, in order to perform stitching in such a monitoring system, it is necessary to first use a marker to adjust (calibrate) the combining position of the images. Calibration is performed, for example, by the following method.

First, as markers, dedicated marker sheets on which patterns are drawn are arranged at four locations around the vehicle in the oblique direction. As a result, these marker sheets are reflected at both ends of the image of the fisheye camera. In other words, the same marker sheet image is displayed on each of the adjacent fisheye cameras. In the calibration, the position of compositing the images captured by the fisheye camera is adjusted so that these images overlap. Regarding the calibration method using these markers, Patent Document 3 discloses a data processing system for calibration between two or more cameras fixed on a vehicle. Further, Patent Document 4 discloses a method of adjusting the position of the marker and parameters used for calibration. Furthermore, Non-Patent Literature 1 discloses Flying View (registered trademark) that stitches fish-eye camera images in a bowl shape and displays images from any position in the sky as a viewpoint.

JP 2017-512334 A JP 2012-227913 A JP 2017-506348 A JP 2012-239157 A

Hidetaka Komai, "Flying View", [online], 2018, Oki Electric Industry Co., Ltd., [searched February 8, 2021], Internet, <URL: https://www.oki.com/jp/iot/ doc/2018/img/ceatec/ceatec_flyingview.pdf>

Each camera needs to accurately recognize the position of the marker, the pattern, and the like. Here, for example, if the place where the marker is placed is not flat and has unevenness, the marker sheet will be deformed and the camera will not be able to recognize the marker accurately. Therefore, when performing calibration, there are restrictions such as avoiding places with unevenness.

SUMMARY OF THE INVENTION It is an object of the present invention to improve the degree of freedom of location in a calibration method, a calibration apparatus, and a calibration system.

A calibration method according to the present invention includes the steps of measuring the shape of a projection plane, the steps of creating corrected figure data corrected in accordance with the shape of the projection plane, and projecting the projected figure based on the corrected figure data onto the projection plane. photographing the projected figure from a plurality of viewpoints; and calibrating the combining position of the images based on the projected figure photographed from the plurality of viewpoints.

A calibration apparatus according to the present invention includes a processing unit that measures the shape of a projection plane and creates correction figure data that is corrected in accordance with the shape of the projection plane, and a projection apparatus that projects a projection figure based on the correction figure data. A projection unit for projecting onto a surface, and an adjustment unit for calibrating a combining position of images based on projection figures photographed from a plurality of viewpoints are provided.

A calibration system according to the present invention includes a projection device that measures the shape of a projection plane and projects a projected figure on the projection plane based on corrected figure data corrected in accordance with the shape of the projection plane, and a projected figure from a plurality of viewpoints. and a calibration device that creates corrected figure data and calibrates the combining position of the images based on the projection figures photographed from a plurality of viewpoints.

According to the calibration method, the calibration device, and the calibration system according to the present invention, calibration is performed based on the projected figure corrected in accordance with the projection plane. Therefore, the projection plane can be examined without considering that the marker will be placed. Therefore, the degree of freedom of location can be improved in the calibration method, calibration device, and calibration system.

1 is a schematic configuration diagram of a calibration system 1 according to Embodiment 1; FIG. 3 is a functional block diagram of the calibration device 4 according to Embodiment 1. FIG. 4A and 4B are diagrams for explaining the operation of the measuring device 21 according to the first embodiment; FIG. FIG. 4 is a diagram for explaining a method of creating 3D information according to Embodiment 1; FIG. FIG. 4 is a diagram for explaining a method of correcting a plane according to Embodiment 1; FIG. FIG. 5 is a diagram showing an example of correction according to Embodiment 1; FIG. 4 is a diagram for explaining a segment dividing method according to Embodiment 1; FIG. FIG. 4 is a diagram for explaining a height difference correction method according to Embodiment 1; FIG. 4A and 4B are diagrams for explaining a projection method according to Embodiment 1; FIG. 4 is a flowchart showing a calibration method according to Embodiment 1; It is a figure for demonstrating the calibration method based on a comparative example. FIG. 10 is a diagram for explaining deformation of a marker sheet according to a comparative example; 2 is a schematic configuration diagram of a calibration system 1A according to Modification 1 of Embodiment 1. FIG.

Embodiment 1.
FIG. 1 is a schematic configuration diagram of a calibration system 1 according to Embodiment 1. As shown in FIG. The calibration system 1 performs calibration as means for adjusting stitching in a monitoring system used for monitoring objects such as vehicles. As shown in FIG. 1, the calibration system 1 includes projection devices 2a-2d, imaging devices 3a-3d, and a calibration device 4. As shown in FIG. First, the basic configurations of the projection devices 2a to 2d, the imaging devices 3a to 3d, and the calibration device 4 will be described.

The projection devices 2a to 2d are attached to the roof of the vehicle body and project a projection figure P obliquely onto a projection plane S. As shown in FIG. The projection device 2a is provided diagonally forward to the left with respect to the orientation of the vehicle. Similarly, the projection device 2b is provided obliquely forward right. The projection device 2c is provided diagonally to the rear left. The projection device 2d is provided obliquely rearward to the right. The pattern of the projection figure P is not particularly limited, but the same pattern as that used in the conventional marker is projected. The projection plane S is an area within a predetermined range determined from the installation angles and heights of the projection devices 2a to 2d in the place G such as the ground or floor where the vehicle or the like is parked.

The imaging devices 3a to 3d are, for example, fisheye cameras. The photographing device 3a is provided in the front part of the vehicle body. Similarly, the photographing device 3b is provided on the left side of the vehicle body. The photographing device 3c is provided on the right side of the vehicle body. The photographing device 3d is provided at the rear portion of the vehicle body. A projection figure P is reflected at both ends of the image shot by each of the imaging devices 3a to 3d, and two adjacent ones of the imaging devices 3a to 3d shoot the same projection figure P, respectively. As a result, images captured by the image capturing devices 3a to 3d when conventional markers are arranged at four locations around the vehicle in the oblique direction are simulated. Note that the adjacency of the photographing devices 3a to 3d indicates an arrangement relationship other than the combination of the front, rear, left, and right, such as the front photographing device 3a and the left photographing device 3b.

The calibration device 4 creates corrected graphic data, which is data of the projected graphic P. FIG. The corrected graphic data is obtained by correcting the original graphic data according to the shape of the projection plane S. FIG. In other words, the projected figure P is the figure projected onto the projection plane S by the projector 2 after the corrected figure data is output. Further, the calibration device 4 performs calibration so that the projection figures P photographed by the photographing device 3 overlap each other. The calibration device 4 is configured by, for example, a CPU (Central Processing Unit), a terminal device such as a PC, or dedicated hardware. The calibration device 4 and the projection device 2 are communicably connected by wire or wirelessly. The calibration device 4 and the imaging device 3 are communicably connected by wire or wirelessly. A detailed method of creating the correction figure data will be described later.

FIG. 2 is a functional block diagram of the calibration system 1 according to the first embodiment. Here, as the projection device 2, the projection device 2a representing the projection devices 2a to 2d will be described. The other projection devices 2b to 2d also have the same configuration as the projection device 2a. Also, as the photographing device 3, the photographing device 3a representing the photographing devices 3a to 3d will be described. Other photographing devices 3b to 3d also have the same configuration as the photographing device 3a.

The projection device 2 has a measurement device 21 and a projection lamp 22 . The measuring device 21 is a laser rangefinder that irradiates a projection plane with laser light and measures the distance between the projection device 2 and the projection plane. The measurement device 21 transmits the measurement result to the measurement section 42 of the calibration device 4 . The projection lamp 22 outputs the corrected figure data created by the calibration device 4 and projects the projection figure.

A photographing device 3 photographs a projected figure. Further, the photographing device 3 transmits the photographed image data of the projection figure to the adjusting section 45 of the calibration device 4 .

The calibration device 4 has a storage unit 41 and a measurement unit 42, a processing unit 43, a projection unit 44, and an adjustment unit 45 as functional units. The storage unit 41 is, for example, a non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, EEPROM. Each functional unit is implemented by software, firmware, or a combination of software and firmware. Software and firmware are written as programs and stored in the storage unit 41 .

Graphic data is stored in the storage unit 41 . The figure data is stored as data of the wavelength of light expressed as a figure, and is the original shape data of the corrected figure data. The graphic data is corrected as corrected graphic data by the calibration device 4 and output by the projection lamp 22 .

The measurement unit 42 controls the measurement device 21 to acquire the distance from the measurement device 21 to the projection plane. FIG. 3 is a diagram for explaining the operation of the measuring device 21 according to the first embodiment. As shown in FIG. 3, the measurement unit 42 directs the measurement device 21 toward the projection plane S and scans the projection plane S at predetermined time intervals. Thereby, the measurement unit 42 acquires the distance from the measurement device 21 for each of the plurality of measurement points on the projection plane S. In FIG. 3, the measurement point E and the distance L from the measurement device 21 to the measurement point E are shown as an example.

The processing unit 43 creates correction graphic data. Specifically, a plurality of processes described below are performed. FIG. 4 is a diagram for explaining a method of creating 3D information according to Embodiment 1. FIG. In FIG. 4, a cross section in the vertical direction of the projection plane S is schematically shown for simplification of explanation. In addition, in FIG. 4, a vertical section of the estimated plane F is indicated by a dashed line. First, the processing unit 43 estimates the plane F as shown in FIG. The plane F is a horizontal plane that is used as a reference when creating 3D information of the projection plane S. For example, the installation angle, height, or measurement result of the measuring device 21, or various sensors (not shown) that detect horizontality. estimated from the output results of Then, the 3D information D of the actual projection plane S is created by comparing the distances assumed for the plane F and the actually measured distances at a plurality of measurement points on the projection plane S. This 3D information D includes, for example, information about the slope formed on the projection plane S or the dimensions of the uneven shape. The information about the tilt is, for example, the angle of tilt. The dimension of the uneven shape is, for example, the height difference with respect to the plane F, or the like.

Then, the processing unit 43 associates the image range of the graphic data with each measurement point of the 3D information. That is, which image range in the figure data is output to which region on the projection plane S is determined.

Further, the processing unit 43 corrects the graphic data stored in the storage unit 41 based on the 3D information of the projection plane S to create corrected graphic data. In correcting the graphic data, for example, a known technique such as a trapezoidal correction function of a projector can be applied. Correction of figure data includes plane correction and height difference correction.

First, the processing unit 43 corrects the plane of the figure data. Planar correction is correction based on the distance from the reference point on the plane. FIG. 5 is a diagram for explaining a plane correction method according to the first embodiment. As shown in FIG. 5, the processing unit 43 sets the measurement position closest to the measuring device on the estimated plane F as the reference point R. As shown in FIG. Then, in the 3D information D of the projection plane S, the processing unit 43 compresses the graphic data so that the size of the image range of the corresponding graphic data becomes smaller as the distance from the reference point R increases. In the 3D information D of the projection plane S, the processing unit 43 enlarges the graphic data so that the size of the image range of the corresponding graphic data increases as the distance from the reference point R decreases.

FIG. 6 is a diagram showing a correction example according to the first embodiment. 6A shows a projected figure P, and FIG. 6B shows corrected figure data corresponding to the projected figure P. FIG. For example, to project a circle as shown in FIG. 6(a), an ellipse as shown in FIG. 6(b) is generated and the scale is adjusted. In this way, on the estimated plane, the scale of the figure data is changed based on the distances between the plurality of measurement points on the projection plane and the reference point, and the plane of the figure data is corrected. Degradation of the projection figure P can be suppressed even at a position far from the reference position in .

Next, the processing unit 43 corrects the height difference. Correction of the height difference is correction performed on the inclination with respect to the plane F or the uneven shape of the projection plane S. FIG. FIG. 7 is a diagram for explaining a segment dividing method according to the first embodiment. As shown in FIG. 7, the processing unit 43 divides the 3D information D into segments a to g each time the tilt angle changes.

FIG. 8 is a diagram for explaining the height difference correction method according to the first embodiment. As shown in FIG. 8, in the inclined segment T that is inclined with respect to the plane F, the height differs for each measurement point. That is, the measurement point on the inclined segment T has a height difference with respect to the plane. In the inclined segment T of the 3D information D on the projection plane S, the graphic data is enlarged so that the size of the image range of the corresponding graphic data increases as the distance from the projection device 2 decreases. Also, in the inclined segment T of the 3D information D on the projection plane S, the graphic data is compressed so that the size of the image range of the corresponding graphic data becomes smaller as the distance from the projection device 2 increases. In this way, based on the distances between a plurality of measurement points having height differences with respect to the estimated plane and the projection device 2, the scale of the figure data is changed and the height difference is corrected, whereby the projection plane Degradation of the projection figure P can be suppressed even when S has an inclination or an uneven shape.

The projection unit 44 transmits the created correction graphic data to the projection device 2 . As a result, the projection lamp 22 outputs the corrected figure data, and the projection figure P is projected onto the projection plane S. FIG. FIG. 9 is a diagram for explaining a projection method according to Embodiment 1. FIG. As shown in FIG. 9, when viewed from the projection device 2, the projection figure P is projected as a figure in which the end points of each segment are continuous. As a result, the photographing device 3 can photograph an image similar to the image photographed when the conventional marker is arranged in the oblique direction of the vehicle.

The adjuster 45 performs calibration. In the calibration, the image combining position is adjusted so that the projection figures P photographed by the adjacent photographing devices 3a to 3d are overlapped. The calibration result is stored in the storage unit 41 or a device external to the calibration system 1 and used for stitching.

10 is a flowchart showing a calibration method according to Embodiment 1. FIG. A calibration method according to the first embodiment will be described with reference to FIG. First, the measurement unit 42 measures the distance between the projection plane and the measurement device 21 using the measurement device 21 (S1). Next, the processing unit 43 creates 3D information of the projection plane based on the measurement result of the projection plane (S2), and associates the image range of the graphic data with each measurement position of the 3D information (S3).

Subsequently, the processing unit 43 determines a reference point from the 3D information (S4), and corrects the plane of the graphic data based on this reference point (S5). Then, the processing unit 43 divides the 3D information into segments (S6), and corrects the height difference for the segments inclined with respect to the plane based on the distance from the measuring device 21 (S7). Through the processing of S4 to S7, the graphic data is corrected to create corrected graphic data.

Subsequently, the projection unit 44 outputs the correction figure data to the projection lamp 22 to project the projection figure (S8). Then, the adjustment unit 45 performs calibration based on the projected figures photographed by the adjacent photographing devices 3a to 3d (S9).

FIG. 11 is a diagram for explaining a calibration method according to a comparative example. FIG. 12 is a diagram for explaining deformation of a marker sheet according to a comparative example. As shown in FIG. 11, in the calibration method according to the comparative example, special marker sheets M with patterns are arranged as markers at four locations around the vehicle in the oblique direction. At this time, if the place G where the marker sheet M is arranged is not flat and has unevenness, the marker sheet M is deformed as shown in FIG. As described above, the calibration method according to the comparative example has limitations such as avoidance of unevenness.

According to the calibration method of the first embodiment, calibration is performed based on the projected figure corrected in accordance with the projection plane. Therefore, the projection plane can be examined without considering that the marker will be placed. Therefore, in the calibration method, it is possible to improve the degree of freedom of location.

Although the embodiments of the present invention have been described above, the present invention is not limited to the configurations of the above-described embodiments, and various modifications and combinations are possible within the technical scope of the present invention. . For example, the measurement device 21 measures the distance from the measurement device 21 to the projection plane, but alternatively or additionally, the angle from the measurement device 21 to the projection plane may also be measured. Also in this case, in the first embodiment, 3D information similar to the case where 3D information of the projection plane is created based on the distance between the projection device and the projection plane can be created.

The projection device 2 of Embodiment 1 is provided on the roof of the vehicle body so as to project the projection figure from a direction oblique to the projection plane, but it is installed at a position directly above the projection plane. may Moreover, the projection device 2 may be provided at a position other than the vehicle body, such as a ceiling provided above the vehicle body, as long as the projection device 2 is connected to the calibration device 4 . According to these forms, the correction amount of figure data can be reduced as compared with the case of projecting a projection figure from a direction oblique to the projection plane.

Further, the photographing device 3 may be configured to give feedback to the projection device 2 so as to change the display position, size, or the like of the photographed projection figure.

Also, the storage unit 41 may store a plurality of graphic data. In this case, the graphic data to be used can be easily changed to graphic data projected as a highly recognizable projected graphic suitable for the calibration environment.

Further, in general, depending on the uneven shape of the projection surface S, there is a possibility that part of the projected figure P may be lost, and part of the pattern of the projected figure P photographed by two adjacent ones of the photographing devices 3a to 3d may be different. There is However, by maintaining the size of the projection figure P and projecting the projection figure P with a sufficient distance from the vehicle, the uneven shape becomes relatively small with respect to the size of the projection figure P. . Therefore, it is possible to ignore the difference in appearance of the projected figure P from the adjacent photographing devices 3a to 3d, thereby suppressing deterioration in calibration accuracy.

FIG. 13 is a schematic configuration diagram of a calibration system 1A according to Modification 1 of Embodiment 1. As shown in FIG. As shown in FIG. 13, the calibration system 1A has projection devices 2a1, 2a2, 2b1, 2b2, 2c1, 2c2, 2d1, and 2d2. The projection devices 2a1 and 2a2 are provided in the vicinity of the imaging device 3a so as to correspond to the viewing angle of the imaging device 3a, and project a projection figure P obliquely forward right and obliquely forward left of the vehicle, respectively. Similarly, the projection devices 2b1 and 2b2 are provided in the vicinity of the imaging device 3a so as to correspond to the viewing angle of the imaging device 3b, and project a projection figure P obliquely forward left and obliquely rearward left of the vehicle, respectively. The projection devices 2c1 and 2c2 are provided in the vicinity of the imaging device 3a so as to correspond to the viewing angle of the imaging device 3c, and project a projection figure P obliquely rearward left and obliquely rearward right of the vehicle, respectively. The projection devices 2d1 and 2d2 are provided near the photographing device 3a so as to correspond to the viewing angle of the photographing device 3d, and project a projection figure P obliquely to the rear right and obliquely to the right of the vehicle, respectively. As a result, for example, one projection figure P is projected by the projection devices 2a1 and 2c2. Other projection figures P are similarly projected by two projectors. According to the calibration system 1A, the projection devices 2a1, 2a2, 2b1, 2b2, 2c1, 2c2, 2d1, and 2d2 are provided corresponding to the viewing angles of the photographing device 3, and one projection is performed by two projection devices. Project a figure P. Therefore, it is possible to avoid the phenomenon that the pattern of the projected figure P photographed by the adjacent photographing devices 3a to 3d is partially different, which occurs when one projected figure P is projected by one projection device. .

1 calibration system 2 projection device 2a projection device 2b projection device 2c projection device 2d projection device 2a1 projection device 2a2 projection device 2b1 projection device 2b2 projection device 2c1 projection device 2c2 projection device 2d1 Projection device 2d2 Projection device 3 Photographing device 3a Photographing device 3b Photographing device 3c Photographing device 3d Photographing device 4 Calibration device 21 Measurement device 22 Projection lamp 41 Storage unit 42 Measurement unit 43 Processing section, 44 projection section, 45 adjustment section.

Claims (8)

measuring the shape of the projection plane;
creating corrected figure data corrected in accordance with the shape of the projection plane;
a step of projecting a projected figure based on the corrected figure data onto the projection plane;
photographing the projected figure from a plurality of viewpoints;
A calibration method, comprising: calibrating an image combining position based on the projected graphics photographed from the plurality of viewpoints. The step of creating the correction figure data includes:
2. The calibration method according to claim 1, further comprising creating 3D information of said projection plane and creating said correction figure data based on said 3D information. The step of creating the corrected figure data based on the 3D information includes:
3. The calibration method according to claim 2, further comprising the step of creating the 3D information of the projection plane based on the distance between the projection device that projects the projection figure and the projection plane. The step of creating the corrected figure data based on the 3D information includes:
4. The calibration method according to claim 2, further comprising the step of creating the 3D information of the projection plane based on an angle between a projection device that projects the projection figure and the projection plane. The step of creating the correction figure data includes:
On the estimated plane, the corrected figure data is created by changing the scale of the figure data, which is the original shape data of the projected figure, based on the distances between the plurality of measurement points on the projection plane and the reference point. The calibration method according to any one of claims 2 to 4, comprising the step of: The step of creating the correction figure data includes:
generating the corrected graphic data by changing the scale of the graphic data based on the distance between the plurality of measurement points having height differences with respect to the plane and the projection device that projects the projected graphic; A calibration method according to claim 5, comprising: a processing unit that measures the shape of a projection plane and creates corrected figure data corrected in accordance with the shape of the projection plane;
a projection unit that causes a projection device to project a projection figure based on the correction figure data onto the projection plane;
A calibration device, comprising: an adjusting unit that calibrates a position for synthesizing images based on the projected graphics photographed from a plurality of viewpoints. a projection device that measures the shape of a projection plane and projects onto the projection plane a projection figure based on corrected figure data corrected in accordance with the shape of the projection plane;
a photographing device for photographing the projected figure from a plurality of viewpoints;
A calibration system, comprising: a calibration device that creates the corrected figure data and calibrates a position for synthesizing images based on the projected figure photographed from a plurality of the viewpoints.

Images