WO2024079511A1

WO2024079511A1 - Method and apparatus for calibrating a laser range finder of a camera

Info

Publication number: WO2024079511A1
Application number: PCT/IB2022/059869
Authority: WO
Inventors: Pedro BASTARDO; Rui Fernandes; Andrea CANNIZZARO; Rui Henriques; Nelson CARDOSO; Pedro Miguel CRUZ; Daniel RANZAL; Pedro José DA SILVA CARVALHO PEREIRA DE SOUSA; Susana Isabel BRAGA DIAS DE AGUIAR; Paulo José DA SILVA TAVARES; Pedro Miguel GUIMARÃES PIRES MOREIRA; Francisco LOBATO DA ROCHA BARROS
Original assignee: Bosch Security Systems - Sistemas De Segurança, S.A
Priority date: 2022-10-13
Filing date: 2022-10-14
Publication date: 2024-04-18

Abstract

A method and apparatus for calibrating a laser range finder of a camera comprising an image sensor and an optical lens defining a camera optical axis, the method comprising the steps of: providing a first flat surface perpendicular to the camera optical axis at a first predetermined distance from camera; emitting a laser beam of the laser range finder onto the first surface; capturing an image from the sensor; measuring a first image location of a laser spot centre of the laser beam spot on the first surface, said first image location comprising a horizontal and a vertical position; providing a second flat surface perpendicular to the camera optical axis at a second predetermined distance; repeating the previous operation and determining, from the measured first and the second image location of the laser spot, an image location at an infinite distance of the laser.

Description

D E S C R I P T I O N

METHOD AND APPARATUS FOR CALIBRATING A LASER RANGE FINDER OF A CAMERA

TECH NICAL FIELD

[0001] The present disclosure belongs to the technical field of a sensor range finder device to be coupled into video surveillance systems (e.g. cameras) allowing to measure distances.

BACKGROUND

[0002] Document US2022/0230519A1 providing an invention relates to a machine learning system which is configured, on the basis of a plurality of images captured in succession, to detect smoke within the images. The machine learning system comprises a convolutional recurrent neural network. The invention also relates to a method for detecting smoke by means of this machine learning system.

[0003] Document KR10-2018-0096100A describes a camera aligning apparatus using a laser range finder and a method thereof. The camera aligning apparatus comprises: a multi sensor camera having a laser range finder and a plurality of optical sensors; a driving unit on which the multi sensor camera is loaded and adjusting a direction of orientation of the multi sensor camera; an alignment point display unit disposed at a predetermined position and irradiated with laser beam emitted from the laser range finder to form a laser point; and a control unit controlling the driving unit so that the multi sensor camera orients the alignment point display unit in an alignment mode, receiving an image obtained by photographing the laser point formed on the alignment point display unit by a plurality of optical sensors, adjusting a position of a region of interest so that the centre position of the laser point is the centre of a region of interest in a predetermined size from images obtained from each of the plurality of optical sensors, and aligning the optical sensors. Therefore, operation time may be improved.

[0004] These facts are disclosed in order to illustrate the technical problem addressed by the present disclosure. GENERAL DESCRIPTION

[0005] The present disclosure belongs to the technical field of a sensor range finder device to be coupled into video surveillance systems (e.g. cameras) allowing to measure distances, so as to making a better and quicker focus of complex scenes trough laser focus.

[0006] The present disclosure discloses a method for calibrating a laser range finder of a camera comprising an image sensor and an optical lens defining a camera optical axis.

[0007] The present disclosure details a method and setup required for an alignment process of a laser range finder to a surveillance camera's video sensor, where the laser spot is calibrated to the image centre for object depth measurement inside a certain settable range.

[0008] The method includes the calculation of the laser spot's centre (e.g. following ISO 11146-1) at smaller distances and by least squares fitting for obtaining model parameters, allowing the alignment of the laser range finder with the surveillance camera even if there are no conditions to implement measurements at the projected operational distances. Thus, this represents a favourable option for factory implementation.

[0009] It is also disclosed a method for calibrating a laser range finder of a camera comprising an image sensor and an optical lens defining a camera optical axis, comprising the steps of: providing a first flat surface substantially perpendicular to the camera optical axis at a first predetermined optical distance from the camera; emitting a laser beam of the laser range finder onto the first surface; capturing an image from the image sensor; measuring a first image location of a laser spot centre of the laser beam spot on the first surface, said first image location comprising a horizontal and a vertical position; providing a second flat surface substantially perpendicular to the camera optical axis at a second predetermined optical distance from the camera; emitting a laser beam of the laser range finder onto the second surface; capturing an image from the image sensor; measuring a second image location of a laser spot centre of the laser beam spot on the second surface, said second image location comprising a horizontal position and a vertical position; determining, from the measured first image location and the second image location of the laser beam spot, an image location at an infinite distance of the laser beam spot, said image location comprising a horizontal position and a vertical position; calculating a difference between said determined image location and a predetermined image location at an infinite distance corresponding to the laser beam spot being at the image centre at a predetermined distance.

[0010] In an embodiment, method comprises a step of adjusting the laser range finder by the calculated difference.

[0011] In an embodiment, the image location (a, b) at an infinite distance of the laser beam spot, is determined by:

where ul is the horizontal image location and vl is the vertical image location of the first image location of the laser spot centre, u2 is the horizontal image location and v2 is the vertical image location of the second image location of the laser spot centre, Z1 is the first predetermined optical distance, and Z2 is the second predetermined optical distance.

[0012] In an embodiment, the method comprises a step of adjusting the laser range finder to:

where x_c,y_c are the image centre coordinates corresponding to the image location where the laser beam spot is to be at the predetermined distance (ZAL), and (a', b') are an image location at an infinite distance corresponding to the laser beam spot being at the image centre at the predetermined distance (ZAL).

[0013] In an embodiment, the method further comprises the steps of: providing a third flat surface substantially perpendicular to the camera optical axis at a third predetermined optical distance (Z3) from the camera; emitting a laser beam of the laser range finder onto the third surface; capturing an image from the image sensor; measuring a third image location of a laser spot centre of the laser beam spot on the second surface, said third image location comprising a horizontal position (u3) and a vertical position (v3); determining, from the measured first, second and third image locations of the laser beam spot, the image location (a, b) at an infinite distance of the laser beam spot.

[0014] In an embodiment, the image location (a, b) at an infinite distance of the laser beam spot, is determined by:

where ui is the horizontal image location and vi is the vertical image location of the first image location of the laser spot centre, U2 is the horizontal image location and

V2 is the vertical image location of the second image location of the laser spot centre,

U3 is the horizontal image location and V3 is the vertical image location of the third image location of the laser spot centre, Zi is the first predetermined optical distance, Z2 is the second predetermined optical distance and Z3 is the third predetermined optical distance, Zo is a position of the optical centre of the camera.

[0015] In an embodiment, the method comprises a step of adjusting the laser range finder to:

where x_c,y_c are the image centre coordinates corresponding to the image location where the laser beam spot is to be at the predetermined distance (ZAL), and

are an image location at an infinite distance corresponding to the laser beam spot being at the image centre at the predetermined distance (ZAL).

[0016] In an embodiment, the camera is a video surveillance camera. [0017] In an embodiment, the video surveillance camera is real-time video surveillance camera, in particular real-time having a frame-per-second rate above 30 frames per second.

[0018] In an embodiment, the method comprises measuring an image location of a laser spot centre of the laser beam spot using a centre of mass calculation, in particular wherein the centre of mass calculation is as defined on the ISO 11146-1:2021 standard (2021-07).

[0019] In an embodiment, the flat surfaces are provided by a same surface which is movable between said predetermined optical distances from the camera.

[0020] In an embodiment, the laser of the laser range finder is an infrared or near infrared laser.

[0021] In an embodiment, the image sensor is a visible light image sensor capable of capturing infrared or near infrared light.

[0022] Preferably, the laser range finder is a time-of-flight light-based range finder.

[0023] It is also disclosed a camera calibrated by the method of any of the previous embodiments.

[0024] It is also disclosed an apparatus comprising a computer arranged to carry out the method of any of the disclosed embodiments.

[0025] It is also disclosed a non-volatile computer readable medium comprising computer program instructions which when executed on a computer causes the computer to carry out the method of any of the disclosed embodiments.

BRI EF DESCRI PTION OF THE DRAWI NGS

[0026] The following figures provide preferred embodiments for illustrating the disclosure and should not be seen as limiting the scope of invention.

[0027] Figure 1: Schematic representation of an embodiment for two-dimensional simplification of the alignment matter.

[0028] Figure 2: Flowchart of the alignment procedure according to an embodiment. [0029] Figure 3: Schematic representation of an embodiment for estimation of the parameters for the vertical image coordinate: (a) linear regression using 1/Z as a variable; (b) the same relationship shown changing the x-axis to the value of Z.

[0030] Figure 4: Schematic representation of an image of a surface with multiple laser spots shown at different distances (Z), with an indication of the predicted trajectory of the laser spot (depicted as a rectilinear line) which terminates at infinite (depicted as X).

DETAILED DESCRI PTION

[0031] The present disclosure relates to a sensor range finder device configured to be coupled into video surveillance systems for allowing to measure distances, so as making a better and quicker focus of complex scenes trough laser focus.

[0032] The image position of the range finder's laser spot varies with the distance of the target to the camera, approaching a certain image location as this distance increases and approaching one of the edges of the image as the distance decreases, until it no longer appears on the image for short enough distances. The image coordinates (u, v) of the centre of the laser spot as a function of the distance Z can be described by the following equations:

[0033] The parameters Xo, Yo, a and b are all influenced by the camera sensor and lens properties, but Xo and Yo depend on the position of the LRF (Laser Range Finder) relative to the camera, while a and b depend on its orientation and are the coordinates of the image point which the laser spot converges to for large distances. Zo is the position of the optical centre of the camera.

[0034] The invention consists of a mechanism for adjusting the tilt of the range finder relative to the camera, a physical setup which measures the image location of the laser spot for specific distances and a set of supporting algorithms which compute the mathematical model from measurements and guide the tilt adjustment based on the result. [0035] The goal is the adjustment of the model parameters (namely a and b) by tilting the LRF, such that the laser spot converges towards the centre of the image or, alternatively, such that the laser spot lies at the centre of the image for a specific operational distance.

[0036] The variable distance setup includes a fixed location for the camera and one or more flat surfaces which can be placed at the desired distances, parallel to the camera sensor. This can be implemented either through a plane which moves along the direction of the camera axis, or a series of removable surfaces at set distances. This setup should be shielded from light as well as possible, to ensure that light sources other than the LRF's transmitter do not interfere with the measurement.

[0037] The alignment procedure starts with an initial measurement of the image coordinates of the centre of the laser spot for at least three distinct distances, for an arbitrary configuration of the tilting mechanism. For each distance, an image with the LRF turned on is acquired, and the centre of the laser spot in the image is computed, for instance as the centroid of the intensity distribution of an area surrounding the laser spot, as outlined on ISO 11146-1. The diFstance to the target can be assumed from the physical setup's design or it can be measured, either by the LRF directly or through an independent measurement system.

[0038] Based on the measurements obtained, the model parameters are determined through nonlinear least squares fit. The ideal values for the parameters a and b can now be calculated based on the desired alignment distance, that is, the distance for which the laser spot is at the image centre. If the LRF is intended to be parallel to the camera, these goal parameters should be set to the image centre coordinates. Otherwise, if the centre of the image is at (x_c, y_c) and the alignment distance is Z_ai, then the following expressions can be used:

[0039] Keeping the target at a set distance, the value of the laser spot location for that distance in the goal configuration can be given by replacing a and b in the obtained model with these new values, and inputting the distance to the target. The tilt can now be mechanically adjusted, with periodic or live feedback of the location of the laser spot through repeatedly acquiring images and computing the laser spot's centroid, until the goal location is reached.

[0040] The tilt adjustment affects most significantly the parameters a and b, but it can also slightly change the other parameters. Consequently, the process should be restarted at this point, in order to correct the entire model. When, after the model is computed, the goal values of a and b coincide with the obtained values, the LRF can be considered to be aligned and the process can be stopped.

[0041] A procedure for aligning the range finder with the camera was devised and tested based on the mathematical formulation of the position of the laser spot depending on distance to the camera. Here, we must consider the three-dimensional reality, instead of the two-dimensional simplification presented in the previous section. Because the methodology relies on the calculation of the laser spot's location in the image, it is helpful to consider each image coordinate separately.

[0042] The equation including the alignment distance Zal can be rewritten for each image coordinate, it and v. If xo and yo correspond to the quantity fD for each

coordinate and a and b are eq

^Mual to C - , then: Zal '

[0043] In this formulation, the laser spot's centre will approach the pixel coordinates (a, b} along a line of slope yo/xo as the distance Z approaches infinity. The parameters xo and yo depend on the position of the LRF relative to the camera, whereas a and b depend on the orientation of the LRF. As the relationship between each image coordinate and 1/Z is linear, we can obtain the values of xo, yo, a and b from a linear regression. This calculation can be done using a minimum of only two images at different known distances.

[0044] The concept can be demonstrated using the images taken for the spot size tests, where the range finder and the camera were at fixed positions while images of the laser spot were acquired at several known distances. The coordinates of centre of the laser spot were also calculated as an intermediate result in the spot size computation. The model fit is shown in Figure 3.

[0045] A visual representation of the laser spot's trajectory as the distance to the target increases is presented in Figure 4, on an image consisting of the laser spots at different distances superimposed in a single image.

[0046] The knowledge of the parameters allows to set a desired value for the alignment distance, eliminate unwanted misalignments, and obtain a mathematical expression that provides the location of the laser spot in the image as a function of distance.

[0047] Since the model's parameters can be obtained with a minimum of two measurements at different distances, the minimal setup for the alignment method involves two flat surfaces at set distances from the LRF-enabled camera where one of them is removable, or a flat surface which can move to different distances. In this proposed setup, it is assumed that the camera sensor can detect the laser spot and the LRF assembly allows its orientation to be adjusted. The simplest version of the procedure is comprised of the following steps: measuring the image location of the laser spot centre when the beam hits the first surface, using a centre of mass calculation such as the one defined on the ISO 11146-1 standard; measuring the image location of the laser spot centre when the beam hits the second surface, after removing the first surface; calculating the model parameters using the values of the measured laser spot locations and the distances to the surfaces; calculating the difference between the desired values of a and b and the obtained values; for one of the measured distances, adding this difference to the previously obtained image coordinates to obtain the goal coordinates for the laser spot at that distance; adjusting the LRF's orientation until the laser spot reaches the goal coordinates.

[0048] This method allows for the alignment of the range finder with the camera even if there are no conditions to implement measurements at the projected operational distances, as it is able to extrapolate the location of the laser spot at any distance from measurements performed at short distances.

[0049] There are some steps which can be taken in order to improve the accuracy of the process. Measuring the laser spot location for more than two distances is possible, especially if the distances are as large as possible, as allowed by space constraints. In this case, the parameters are not obtained from direct substitution in the model's equations, but by linear least squares fit, such as the one presented in Figure 3.

[0050] There can also be uncertainty as to where the distance corresponding to Z=0 is precisely located, which could be especially influential in calculations involving very short distances. A solution is to include this distance in the model, substituting Z with Z

- Zo, where Zo is a new parameter to be computed. At least three measurements at different distances are then necessary, instead of two, and the equations are no longer linear, so a nonlinear optimisation process is required to estimate all the parameters.

[0051] Finally, while theoretically changing the orientation of the laser changes only the parameters a and b, keeping xo and yo constant, this does not happen in practice, since due to the real setup of the assembly, the rotation of the LRF causes a slight translation relative to the camera, which makes xo and yo, which depend on the position of the LRF, change slightly if the rotation is large. Therefore, it is recommendable that the process is repeated one more time after the first alignment, so that the final fine tuning of the orientation is made with the correct position values.

[0052] The term "comprising" whenever used in this document is intended to indicate the presence of stated features, integers, steps, components, but not to preclude the presence or addition of one or more other features, integers, steps, components or groups thereof.

[0053] The disclosure should not be seen in any way restricted to the embodiments described and a person with ordinary skill in the art will foresee many possibilities to modifications thereof. The above-described embodiments are combinable. The following claims further set out particular embodiments of the disclosure.

Claims

C L A I M S Method for calibrating a laser range finder of a camera comprising an image sensor and an optical lens defining a camera optical axis, comprising the steps of: providing a first flat surface substantially perpendicular to the camera optical axis at a first predetermined optical distance (Zi) from the camera; emitting a laser beam of the laser range finder onto the first surface; capturing an image from the image sensor; measuring a first image location of a laser spot centre of the laser beam spot on the first surface, said first image location comprising a horizontal (ui) and a vertical (vi) position; providing a second flat surface substantially perpendicular to the camera optical axis at a second predetermined optical distance (Z2) from the camera; emitting a laser beam of the laser range finder onto the second surface; capturing an image from the image sensor; measuring a second image location of a laser spot centre of the laser beam spot on the second surface, said second image location comprising a horizontal position (U2) and a vertical position (V2); determining, from the measured first image location and the second image location of the laser beam spot, an image location at an infinite distance of the laser beam spot, said image location comprising a horizontal position (a) and a vertical position (b); calculating a difference between said determined image location (a, b) and a predetermined image location at an infinite distance corresponding to the laser beam spot being at the image centre at a predetermined distance (ZAL). Method according to the previous claim comprising a step of adjusting the laser range finder by the calculated difference. Method according to any of previous claims, wherein the image location (a, b) at an infinite distance of the laser beam spot, is determined by:

where ui is the horizontal image location and vi is the vertical image location of the first image location of the laser spot centre, U2 is the horizontal image location and V2 is the vertical image location of the second image location of the laser spot centre, Zi is the first predetermined optical distance, and Z2 is the second predetermined optical distance. Method according to the previous claim comprising a step of adjusting the laser range finder to:

, 0 a = x_r — —— c z_al

where x_c, y_c are the image centre coordinates corresponding to the image location where the laser beam spot is to be at the predetermined distance (ZAL), and (a', b') are an image location at an infinite distance corresponding to the laser beam spot being at the image centre at the predetermined distance (ZAL). Method according to any of previous claims, further comprising the steps of: providing a third flat surface substantially perpendicular to the camera optical axis at a third predetermined optical distance (Z3) from the camera; emitting a laser beam of the laser range finder onto the third surface; capturing an image from the image sensor; measuring a third image location of a laser spot centre of the laser beam spot on the second surface, said third image location comprising a horizontal position (U3) and a vertical position (V3); determining, from the measured first, second and third image locations of the laser beam spot, the image location (a, b) at an infinite distance of the laser beam spot. Method according to the previous claim, wherein the image location (a, b) at an infinite distance of the laser beam spot, is determined by:

where ui is the horizontal image location and vi is the vertical image location of the first image location of the laser spot centre, U2 is the horizontal image location and V2 is the vertical image location of the second image location of the laser spot centre, U3 is the horizontal image location and V3 is the vertical image location of the third image location of the laser spot centre, Zi is the first predetermined optical distance, Z2 is the second predetermined optical distance and Z3 is the third predetermined optical distance, Zo is a position of the optical centre of the camera. Method according to the previous claim comprising a step of adjusting the laser range finder to: a = x_c — — - —

Zal ~ Z_o

where x_c, y_c are the image centre coordinates corresponding to the image location where the laser beam spot is to be at the predetermined distance (ZAL), and (a', b') are an image location at an infinite distance corresponding to the laser beam spot being at the image centre at the predetermined distance (ZAL). Method according to any of the previous claims, wherein the camera is a video surveillance camera, in particular the video surveillance camera is real-time video surveillance camera, further in particular real-time having a frame-per-second rate above 30 frames per second. Method according to any of the previous claims, comprising measuring an image location of a laser spot centre of the laser beam spot using a centre of mass calculation, in particular wherein the centre of mass calculation is as defined on the ISO 11146-1:2021 standard (2021-07). Method according to any of the previous claims, wherein the flat surfaces are provided by a same surface which is movable between said predetermined optical distances from the camera. Method according to any of the previous claims, wherein the laser of the laser range finder is an infrared or near infrared laser. Method according to any of the previous claims, wherein the image sensor is a visible light image sensor capable of capturing infrared or near infrared light. Camera calibrated by the method of any of the previous claims. Apparatus comprising a computer arranged to carry out the method of any of the claims 1-12. Non-volatile computer readable medium comprising computer program instructions which when executed on a computer causes the computer to carry out the method of any of the claims 1-12.