RU2705644C1 - Helmet calibration device - Google Patents

Abstract

FIELD: aviation.

SUBSTANCE: invention relates to a device for calibrating a helmet, in particular a helmet used by an aircraft pilot. Device includes memory, camera and mechanical drive, to which helmet is connected during calibration so that it can move relative to camera. Processor programmed to control the mechanical drive to move the helmet relative to the camera through a sequence of discrete points on the template of calibration targets and in each of discrete control points of the camera for obtaining a digital image is attached to the camera and the mechanical drive. For each of the images, the processor determines the position in the image of at least one of several reference marks and uses this position in the image of at least one reference mark together with the position of the mechanical drive for calibrating the helmet.

EFFECT: invention provides more accurate determination of helmet position.

5 cl, 2 dwg

Description

Technical field

The present invention relates to a device for calibrating a helmet, in particular a helmet used by an airplane pilot.

State of the art

In modern aircraft combat systems, the pilot uses a helmet to control other systems in the aircraft, such as weapons systems, just looking in a certain direction. This is usually achieved by placing at least one camera in the cockpit and at least one reference mark on the helmet.

However, it should be borne in mind that the accuracy of the device largely depends on the proper calibration of the device, since each helmet is slightly different in shape, and if the reference mark is located on the helmet even with a slight deviation, this will greatly affect the accuracy of determining the position of the helmet and, as a result, on the accuracy of helmet-controlled systems.

This application provides an improved helmet calibration device.

Disclosure of invention

A device for calibrating a helmet with a group of optical reference marks located on it is proposed, comprising:

memory;

a camera;

mechanical drive to move the helmet or camera relative to each other during the calibration process;

a processor connected to a camera and a mechanical drive programmed for:

controlling a mechanical drive to move the helmet or camera relative to each other through a group (sequence) of discrete points on a template of calibration targets;

at each of the discrete points, control the camera to obtain a digital image;

for each of the images, determining the position in the image of at least one of the optical reference marks; and

using the position in the image of at least one optical reference mark, together with the position of the mechanical drive, for calibrating the helmet.

In accordance with another particular embodiment, when using a device for calibrating a helmet with several optical reference marks located on it, it is provided:

controlling a mechanical drive to move the helmet or camera relative to each other through a sequence of discrete points on the template of calibration targets;

at each of the discrete points, camera control for digital image acquisition;

for each of the images, determining the position in the image of at least one of several optical reference marks; and

using the position in the image of at least one optical reference mark, together with the position of the mechanical drive, for calibrating the helmet.

Brief Description of the Drawings

Below the invention is described in more detail with reference to the accompanying drawings, in which:

in FIG. 1 shows a particular embodiment of a device for calibrating a helmet; and

in FIG. 2 schematically shows a camera for use in a device in accordance with the present application.

Detailed Description of the Invention

The system and methodology described herein relates to a helmet calibration device.

On the helmet 10, the calibration of which is carried out in accordance with this application, there are several reference marks 12. The calibration device can operate with any number of light emitting diodes (LEDs). For various helmet position / orientation tracking devices using the calibrations of this device, a number of LEDs may be required.

In a particular embodiment, several reference marks 12 are light emitting diodes (LEDs).

A helmet 10 is attached to a mechanical actuator 14 to perform calibration.

In the embodiment shown in the drawings, the mechanical drive 14 is a robot arm 14 having six degrees of freedom.

The helmet 10 is connected to the robot arm 14.

In practice, each helmet has its own, different attachment points, so a transitional holder is required to attach the helmet to the manipulator.

To obtain images of the helmet 10 and reference marks 12 in several positions, a camera 16 is used, as will be described in more detail below.

For the calibration process, the camera 16 is fixedly mounted relative to the helmet 10 so that the helmet 10 moves relative to the camera.

It is possible to move the camera with a stationary helmet.

The camera 16 itself must be properly calibrated, which should ensure that there is no distortion in the resulting image, which would lead to incorrect calibration of the target. An example of a camera calibration technique is described in Applicant's published application PCT / IB2012 / 056820, the contents of which are incorporated herein by reference.

A processor 18 is connected to the camera 10 and the robotic arm 14, which, in the present embodiment, is part of the computer 20.

The processor 18 is programmed to control a mechanical drive 14, providing the movement of the helmet 10 relative to the camera 16 through several discrete points on the template calibration targets.

In the ideal case for each LED, the angles of the camera's field of view should form a cone with an angle from the axis of 45 degrees and a vertex pointing towards the LED. This will mean that the camera vectors are orthogonal, which ensures the greatest noise immunity.

At each of the discrete points, the processor 18 instructs the camera 16 to generate a digital image.

Each digital image is stored in memory 22 associated with the processor 18, for further processing of this digital image.

At the same time, or with some delay, the processor 18 extracts each of the digital images and determines the position of each of several reference marks in the image.

The processor 18 then calibrates the helmet 10 using the position of the at least one reference mark in the image.

It should be borne in mind that this calibration requires knowledge of the intrinsic parameters of the measuring chamber 16 (distorted / undistorted ratio (DU - from the English Distorted-to-Undistorted) and focal length), as well as the absolute orientation of the camera relative to the robot arm 14. The latest data, in turn, require knowledge of the orientation of the holder (helmet) and the offset orientation of the camera. This data is usually directly output to the photogrammetric camera calibration device.

The robot manipulator 14 is used to simulate a group of calibrated cameras with known orientations observing each optical reference point. Each camera in the group uses its own parameters to determine the vector directed from the camera to the optical reference point. The closest intersection point of these vectors is used as the measured position of the optical reference point. It would be impractical to use a large group of calibrated cameras directed inward, rigidly mounted in known positions so that they can, by performing a triangulation survey, determine the positions of points on all sides of a three-dimensional object placed in the center of the group. Instead, a robot manipulator is used, representing a helmet located with different orientations, to one calibrated camera, the orientation of which relative to the robot is known.

In each orientation presented, the camera captures the centroid of the optical reference points (reference marks 12) and uses its own parameters to determine the vector directed to the optical reference point. This vector and camera orientation are further calculated relative to the tip of the robot arm using the camera’s own parameters and the known robot orientation. The result is a bunch of vectors and their corresponding known points defined in the general coordinate system of the tip of the robot arm.

To calculate the closest intersection point of all vectors of the beam, the closest intersection point of each intersection of each possible pair of vectors is calculated and averaged.

After determining this averaged pairwise triangulated position, it is used as a starting point for further numerical approximation. This process is repeated for each reference mark 12 on the helmet.

The above procedure is performed by a processor 18 using several modules.

In the context of the present description, it is assumed that the "module" includes a recognizable part of the code, computational or executable instructions, data, or computational task to perform a specific function, operation, processing or procedure. It follows that the module does not have to be implemented in the form of a program; a module may be implemented as a program, hardware, or a combination of software and hardware. In addition, the module does not have to be combined in one device, but can be distributed across several devices.

According to the attached drawings and a more detailed description of the technique, a vector directed from the camera to the object in the field of view (FOV - from the English field of view) of the camera can be determined if the camera’s own parameters (focal length, pixel size, main point and skewed image axes (assumed to be zero in modern imaging devices)).

We assume that the distortion of the lens is either negligible or has already been taken into account. Image coordinates are converted to two-dimensional (2D) space relative to the main point using pixel sizes. When changing the scale (see Fig.), It is important to take into account the difference in the symbols of the positive direction for the image and the coordinate system (SC) of the camera. In the general case, it is impossible to determine the distance to an object from its position in the field of view of a single camera, therefore the vector is reduced to a unit vector (EB). The third dimension is the focal length of the camera. Equation (1) illustrates the implementation of this operation:

Where

= вектор, заканчивающийся в инвертированной плоскости изображения, направленный от камеры к объекту,

= vector ending in the inverted plane of the image, directed from the camera to the object,

= ЕВ, направленный от центра камеры к объекту,

= EB, directed from the center of the camera to the subject,

= положение пикселя неискаженного 2D изображения объекта,

= pixel position of an undistorted 2D image of an object,

= положение пикселя пересечения оптической оси,

= position of the pixel of the intersection of the optical axis,

pix_w = pixel width in the camera image,

pix_h = pixel height in the camera image,

FLen = focal length of the lens of an equivalent pinhole iris model.

It should be noted that the focal length and pixel size must be determined in the same units.

Now we will determine the nearest intersection point of two three-dimensional (3D) lines. The exact intersection of two 3D lines in free space is unlikely. Instead, they form the closest intersection point. In three-dimensional space, a line is usually defined by a unit vector (EB) showing its direction, and a 3D point through which the line passes. At the points on the two lines where they are closest, the line segment between the two lines will be perpendicular to both lines. Since the scalar product of perpendicular lines is equal to zero, two equations (equal to zero scalar products of a line segment and each line) with two unknowns (the distance to the line segment in EB of each line from the known point of each line) can be composed and then solved together. Then, the average position of two points on the line closest to each of them can be taken as the nearest intersection point. Formally, this is expressed by equation (2):

Where

= положение ближайшей точки пересечения,

= position of the nearest intersection point,

, and

, and

= точка на линии 1,

= point on line 1,

= единичный вектор направления линии 1,

= unit direction vector of line 1,

= точка на линии 2,

= point on line 2,

= единичный вектор направления линии 2.

= unit direction vector of line 2.

Using the Brown distortion model, it can be shown that with the corresponding parameters of the Brown distortion model, the effect of lens distortion is both in the direction from the distorted image to the undistorted image (DU), and in the direction from the distortion-free image to the distorted image (UD - from English Undistorted- to-Distorted), can be successfully modeled. Each direction will require a separate set of parameters.

The determination of the parameters DU and UD can be performed using any suitable set of programs for photogrammetric calibration.

The basic Brown model expresses the coordinates of the input pixel relative to the main point. Then, to obtain the position of the pixel at the output, a radial and tangential offset is added to the distance of the point at the input from the main point.

The radial and tangential displacements are in the form of polynomials, depending on the distance of the point at the entrance from the main point. The distortion parameters are formed by the coordinates of the main point and the coefficients of the radial and tangential polynomials. Formally, this is expressed by equation (3):

Where

= Модель дисторсии Брауна [4, 5],

= Brown distortion model [4, 5],

= точка изображения на выходе,

= point of output image,

= точка изображения на входе,

= input image point,

= центр дисторсии,

= center of distortion,

R _n = N ^th the radial distortion coefficient,

T _n = N ^th tangential distortion coefficient,

N _R = number of radial parameters,

N _T = number of tangential parameters, and

It should be noted that it is impossible to use one tangential parameter, it requires zero, or two or more tangential parameters. Whether or not equation (3) creates a distortion of the image point depends on what parameters are entered into it.

Further throughout the present description, the introduction of equations called DU parameters into equation (3) means that the point will be transformed from a distorted region distortion to an undistorted region distortion.

Similarly, the parametric vector UD-parameters will be used to transform the distorted coordinates of the pixels into the corresponding coordinates in the distorted distortion region.

The following is a description of the determination of the spatial position of the light emitting diodes (LEDs) on the helmet.

This calibration requires its own parameters of the measuring chamber (DU properties and focal length), as well as the absolute orientation of the camera relative to (with respect to) the robot. For this, in turn, it is necessary to know the orientation of the holder and the offset orientation of the camera.

A robotic arm is used to simulate a group of calibrated cameras with known orientations observing each LED. The closest intersection point of these vectors is used as the measured position of the LEDs. It is impractical to use a large group of calibrated cameras directed inward, rigidly mounted in known positions so that they can, by performing a triangulation survey, determine the positions of points on all sides of a three-dimensional object placed in the center of the group.

To simulate this procedure, a robot manipulator is used, representing a helmet located with a different orientation, to one calibrated camera, the orientation of which relative to the robot is known. For each orientation in which the helmet is presented, the orientation of the calibrated camera relative to the helmet is calculated, which actually adds another camera to the virtual group of cameras, each of which looks inward at the helmet.

In each orientation presented, the camera captures the image of the LED (s) and determines the coordinates of each LED (s) in the image. The specific method for determining the coordinates of the LED image can be changed, in a particular embodiment, the methods described in previously published patent application No. PCT / IB2012 / 056820 are used.

Then, based on the camera calibration parameters, the corresponding vector associated with the image coordinate is calculated. Further, this vector and camera orientation are converted to the coordinate system (SC) of the robot arm using the known camera orientation relative to the robot base SC and the orientation of the robot arm relative to the robot base SC.

The result is a bunch of vectors and associated known points defined in the general SC of the robot manipulator.

To calculate the nearest intersection point of all vectors of the beam, an initial estimate of the nearest point of each intersection for the entire beam is determined and its numerical approximation is performed. The initial estimate is calculated by averaging the nearest intersection point for each possible pair of vectors. This initial estimate is mathematically expressed by equation (4):

Where

= первоначальное положение СИД относительно манипулятора робота,

= initial position of the LED relative to the robot arm,

N = number of kinds of LEDs,

= положение СИД относительно манипулятора, полученное триангуляцией из положений i и j,

= LED position relative to the manipulator, obtained by triangulation from positions i and j,

= преобразование положения камеры относительно концевого захвата в положении n,

= conversion of the position of the camera relative to the end grip in position n,

= ЕВ от камеры к СИД с манипулятором в положении n,

= EB from camera to LED with manipulator in position n,

= согласно уравнению (2),

= according to equation (2),

= согласно уравнению (1),

= according to equation (1),

= модель дисторсии Брауна [4, 5],

= Brown distortion model [4, 5],

= параметры исправления дисторсии, согласно, например [3],

= distortion correction parameters, according to, for example [3],

= собственные параметры камеры, согласно, например [3],

= own camera parameters, according to, for example [3],

= положение пикселя СИД для ориентации i робота,

= LED pixel position for robot orientation i,

= ориентация камеры относительно робота по оси робота, и

= orientation of the camera relative to the robot along the axis of the robot, and

= ориентация манипулятора в положении n относительно робота по оси робота.

= orientation of the manipulator in position n relative to the robot along the axis of the robot.

After determining the initial estimate using equation (4), it is used as a starting point for a further numerical approximation.

This approximation seeks to find a point with a minimum rms value of the sum of the perpendicular distances from each vector in the beam. In a particular embodiment, the leapfrog algorithm (frog jump) is used. Equation (5) mathematically expresses the cost function of the rms value of the sum of perpendicular distances.

Where

C ^HLED = cost function to determine the conversion of the position of the helmet LED.

= предполагаемое положение СИД относительно манипулятора,

= assumed position of the LED relative to the manipulator,

N = number of kinds of LEDs,

= положение камеры относительно манипулятора, находящегося в положении i,

= camera position relative to the manipulator in position i,

= ЕВ от камеры к СИД при i ориентации робота,

= EV from camera to LED with i orientation of the robot,

= в соответствии с уравнением (1),

= in accordance with equation (1),

= модель дисторсии Брауна [4, 5],

= Brown distortion model [4, 5],

= параметры исправления дисторсии, согласно, например, [3],

= distortion correction parameters, according to, for example, [3],

= собственные параметры камеры, согласно, например, [3],

= own camera parameters, according to, for example, [3],

= положение пикселя СИД для ориентации i

= LED pixel position for orientation i

= известная ориентация камеры относительно робота, и

= known orientation of the camera relative to the robot, and

= переданная ориентация концевого захвата относительно оси робота.

= transmitted orientation of the end gripper relative to the axis of the robot.

The above process is repeated for each LED on the helmet.

Fixing data on all the required centroids on the helmet is usually performed by sequential viewing during one movement.

It will be shown below how the positions of light emitting diodes can be expressed in different SCs. This may be necessary if it is not possible to install the helmet on the robot arm so that the proposed coordinate systems of the helmet and the robot end grip coincide or even are set on the same axis. In addition, installing a helmet on a robot arm may not be repeatable (although it must be rigid in order for calibration to be effective).

For this calibration, a set of LED reference positions is required that the LED positions of the measured set should match as closely as possible. Usually this is the result of a theoretical study of the supporting provisions, the implementation of which would be achieved through the use of advanced technological equipment.

This calibration determines the orientation shift, which, being applied to each of the measured LED positions, ensures their maximum compliance with the LED reference positions.

The accuracy of this correspondence is expressed numerically by the root mean square (RMS) value of the distances between the measured LED positions (after taking into account the orientation shift) and the LED reference positions.

The orientation that provides the minimum SLE for distance deviation is determined by numerical optimization. In the given particular embodiment, the “leapfrog” algorithm is again used, where the initial position is set by the transitional holder used to attach the helmet to the end grip of the robot.

The cost function that determines the SLE deviation of the distances is expressed by equation (6):

Where

C ^{M → T} = the cost function of the corrective conversion of the measured LED position to theoretical,

= предполагаемая ориентация, обеспечивающая совмещение измеренных и теоретических массивов точек СИД,

= assumed orientation, ensuring the combination of measured and theoretical arrays of LED points,

= теоретическое положение СИД i,

= theoretical position of LED i,

= уточненное измеренное положение СИД i,

= updated measured position of LED i,

, и

, and

= измеренное положение СИД i шлема относительно манипулятора робота

= measured position of LED i of the helmet relative to the robot arm

The procedure described above is practically implemented using the steps below.

Data collection phase:

For each entry in the list of robot orientations:

one). Set the robot in the specified orientation.

2). Fix and memorize the exact orientation adopted by the robot arm.

3). Fix and store the image of the optical reference points for this orientation.

Helmet calibration phase:

one). Retrieve the camera’s own parameters from the memory, including the focal length, pixel sizes, the position of the main point and the correction parameters for distortion and distortion of the lens.

2). Extract from the memory the orientation of the camera relative to the robot arm.

3). For each optical reference point on the helmet:

3.1). For each captured image of this optical reference point:

3.1.1). Retrieve this image from memory.

3.1.2). Determine the pixel coordinates of the optical reference point in the image.

3.1.3). Convert image coordinates to 3D vector in the camera coordinate system using the extracted own camera parameters.

3.1.4). Retrieve from the memory the obtained orientation of the robot arm for this image.

3.1.5). Calculate the orientation of the camera relative to the end of the robot arm using the extracted orientation of the robot arm and the extracted camera orientation.

3.1.6). Calculate the projection of the 3D vector in the coordinate system of the tip of the robot arm.

3.1.7). Determine the approximate closest intersection point of all 3D vectors for this optical reference point.

3.1.8). Perform a numerical refinement of this nearest intersection point.

3.1.9). Remember this point as the measured position of the optical reference point.

3.2). Save this set of optical reference points in memory.

Helmet adjustment phase:

one). Retrieve a set of model optical reference positions from memory.

2). Retrieve a set of measured optical reference positions from memory.

3). Determine the sum of the offsets between the corresponding optical reference points.

4). Find the orientation, when applied to the measured optical reference positions, the minimum value of the sum of the displacements is reached.

5). Apply the found orientation to the measured optical reference positions to generate a set of corrected optical reference positions.

6). Store corrected optical reference positions.

The device can also be used to verify the accuracy of the helmet position / orientation tracking device, i.e., calibrate cameras, aircraft and helmet. This is done by installing the robot arm in a combat aircraft, and presenting the calibrated helmet in known positions to a group of calibrated cameras. Then the differences between the optical measurements and the known orientations are calculated. If desired, this information can be used to further refine the calibration of cameras, helmet or aircraft, to increase the installed working accuracy of a particular installed helmet position / orientation tracking device.

Thus, it should be understood that the calibration of the helmet position / orientation tracking device satisfies the need for manufacturers and end-users in helmet tracking technology to ensure optimal performance of their devices. Such calibration can be used as initial calibration, calibration after any incidents, for example, a hard fit, and periodic recalibrations according to logistics plans.

Claims (25)

1. A device for calibrating a helmet with a group of optical reference marks placed on it, comprising: memory; a camera; mechanical drive to move the helmet or camera relative to each other during the calibration process; a processor connected to a camera and a mechanical drive programmed for: controlling a mechanical drive to move the helmet or camera relative to each other through a group of discrete points on a template of calibration targets; at each of the discrete points, control the camera to obtain a digital image; for each of the images, determining the position in the image of at least one of the optical reference marks; and using the position in the image of at least one optical reference mark, together with the position of the mechanical drive, for calibrating the helmet. 2. The device according to claim 1, in which the processor is configured to, when determining the position of at least one of the reference marks: determining the coordinates of the pixels of the optical reference mark in the image; transformation of image coordinates into a 3D vector in the camera coordinate system using the camera’s own parameters extracted from the memory; extracting from the memory the orientation of the robot arm for the image; calculating the orientation of the camera relative to the tip of the robot arm using the extracted orientation of the robot arm and the extracted camera orientation; calculating the projection of the 3D vector in the coordinate system of the tip of the robot arm; determining the approximate closest intersection point of all 3D vectors for this optical reference point; and performing numerical refinement of this nearest intersection point as the measured position of the optical reference mark. 3. The device according to claim 2, which provides for the storage in memory of certain positions of each of the optical reference marks. 4. The device according to any one of paragraphs. 1-3, in which the processor is configured to, when using a specific position of at least one optical reference mark in the image, along with the position of the mechanical drive for calibrating the helmet: extracting from memory a set of model reference positions of optical reference marks; extracting from memory a set of certain reference positions of optical reference marks; calculating the sum of the displacements between the corresponding optical reference points; determine the orientation, when applied to the measured optical reference positions, the minimum value of the sum of the displacements is achieved; and applying a specific orientation to the measured optical reference positions to generate a set of corrected optical reference positions. 5. The device according to claim 4, which provides for the storage in memory of a set of corrected optical reference positions.

Images