RU2608350C2 - Simulating installation involving method and device for determining coordinates of object in two dimensions - Google Patents

Abstract

FIELD: measurement technology; physics.

SUBSTANCE: invention relates to techniques for determining coordinates of an object in two dimensions by simulating installations. Proposed is a device for determining coordinates of a control device driven by the user in two dimensions. Device comprises the first camera for infrared imaging including an infrared image of the control device and an infrared image of the user and the second camera to generate a calibration image of the control device and of the user image and a stream of images of the control device and the user images. Device also has the first computing module to generate a training image configured to extract a vector for an edge contour of the control device image from the infrared image using the model of the active contour.

EFFECT: technical result is higher accuracy of detecting an object in two dimensions owing to the use of one infrared camera.

13 cl, 4 dwg

Description

This invention relates to a simulator, including a method and apparatus for determining the coordinates of an object in two dimensions. More specifically, but not limited to, this invention relates to a simulator for training a specialist.

Traditionally, specialists, such as soldering irons, study their specialty at work as an apprentice. The apprentice gains various required skills, trying to repeat the work of his teacher. The training provides a focused personal training experience. However, this form of training is unchanged, as the teacher’s time is spent on many students. Moreover, at least in the early stages of training, the student makes mistakes that will incur costs for the teacher and keeps him / her from hiring apprentices in the future.

Vocational training courses are developed to provide students with the initial experience necessary to start working as a specialist in order to reduce the initial costly period of training. However, these courses are relatively expensive due to the number of student mistakes made in the early stages.

According to a first aspect of the invention, there is provided a device for determining the coordinates of an object in two dimensions, comprising: a first camera for forming an infrared image of the object; a second camera for generating a calibration image of the object and the image stream of the object; a first computing module for generating a training image, configured to extract a vector corresponding to the edge of an object from an infrared image using an active loop model, and with the possibility of applying the vector to a calibration image to form a training image; adaptive correlation filter created from a training image; and a second computing module, configured to correlate at least one image of the image stream from the second camera with an adaptive correlation filter to determine the x-y coordinates of the object by comparing the amplitude of the maximum peak in the correlation plane in the adaptive correlation filter with a detection threshold.

Applying the active contour model to an image from an infrared camera, which, in turn, is used to create a training image for an adaptive correlation filter, the device can accurately track an object using only one infrared camera. In the prior art, multiple infrared cameras are used to track an object. In this regard, the present invention reduces the cost of tracking an object by reducing the need for many expensive infrared cameras. The active circuit model works synergistically with an infrared camera, since the contrast between the object and the person allows you to form an exact vector corresponding to the edge of the object. In this regard, the device operates in cluttered environments such as a living room.

In this regard, the present invention can use a standard camera, for example a color camera, such as a VGA camera, which provides images in the visible spectrum that can be used by this device.

Preferably, a plurality of rotatable training images is created, wherein these rotatable training images are created by rotating the training image, wherein an adaptive correlation filter is created from the rotatable training images.

By rotating the training images, the device can maintain tracking in relation to the object, that is, it can accurately and stably measure the position of the object over a longer period of time without updating the adaptive correlation filter with a new training image. Rotated training images increase the device's tolerance for changes in orientation, changes in scale and position.

Preferably, the training image or rotated training images are created with the ability to periodically update the adaptive correlation filter. The adaptive correlation filter can be updated every 0.5-1.5 seconds, preferably every second.

By updating the adaptive correlation filter with a new training image and using a set of rotated training images, the adaptive correlation filter is “retrained”. This increases the accuracy of the adaptive correlation filter for the next image from the second camera.

The adaptive correlation filter may be a MACH filter, an OT-MACH filter, or the like.

According to a second aspect of the invention, there is provided a method for determining the coordinates of an object in two dimensions, the method comprising the steps of: receiving an infrared image including an object from an infrared camera; receiving a stream of images including an object and a calibration image including an object from a camera; forming a vector corresponding to the edge of the object from an infrared image using the active contour model; forming a training image by extracting an object from a calibration image using a vector; create an adaptive correlation filter using a training image; and determining the x-y coordinates of the object in the correlation field by correlating at least one image from the image stream with an adaptive correlation filter.

The method may further comprise the step of generating a plurality of rotatable training images, this set of rotatable training images being created by rotating the training image, and creating an adaptive correlation filter from the rotatable training images.

A training image or a plurality of rotatable training images may periodically update an adaptive correlation filter. The adaptive correlation filter can be updated every 0.5-1.5 seconds or, preferably, every second.

A computer program executed on a computer-readable medium may be configured to implement a method according to a second aspect of the invention.

Embodiments of the invention will now be described by way of example and with reference to the drawings, in which:

Figure 1 illustrates a simulator installation including a computer, a head-mounted display, a control device, and a camera unit;

2 is a flowchart illustrating a method of an embodiment of the present invention for measuring coordinates along the x axis and y axis of a control device;

Figure 3 illustrates the OT-MACH filter of an embodiment according to Figure 2; and

Figure 4 illustrates, for reference only, the method of measuring the coordinate along the z axis of the control device.

Figure 1 illustrates an overview of the simulator 1. The simulator 1 includes a control device 100, a computer 200, a camera unit 300, and a head-mounted display 400. For the purposes of this description, the computer 200 is configured to run a computer program that simulates a learning scenario as for example using a blowtorch or bending a pipe.

Computer 200 receives data from control device 100 and camera unit 300. The control device 100 includes various sensors for measuring spatial characteristics, such as acceleration and orientation, and for measuring user input data. The control device 100 outputs the data from the sensors to the computer 200. The camera unit 300 includes a first camera 310 and a second infrared camera 320 for acquiring an image. The camera unit 300 outputs image data to a computer 200.

The computer 200 is configured to process data from the control device 100 and the camera unit 300 in the form of input variables to a computer program. The control device 100 provides spatial data such as acceleration and orientation and user input data, while the camera unit 300 provides images that can be processed by the method of the present invention to track the control device 100. A computer program that can simulate a learning scenario, in connection with this, it can give the user an immersive and accurate simulation of real skills, such as using a blowtorch or bending a pipe. The tracking method of the control device 100 is described below in more detail.

Tracking

In normal use, the simulator 1 is installed in the room, with the camera unit 300 facing the control device 100. Typically, the camera unit 300 will be placed against the wall and facing the control device 100 in the center of the room. The control device 100 is held by the user.

For the purpose of the description, three dimensions are indicated along the Cartesian axes x, y and z, the z axis being located in the direction from the camera unit 300 to the control device 100 (i.e., this axis is parallel to the floor). Both the x axis and the y axis are orthogonal to the z axis and to each other. The computer 200 is configured to calculate coordinates along the x axis and y axis of the control device due to the first method and calculate the coordinates along the z axis due to the second method.

The first method, i.e. the method of calculating the coordinates along the x axis and y axis of the control device 100 will now be described with reference to Figures 2 and 3. The method is performed on the computer 200 using image data from the camera unit 300. The camera unit 300 obtains a calibration image by means of the first camera 310 and obtains an infrared image by means of the second infrared camera 320. As noted above, the camera unit 300 is facing a control device 100 that is held by a user. In this regard, the calibration image and infrared image include a control device 100 and a user.

Figure 2 illustrates an overview of the first method. As a preliminary step, subtracting the background of the infrared image due to temporal differentiation distinguishes the control device 100 and the user from a constant background. This forms a processed infrared image that includes only the control device and the user, suitable for the following steps.

The active loop model is applied to the processed infrared image to create an accurate vector contouring the edge of the control device 110. The control device 110 is clearly distinguishable from the user in the processed infrared image by using reflective infrared radiation from the coating on the control device 110.

The active loop model operates on the principle of minimizing energy to determine the edge vector of the control device 100 in the processed infrared image. The energy of each point of the vector is calculated based on adjacent pixels. The Gaussian difference (DoG) of the filtered image is calculated to highlight the edges of the control device 100. This energy minimization process is an iterative continuous process until the edge vector of the control device is calculated accurately. The energy function calculated and iterated for each point of the vector is described in the equation below, where i is the number of iterations, varies from 1 to n, where n is the number of points on the vector, and E ^* _vector is the calculated energy of the point of the vector.

E ^* _vector = α.E _int (ν _i ) + β.E _int (ν _i )

Computer 200 includes a configuration file for changing the number of iterations i obtained to accurately calculate the edge vector of the control device 100.

Once the edge vector of the control device 110 is calculated, the vector is then applied to the calibration image from the first camera to extract the control device 110. The extracted control device 110 is then applied to the center of the clean background, which forms a training image suitable for the OT-MACH filter.

In this embodiment, the training image is further processed to form a plurality of rotatable training images for the OT-MACH filter. For example, the training image is rotated with a two degree increase between -6 degrees and +6 degrees, thus obtaining 7 rotated training images. These rotated training images are multiplexed and injected into the OT-MACH filter.

Next, the operation of the OT-MACH filter will be described in more detail with reference to Figure 3. The OT-MACH filter (Optimal Trade-off Maximum Average Correlation Height) is performed on a computer 200 using the FFTW library (“Fastest Fast Fourer Transform in the West”). The FFTW library is a C routine library for performing the discrete Fourier transform in one or more dimensions. The FFTW library interoperates with Intel’s OpenCV Intel’s library of machine vision, making the OT-MACH filter efficient in terms of processing time and frequency.

As shown in the left column in Figure 3, the OT-MACH filter receives a set of rotated training images t _{i = 1-N} , where N is the number of rotated training images. Each rotated training image undergoes a Fourier transform FT (T _i ). The FFTW result is not biased FFT. The offset of the zero component of the FFT to the center of the spectrum is performed using the following function, designated C.

cνFFTWShift ()

This function has the effect of swapping the upper left quadrant with the lower right quadrant and swapping the upper right quadrant with the lower left quadrant.

The OT-MACH filter is expressed in the equation below, where m _x is the middle of the vector x _{1-N of} rotated training images in the frequency range, C is the diagonal matrix of the power spectral density of any selected noise model, D _x is the diagonal matrix of the average power spectral density of the rotated training image , and S _x denotes the similarity matrix of a set of rotational training images. These parameters are extracted from the training image. Alpha, beta and gamma are non-negative optimal trade-offs that allow the OT-MACH filter to adapt to environmental conditions, such as light levels. α, β, and γ can be changed in the configuration file.

h =

Computer 200 receives a stream of images from the first camera. As shown in the right column of Figure 3, the set of sub-images S _{k = 1-N} , where N is the number of sub-images, is obtained from one image from this stream. Each sub-image undergoes a Fourier transform FT (S _k ). The Fourier transform sub-images correlate with the OT-MACH filter in the frequency range using the function below.

conj (FT (h)) FT (S _k )

Further, each sub-image is classified as either in-class or out-of-class by comparing the amplitude of the maximum peak in the correlation plane with the detection threshold. The detection threshold is given in the equation below.

A correlation field is formed for each intraclass sub-image. The position of the control device 100 in the x and y direction corresponds to the highest value in the correlation field.

An OT-MACH filter is applied to each mth image from the first camera to generate a correlation field and to determine the position of the control device 110. The parameter m can be changed in the configuration file. The OT-MACH filter can be updated, that is, with a new set of rotating training images obtained and applied to the OT-MACH filter, either in real time or at the frequency specified by the parameter in the configuration file.

The second method, that is, for calculating the coordinate along the z axis of the control device 100, is described below for reference only, with reference to Figure 4. The coordinate on the z axis is the distance from the center of gravity of the first and second cameras to the control device 100.

The half-angle of the first camera θ ₁ and the second camera θ _{2 is} calculated using the following expression, where D is the field aperture of the first or second camera, and f is the focal length of the first or second camera.

In Figure 4, the z coordinate can be determined from the following expression, where α _1,2 can be measured using the half-angle of the view and position on the x axis and y axis of the control device 100 calculated using the first method.

Alternatively, if the first and second cameras are calibrated, camera-specific and non-camera-specific parameters can be found using the OpenCV functions.

One skilled in the art will recognize that rotational multiplexing, that is, rotation of a training image, to form a plurality of rotatable training images, is an unimportant feature of the present invention. More specifically, the OT-MACH filter can be created from a training image. One skilled in the art will recognize that creating an OT-MACH filter from a plurality of rotating training images is preferable since it provides an OT-MACH filter tolerance between filter updates so that position accuracy increases and the computer is less likely to not track the control device 100.

One skilled in the art will also understand that updating the OT-MACH filter, that is, generating a new set of rotatable training images or training images, is an unimportant sign. More precisely, the OT-MACH filter can be created from the first set of training images and not updated. Of course, one skilled in the art will understand that updating the OT-MACH filter is highly preferred since it provides a more accurate determination of the position of the control device 100.

Moreover, for the OT-MACH filter, an insignificant feature is an update once every 25 images from the image stream from the first camera (that is, for a common camera, once every second, where the camera captures 25 frames per second). The person skilled in the art will understand that the OT-MACH filter refresh rate can be changed by changing the configuration file.

One skilled in the art will recognize that it is not essential for a filter to be an OT-MACH filter. More specifically, any form of adaptive correlation filter can be used, for example, a MACH filter, an ASEF filter, a UMACE filter, or a MOSSE filter can be used instead.

One skilled in the art will also understand that simulator 1 is not limited to the soldering operation scenarios described above. More specifically, simulator 1 can be used for various forms of virtual reality situations, such as other educational, entertainment, and industrial situations. In particular, the method of teaching an object described above can be used in other situations, for example, in the manufacturing industry.

In the above embodiment, the computer 200 includes a computer program. One skilled in the art will understand that a computer program may be implemented on a computer readable medium, such as a CD or a USB flash drive, or may be downloadable from the Internet. The computer program can also be stored on the server in a remote location, while the user personal computer can send data to the server via a network connection.

The person skilled in the art will also understand that a head-mounted display is a non-essential feature. More specifically, computer 200 may provide graphic material to a computer monitor, projector, TV, HDTV, 3DTV, or the like. A head-mounted display is a preferred feature as it provides an immersive experience for the user, and can also provide data related to the orientation of the user's head, which can then be used in turn by simulator 1.

The person skilled in the art will understand that any combination of features is possible without deviating from the scope of the present invention, which is claimed in the claims.

Claims (24)

1. A device for determining the coordinates of a control device, driven by the user, in two dimensions, containing a first camera for generating an infrared image including an infrared image of a control device and an infrared image of a user; a second camera for generating a calibration image of the control device and the user image and the image stream of the control device and user images; a first computing module for generating a training image configured to extract a vector for a contour of an image edge of a control device from an infrared image including an infrared image of a control device and an infrared image of a user using an active circuit model, and for applying said vector to a calibration image to form a training images and thus separating the image of the control device from the image Vatel; adaptive correlation filter formed from a training image; and a second computing module configured to correlate at least one image from the image stream from the second camera with an adaptive correlation filter to determine the xy coordinate of the control device by comparing the amplitude of the maximum peak in the correlation plane of the adaptive correlation filter with the detection threshold. 2. The device according to claim 1, wherein a plurality of rotatable training images is generated, wherein these rotatable training images are created by rotating the training image, wherein the adaptive correlation filter is created from rotatable training images. 3. The device according to claim 1, in which the training image is generated periodically to update the adaptive correlation filter. 4. The device according to claim 2, in which rotated training images are generated periodically to update the adaptive correlation filter. 5. The device according to p. 3 or 4, in which the adaptive correlation filter is updated every 0.5-1.5 seconds. 6. The device according to claim 1, in which the adaptive correlation filter is an OT-MACH filter. 7. A method for determining the coordinates of a control device driven by a user in two dimensions, the method comprising the steps of: receiving an infrared image including an infrared image of a control device and an infrared image of a user from an infrared camera; receive a stream of images of the control device and user images and a calibration image of the control device and user images from the camera; forming a vector for the contour of the edge of the control device from an infrared image including an infrared image of the control device and an infrared image of the user using the active circuit model, and thereby separating the control device from the user; form a training image by extracting the control device from the user from the calibration image using the specified vector; create an adaptive correlation filter using a training image; and determine the coordinates xy of the control device in the correlation field by correlation of at least one image from the image stream with an adaptive correlation filter. 8. The method of claim 7, further comprising the step of generating a plurality of rotatable training images, wherein these rotatable training images are formed by rotating the training image and an adaptive correlation filter is formed from rotatable training images. 9. The method according to p. 7, in which the training image is formed periodically to update the adaptive correlation filter. 10. The method of claim 8, wherein the rotational training images are generated periodically to update an adaptive correlation filter. 11. The method according to p. 7, in which the adaptive correlation filter is updated every 0.5-1.5 seconds. 12. The method of claim 7, wherein the adaptive correlation filter is an OT-MACH filter. 13. A computer-readable medium containing a program recorded thereon, which, when executed by a computer, causes the computer processor to perform the steps of the method according to any one of claims. 7-12.

Images