DE102011108367A1 - Method of displaying selected object in two-dimensional video image using camera, involves determining position of selected object with respect to its correct position and outputting position in form of label - Google Patents

Abstract

The method involves determining coordinates of selected object by operating location system and transmitting two-dimensional video image to a central computer. The parameters such as angle, focal length, image sensor size, resolution of current image of camera are determined and the determined result is transmitted to central computer. The position of selected object is determined with respect to its correct position and/or perspective in which the two-dimensional video image is calculated and the position is output in the form of a label. An independent claim is included for a device for displaying selected object in two-dimensional video image.

Description

Field of the invention

The invention relates to a method for the mass production of video images, which are generated, for example, by a conventional camera, wherein by means of geometry data, which originate from a locating system independently operating from the conventional camera, such as from a real-time location system, prominent points in those generated by the conventional camera Images in terms of position and / or perspective and / or can be measured with correct coordinates.

The invention further relates to a device for the mass production of video images produced, for example, by a conventional video camera (eg operating with CMOS, CCD or similar technology, hereafter referred to as "conventional camera") using geometry data which originate from a positioning system operating independently of the camera, such as, for example, from a real-time location system, distinctive points in the images produced by the conventional camera can be identified and / or correctly marked with respect to position and / or perspective.

The necessary for the realization of the method or the device features are given in the independent claims 1 and 4 respectively.

In the context of the invention, the term video images is understood to provide at least one selected object in at least one two-dimensional video image with correct coordinates with respect to the position of the selected object in the two-dimensional video image and / or correct this position (in terms of coordinates and / or perspective). to mark in the at least one two-dimensional video image by means of a marker.

Description of the invention

Real-time location systems currently support, in particular, modern logistics processes as well as security-relevant applications. They provide complex data about one or more positions and / or motions of selected objects (e.g., tagged with an electronic tag) within a defined spatial area, typically defined by the field of view of the measurement system. Real-time location systems collect data in digital form and, after creating digital models, make it possible to present located objects in a digital world. In this way, for example, complete buildings, their interior design and architectural features to scale and reproduce.

Using suitable software, the objects - preferably based on their coordinates in a reference system - can then be visualized in three-dimensional views. This procedure is comparable with a virtually simulated production line, in which the located objects are digitally drawn to the correct position with respect to position and / or perspective.

New application scenarios arise when the use case requires that located objects are related to moving and non-existent and / or permanently visible objects.

These are, for example, flows of persons between located security officers or located transport systems in interaction with specialist personnel at interfaces of the material flows within a warehouse administration. Digital models (alone) can not handle the high environmental dynamics in such applications. cover. Obtained visualizations are often too static to generate added value for the relevant business processes.

In order to meet these requirements, conventional cameras are installed for monitoring and recording the current system behavior and, in a continuation, for example, connected to camera systems. For example, the system behavior in connection with flows of people can be recorded as well as the situation in production environments and in security-relevant areas of a company.

However, between the images taken by the conventional camera on the one hand and the position data of the (spatially separated) real-time location on the other hand, a media break occurs.

The information provided by the two systems "conventional camera" on the one hand and "real-time location system" on the other hand are always available digitally. The further information processing process, however, requires further measures to integrate the data from the real-time location system into a video image provided by a conventional camera. Known solutions of the prior art complicate or slow down the processing process and may reduce the quality of the result under certain circumstances.

The video footage provided by a conventional camera provides real images of the environment. The temporal sequence of the delivered pictures allows the behavior of the system to be assessed and evaluated over a period of time.

The data provided by real-time locating systems make it possible to identify and track objects even in scenes that are confusing and insufficiently detailed for conventional cameras.

For moving images from a fixed perspective that does not change by pan, tilt or magnification, object data originating from real-time locating systems (usually their coordinates in a fixed frame of reference) can be obtained by conventional mathematical methods in video images provided by a conventional camera and thus in a real world Integrate image. The position of concrete objects can be marked on the video image preferably by color overlay or border.

If, when the conventional camera moves, a new viewing angle no longer corresponds to the basic perspective (and thus the parameters for the mathematical calculations of selected object positions have changed), the new image section is automatically recognized and the entire image is processed using image processing techniques Imaging system to be freshly calibrated to the changes. A precise visualization of the location data or located objects is therefore still possible.

1 shows an example of an application in a laboratory environment. In the middle is a video picture taken by a conventional camera. Marked (by reference arrows) are the points supplied in the form of 3D coordinates by the real-time locating system and, according to the invention, transformed into the 2D plane of the video image.

State of the art

In the prior art, there are systems which, through hardware and control electronics, allow feedback of a conventional camera to software subsystems with the exact viewing angle, horizontal and vertical tilt, and magnification of the frame. Special sensors and / or feedback from servomotors determine the exact orientation of the camera and pass on the associated data to a software subsystem.

The parameters for mathematical calculations of the positional and / or perspective correct integration of data from a real-time locating system into an image from a conventional camera can thereby be changed or adapted. The tagging of the objects as described above may continue to work correctly.

Solution according to the invention

Since many systems are already installed and a conversion to controllable cameras, according to the requirements just described, for economic and infrastructural reasons is unacceptable, the invention provides a method that has the advantage that the technical effort in the field of data provision from a conventional camera ("video technology") is only in the provision of the video material or the offer of a digital video stream.

The movement of the camera is determined by means of image processing software ("image analysis"). By comparing several images in a temporal sequence, for example, the changes in perspective can be determined.

A combined representation of live image and virtual information is commonly referred to as "augmented reality".

The solution to the problem of video image measurement consists with regard to the device and the method of several known sub-components whose interaction as a whole represents the functionality and thus the inventive solution.

The device according to the invention for the dimensioning of video images consists of the essential components:

I. Real-Time Location System (RTLS)

For position detection, for example, a known in the prior art real-time location system (RTLS) such as the company Ubisense be used.

In principle, any RTLS that has sufficient accuracy is suitable for use. The system returns the position of a (selectable) object within a defined spanned measurement space. The three-dimensional indication of the position can z. B. with x = 2.5, y = 3.8 and z = 1.75 done.

These coordinates as well as the unique designation of the located object are recorded and made available for further processing.

II. Conventional cameras or camera systems

Commercially available cameras (eg in CMOS technology) can be used. An exemplified laboratory design provides IP network cameras in a preferred embodiment. The resolution is z. B. 0.3 megapixels (resolution 640 × 480 pixels). Regardless of the camera and the technical infrastructure, software subsystems are provided with individual images, which produce a video stream as a continuous, temporally ordered image sequence. The position of the conventional camera is known at any time via its coordinates (x _K , y _K , z _K ). In the static placement of one (or more) conventional cameras, the exact position is measured after installation.

If the position of the camera is changed during operation, because they z. B. is mounted on a transport system, then the changed, current position of the conventional camera must be determined in real time, so that the entire system works with consistent data.

For this purpose, the invention provides that the conventional camera - analogous to an object to be located - is also equipped with appropriate positioning electronics.

This ensures that a changed current position of the conventional camera can also be detected and determined by the real-time system belonging to the entire system. The changed position data of the conventional camera, which leads, for example, by its movement (translational and / or rotational) to a new point of view, which no longer coincides with the existing basic perspective, are further processed by an image processing software and lead to - modified because of the changed Abbildungsverhältnissen Parameters for the mathematical calculation of the positions of selected objects, thus ultimately to a (preferably real-time) calibration of the entire imaging system with respect to changes due to the movement of the conventional camera.

For a correct calibration of the entire system, the position of the conventional camera as well as the recorded angle of view must be known. The recorded angle of view can be determined via the sensor size and the focal length of the conventional camera. With regard to limited imaging accuracies (in particular of low-cost CMOS cameras), it may have to be determined empirically, since the computationally determined angle is not necessarily correct. The conventional camera or camera systems constructed therewith preferably have to be leveled horizontally, in order to achieve a correct image position even with lateral pivoting movements.

III. Mathematical calculation

Mathematical methods known per se describe a procedure for integrating each three-dimensionally located object (O _n , n ∈ N) into a two-dimensional image of a real scene or for identifying its exact position. In this case, a perspective transformation is performed, which includes a translation and rotation to move the coordinate origin to the position of the camera and to rotate the coordinate axes so that the third dimension is omitted. This first part of the method is known from the theory of computer graphics and z. B. in the Textbook by Techmer (2003) p. 21 treated.

The second part of the mathematical calculation is the adjustment when changing the perspective. The image analysis methods listed below under point IV allow the recognition of specific image areas and / or image elements. For images along a temporal function, the shift of certain pixels within the two-dimensional image may be an indication of the camera's viewing angle change.

So it is recognized that a particular pixel z. If, for example, it is no longer displayed centrally in the image, a pixel-precise shift in the X and Y directions can be determined, which has led to the absence of a central representation (more).

Since it is known how large the angle of view is (for example, depending on the focal length of the lens of the conventional camera and the size of its image sensor), assuming an equal distribution and the rotation angle can be determined. These rotation angles are now included in the mathematical calculation. For this purpose, the normal vector of the image plane is transformed into the polar coordinate system, also circular coordinate system, and the angle components are adapted by adding the determined rotation angle.

• – ihre aktuelle Lage (Translation)
• – ihre Rotation relativ zu einer fest definierten Orientierungslinie

The conventional camera now has information about:

• - their current situation (translation)
• - Their rotation relative to a fixed orientation line

in eine reale Abbildung an die Position

zu zeichnen. Die Angaben u und v sind dabei die Anzahl der Pixel ausgehend von der Bildmitte in die horizontale und vertikale Bildrichtung.The third part of the mathematical calculation now determines the pixel-precise two-dimensional image position of located objects in order to use the object O ₁ after the transformation

in a real picture to the position

to draw. The data u and v are the number of Pixels starting from the center of the picture in the horizontal and vertical picture direction.

IV. Image-based analysis methods

Image analysis uses gray value-based and contour-based approaches. The mode of operation itself is state of the art. On the other hand, it is not known to use it in connection with the mass-production of video images for the purpose of localization.

The detection of the camera movement and above all the exact new angle is done via two image analysis methods, which are shown below.

Gray value based methods

For initial calibration after commissioning of the entire system or the application (without prior knowledge of the environment and camera position), it is necessary to calibrate the viewing angle of the conventional camera. For this purpose, a picture of the conventional camera is examined for specific templates. A template is a specific pattern or element that can be part of another image. The templates in the application described are also measured geographically in three dimensions. By a gray value-based normalized cross-correlation ( Steger, 2008, p. 214 ) the position of the template can be determined. The methods are relatively robust to additive brightness differences.

Contour-based methods

If no template is found, then a contour-based method can be used to detect edges or contours within two images and to determine the shift within the two images. Possible implementations are the Sobel operator or the Laplace method. The number of pixels that are moved to specific edges can be converted to an angle or a degree via the image angle. As described above, this angle or number of degrees is used for recalibrating the conventional camera or the entire system.

The advantage of the device according to the invention or of the method that can be implemented therewith is that objects which are not visible or recognizable in a real scene recorded by a conventional camera are supplemented by digital information.

According to the invention, it has been recognized that it is advantageous not to recognize the objects within the image, but to obtain the coordinates of the objects of interest from (real-time) location systems and, after mathematical processing of the raw data, the object position with respect to position and / or perspective correctly into that of draw in the picture supplied to the conventional camera. This allows even hidden objects or very small objects to be marked or highlighted.

As the angle of view can be automatically adjusted or adjusted in the case of arbitrary pivoting movements of the conventional camera, objects can thus also be tracked in space - and in real time.

This is possible according to the invention, which is highlighted as an advantage of the device according to the invention, even using standard cameras (webcam o. Ä.).

Exemplary technical fields of application of the invention Systems for monitoring personal safety Employees, goods and transport vehicles in safety-critical areas of traffic infrastructures can not only be located, but also visualized at any time. This is useful when accumulations of people and / or objects affect the overview.

Military or police training

At exercise sites positioning systems are used to evaluate the exercise afterwards. The invention can now be used to enable an evaluation based on real images with simple installation of a conventional camera. People who are not recognizable in the picture are still visualized.

Example: Training firefighters in training buildings

Exemplary construction and operation of the device

• – Echtzeitortungssystem (nachfolgend RTLS – real time locating system)
• – liefert unabhängig vom technischen Verfahren zeitaufgelöste bzw. mit Zeitmarken versehene dreidimensionale Koordinaten eines Objektes relativ zu einem festgelegten Bezugssystem (z. B. P(t₁)(1,3 m, 2,7 m, 6,2 m))
• – die gelieferten Koordinaten entsprechen der Position einer elektronischen Marke (üblicherweise ein RFID-Tag), die an einem Objekt fixiert ist
• – Koordinaten werden vorzugsweise in festgelegten Zeitabständen erfasst
• – Bilderfassungsgerät – konventionelle Kamera (z. B. digitale Videokamera mit Netzwerkschnittstellen)
• – handelsübliche Auflösung (z. B. PAL 720×576 Pixel)
• – verfügt über Schwenk und Neigefunktion, fernsteuerbar
• – liefert Bildmaterial digital über Netzwerk- oder Videoschnittstellen
• – folgende Parameter müssen bekannt sein:
• – Bildverhältnis (z. B. 4:3)
• – Brennweite
• – evtl. extrinsische Einflüsse auf die Kameraoptik
• – die Kamera muss horizontal, also parallel zur X-Y-Ebene ausgerichtet sein
• – Server
• – leistungsfähiger Server
• – zur Analyse des Bildmaterials aus allen zum System gehörenden konventionellen Kameras,
• – zum visuellen Verschmelzen von Kameradaten und Ortungsdaten
• – sowie für die Bereitstellung der Videodaten für Clientanwendungen

For the construction of a device according to the invention in a preferred embodiment, the following components are required:

• - Real-time location system (hereinafter RTLS - real time locating system)
• Independent of the technical procedure, provides time-resolved or timestamped three-dimensional coordinates of an object relative to a fixed reference system (eg P (t ₁ ) (1.3 m, 2.7 m, 6.2 m))
• - The supplied coordinates correspond to the position of an electronic tag (usually an RFID tag), which is fixed to an object
• - Coordinates are preferably collected at fixed intervals
• - Image capture device - conventional camera (eg digital video camera with network interfaces)
• - Commercially available resolution (eg PAL 720 × 576 pixels)
• - has pan and tilt function, remotely controllable
• - Provides digital imagery over network or video interfaces
• - the following parameters must be known:
• - aspect ratio (eg 4: 3)
• - focal length
• - possibly extrinsic influences on the camera optics
• - The camera must be aligned horizontally, ie parallel to the XY plane
• - server
• - powerful server
• For analyzing the image material from all the conventional cameras belonging to the system,
• - for visual merging of camera data and location data
• - as well as providing the video data for client applications

Measurement and processing

• – Analyse des Video-Bildinhaltes
• – Berechnung der Objektposition

When carrying out the method according to the invention, the image data from the conventional camera (the "video stream") and the position data from the real-time location system run synchronously in time on a computer, e.g. For example, a server. The further processing is now divided into two steps:

• - Analysis of the video image content
• - Calculation of the object position

In the analysis of the video image content, an attempt is made to find previously defined patterns or "markers" with pixel precision. Templates can be measured marks, but also other striking elements of the environment, such as posters, murals, signs or other fixed objects. These templates are known to the system and their position (in the form of 3D coordinates) has been measured in advance.

The found template with the highest match (to the predefined data) is used for the following calculation steps.

The subsequent processing step calculates the inclination and tilt angle of the camera based on the position of the camera, the pixel-precise position of the template as well as the spatial coordinates of the template. (This function allows the camera to be freely and arbitrarily controlled.) Now, the coordinates (3D) of the located objects are used to be integrated into the frame of the video stream supplied by a conventional camera. The integration consists, for example, of the identification of the object by a circle. The size of the circle preferably includes information such that it gives the user a spatial feeling and thus suggests distance differences between the objects. In order to be able to draw a better conclusion on the position, the marking of the object is supplemented by a vertical line up to the X-Y axis or to the floor of the environment.

output

The (augmented) image now supplemented by additional information is made available to the user as a video stream in chronological order.

Summary of the procedure of the method according to the invention:
RTLS → WebCam → Extracting the coordinates → Calibrating the image display → Concurrent relation between RTLS and video data → Representation in the image

Non-patent literature

• Steger, C. u. (2008). Machine Vision Algorithms and Applications. Wiley VCH Verlag GmbH & Co. KGaA.
• Techmer, A. (2003). Contour-based image motion and object tracking in real-time applications. Munich: Lincolm GmbH.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited non-patent literature

• Textbook by Techmer (2003) p. 21 [0032]
• Steger, 2008, p. 214 [0040]

Claims (4)

Method for displaying at least one selected object in at least one two-dimensional video image from at least one conventional camera, wherein the coordinates of the at least one selected object are determined by a positioning system operating independently of the at least one conventional camera and provided to a central computer, the coordinates of the position the at least one conventional camera delivering the at least one two-dimensional video image is provided to a central computer which determines parameters of the current image of the conventional camera (in particular image angle, focal length, image sensor size, resolution) and is made available to a central computer, characterized in that central computer on the basis of the data provided, the position of the at least one selected object with respect to its correct position and / or perspective in the at least one zweidimens The video signal is calculated and the position in the at least one two-dimensional video image is output in the form of a marker. Method according to Claim 1, characterized in that the current position data of the at least one conventional camera delivering two-dimensional video images is acquired by a positioning system operating independently of this and provided to a central computer. Method according to at least one of the preceding claims, characterized in that calculation and marking of the at least one selected object in a two-dimensional video image takes place in real time. Apparatus for carrying out the method for displaying at least one selected object in at least one two-dimensional video image from at least one conventional camera according to one of claims 1 to 3, wherein the coordinates of the at least one selected object can be determined by a location system operating independently of the at least one conventional camera and provided to a central computer, the coordinates of the position of the at least one conventional camera delivering the at least one two-dimensional video image can be determined and provided to a central computer, the parameters of the current image of the conventional camera (in particular image angle, focal length, image sensor size, resolution) can be determined and provided to a central computer, characterized in that the position of the at least one selected object with respect to its correct position and / or perspective in the at least one two-dimensional video image can be calculated by means of the central computer on the basis of the provided data and the position in the at least one two-dimensional video image can be output in the form of a marker.

Images