WO2003083526A1

WO2003083526A1 - A method and a device for co-ordinated displaying of a direct image and an electro-optical image

Info

Publication number: WO2003083526A1
Application number: PCT/SE2003/000493
Authority: WO
Inventors: Erik DALÉN; Lennart B. M. Svensson; Hans Biverot
Original assignee: Linus Ab; Biverot, Cecilia, Margareta
Priority date: 2002-03-28
Filing date: 2003-03-26
Publication date: 2003-10-09
Also published as: AU2003216020A1; SE0200983L; SE0200983D0

Abstract

The present invention relates to a method and an arrangement for co-ordinated display of a direct image and an electro-optical image. Displays introduce a certain level of distortion into the image. When the direct image is co-ordinated with an electro-optical image, the distortion that is introduced by the display causes the quality of the combined image to be low, which results in the image being of limited use. In order to improve the image quality, according to the invention, a copy of the direct image is first converted into electro-optical form using a first electro-optical image sensor (1) and is then combined electronically or optically in a combining device (3) with the electro-optical image from a second image sensor (2), after which the combined image is transmitted to a display device (4).

Description

A method and a device for co-ordinated displaying of a direct image and an electro-optical image

The present invention relates to a method and an arrangement for co-ordinated and superimposed display of a primary direct image and a secondary electro- optical image. Electro-optical images, EO images, comprise in this connection, among other things, images generated using infrared cameras and associated apparatus, TV cameras, image intensifiers, CCD arrangements, imaging radar, X-ray sensors, etc.

In many situations, there is a desire to be able to combine a direct visual image with an electro-optical image transmitted to a display. It has, however, proved difficult in practice to achieve such a combination which fulfils reasonable quality requirements. The main reason for this is that displays do not perform in an ideal way, but are subject to a certain level of distortion. In order to make display systems attractive as far as cost is concerned, displays are accepted that produce distortion. In display systems where the image, for example, is generated by superimposing different partial images in two or more layers, it can be advantageous as far as cost is concerned to accept simple designs at the expense of slightly distorted image reproduction.

The object of the present invention is to achieve a method and an arrangement for co-ordinated display of a direct image and an electro-optical image, where the distortion of the display does not have a significant effect on the quality of the image and the coordination between the geometries of the partial images . The object of the invention is achieved by a method according to the first paragraph, characterized in that a copy of the primary direct image is first converted into electro-optical form and is then combined electronically with the secondary electro-optical image, after which the combined image is transmitted to a display device, and an arrangement according to the first paragraph characterized in that the arrangement comprises a first electro-optical image sensor arranged to generate a first electro-optical image which is a copy of a direct image, a second electro-optical image sensor or electronic image generator arranged to generate a second electro-optical image, such as an infrared image, target or symbols, means for combining the electro-optically generated images, and a display device for displaying the combined images.

By creating a copy of the direct image and combining this in the correct scale and geometry with a second electro-optical image before the whole is transmitted to a display device, any distortion that the displays involved may introduce affects both the original images to an equal extent . The images correspond independently of the distortions introduced by the display. In this way, control is achieved over relative geometries between the images .

The method and the arrangement according to the invention have many areas of application. One example is within different types of vehicle, both on the ground and in the air. For example, the crew of a vehicle can be given intensified vision by combining the direct visual image with an infrared image or a radar image. The invention can also replace "HeadUp Displays". In this way, sights can be combined with the direct image with the correct geometry in spite of distortions in the display device. Another possibility is to replace a head mounted display (HMD) with sights or other superimposed images in combination with cursors or to utilize head movements for moving the secondary image in the combined image with display via a line display. In the case of head movements, indication of direction or movement of secondary EO images or other symbols is carried out by means of head movements, using sensors that detect the head position of the viewer, so-called head tracker devices. The head mounted part of the equipment for observation with a line display is hereby simplified, in comparison to corresponding systems according to the HMD principle.

According to an advantageous further embodiment of the method, the combined image is introduced into the viewer's field of vision as a partial image and the partial image is arranged to be a part of an image in which a direct visual partial image is at least partially included. Such a method is very suitable for the driver of a vehicle. The direct visual partial image can correspond to the view through the lower part of the windscreen, while an upper part can consist of, for example, a combined infrared and TV image. Through the lower part of the windscreen, the driver obtains direct visual information about the immediate vicinity, while the combined infrared and TV image provides information about an area further away.

According to yet another advantageous embodiment of the invention, the transition between the direct visual partial image and the combined partial image is designed as a soft transition, where the visual partial image and the combined partial image overlap each other. In particular, the intensity of the partial images at the transition is adjusted so that intensity decreases in the direction towards the adjacent partial image. The introduction of a soft transition gives the viewer the impression of a more coherent image without pronounced divisions into different parts of an image. The method according to the invention can advantageously be used in such a way that the combined partial image is obtained from further away than the visual image.

The display device that is included in the arrangement according to the invention can consist of a line display in association with scanner glasses or a strapdown scanner device. Mirror optics in the form of a visor can be arranged to image the line display. The basic principle for such a display device with the use of scanner glasses is described in our Swedish patent 9101778-0. A strapdown scanning device is described in our Swedish patent 9901979-6.

According to another advantageous embodiment, the display device consists of a conventional display provided with mirror optics in the form of a visor for imaging the display.

In order to achieve a soft transition in a device where the combined images are arranged to constitute a partial image in a system in which another partial image consists of a direct visual image, according to yet another advantageous embodiment, the display device in the arrangement according to the invention is designed to be transparent to varying degrees at the transition with the direct visual image.

Additional advantageous embodiments are apparent from the following patent claims.

The invention will be described in greater detail below in the form of examples with reference to the attached drawings, in which:

Figure 1 shows schematically an arrangement according to the invention for co-ordinated display of a direct image and an electro-optical image. Figure 2 shows an example of how the arrangement according to the invention can display information to a viewer.

Figure 3 shows an arrangement according to the invention with a display device of the line display type with scanner glasses.

Figure 4 shows an arrangement according to the invention with a display device of the line display type with strapdown scanner.

Figure 5 shows an arrangement according to the invention with a target controlled by the viewer using superimposing and display via a line display.

The arrangement shown in Figure 1 for co-ordinated display of a direct image and an electro-optical image comprises a first image sensor 1 and a second image sensor 2. The first image sensor consists of a TV camera that generates a first electro-optical image constituting a copy of a direct image. The second image sensor consists here of an infrared camera that generates a second electro-optical image. The electro- optical images from the two image sensors are transmitted to a combining device 3 which generates a co-ordinated electro-optical image from the transmitted electro-optical images, which co-ordinated electro- optical image is displayed by a display device 4 connected to the combining device 3.

Adding, suppressing, intensifying and time-adjusting functions, etc, can be available in the combining device 3. The functions can be realized by means of a processor programmed in a suitable way. The display device is shown here as a conventional display. Some alternative embodiments will be described with reference to Figures 3 and 4.

With reference to Figure 1, the second image sensor has been shown as an infrared camera. There are, however, many other alternatives, as already mentioned in the first paragraph of the description. Nor does there need to be an image of the surroundings, such as a landscape, but there can be, for example, a target or other symbols geometrically aligned with the first image .

Figure 2 shows what can represent a window 8 in a vehicle, such that a viewer inside the vehicle can see the whole window. Two image fields 5 and 6 can be identified. The visual direct image that the viewer can see falls within the image field 5. The image field 6 situated above the image field 5 comprises a display device 4 in which a combined image is displayed which is the result of combining the direct image that falls within the image field 6 converted to electro-optical form by a TV camera and an electro-optical image from an infrared camera. A strip 7 marks the transition between the image fields 5 and 6. In order to make the transition soft, the image fields 5 and 6 can be arranged with a certain amount of overlap. If the image field 6 is created by a display that is physically present in the image field, this must be of a type that is at least partially transparent. Alternatively, the image is reflected up from a conventional display. Another alternative is to reflect up a display line into the image field 6 and then to create a two- dimensional image using scanner glasses or a strapdown scanner device. By adjusting the transmission of the display at the transition or adjusting the lines in a line display that relate to the transitional area, a soft transition can be created. It is also possible to let a certain small part of the direct light be transmitted through the whole image field 6.

To the right of the image field 6, an additional image field 9 is shown. In a straightforward embodiment, this image field can be included in the image field 5 and accordingly the strip 7 on the right side of Figure 2 is omitted.

In the example shown with reference to Figure 2, objects, for example pedestrians or animals, further away that are imaged by the infrared camera can be emphasized more clearly. The TV image in the image field 6 can still provide colour information, for example red warning lamps, braking lights on other vehicles, etc.

In addition to the electro-optical image above, other information from maps, instruments, radio, telecommunication or other information, so-called telemetry, can be found in the image field 6.

The display device 4 shown in Figure 3 comprises a line display transmitter 10 which is directed at a window 8. The line transmitted by the line display is reflected in the window 8 in the image field 6 and is then directed towards the viewer's eye 11. The viewer wears a pair of scanner glasses 12. The scanner glasses transform the incoming line into a two-dimensional image. It is assumed that the scanner glasses are also provided with a visual channel without scanning function for viewing the direct visual image through the window 5.

The display device shown in Figure 4 likewise comprises a line display 10 which is directed at a window 8 and more specifically the reflecting image field 6. Between the line display 10 and the window 8 there is a scanner device 13. The scanner device can consist of two movable plates 14, 15 which cause the line from the line display to sweep over the image field 6 and create a two-dimensional image. The two-dimensional image is reflected towards a viewer's eyes 11.

An arrangement according to the invention with a target controlled by the viewer is shown in Figure 5. The target is shown by means of a cross in two different positions, A and B. An image of the surroundings is generated in one or more electro-optical image sensors 1 and composited with an electronically generated target 21 which is created in a processor 17. The composited image is displayed using a line display consisting of a line transmitter 10 and a head-mounted scanner 12. In the figure, it is assumed that the target 21 is moved by means of movements of the viewer's 18 head. The movements of the viewer's head are indicated in the figure by the arrow 19. Sensors 20 measure the movements of the viewer's head. Alternatively, a strapdown cursor can be used.

The invention is not limited to the embodiments described above, but can be modified within the framework of the following patent claims and the concept of the invention. For example, there is a plurality of possible applications in addition to those mentioned specifically above.

Claims

1. Method for co-ordinated display of a primary direct image and a secondary electro-optical image of the surroundings or of symbols, such as an infrared image, TV image or the like, characterized in that a copy of the primary direct image is first converted into electro-optical form and then combined electronically with the secondary electro-optical image, after which the combined image is transmitted to a display device retaining the co-ordination between the primary and secondary images in spite of distortion in the display device.

2. Method according to the preceding claim, characterized in that the combined image is introduced into the viewer's field of vision as a partial image and in that the partial image is arranged to be a part of an image in which a direct visual partial image is at least partially included.

3. Method according to Claim 3, [sic] characterized in that the transition between the direct visual partial image and the combined partial image is designed as a soft transition, where the visual partial image and the combined partial image overlap each other .

4. Method according to Claim 3, characterized in that the intensity of the partial images at the transition is adjusted so that the intensity decreases in the direction towards the adjacent partial image.

5. Method according to any one of the preceding Claims 3 or 4, characterized in that the combined partial image is obtained from further away than the visual partial image.

6. Method according to any one of the preceding claims, characterized in that the secondary image is arranged to be movable relative to the primary converted electro-optical image under the control of the viewer.

7. Method according to any one of the preceding claims, characterized in that the copy of the primary direct image of the surroundings is generated in electro-optical form.

8. Method according to any one of the preceding claims, characterized in that the copy of the primary direct image of the surroundings is generated using electro-optical image sensor technology, such as TV, infrared, UV or image intensifiers .

9. Arrangement for co-ordinated display of a direct image and an electro-optical image, such as an infrared image, TV image or the like, comprising electro-optical image sensors and a display device, characterized in that the arrangement comprises a first electro-optical image sensor arranged to generate a first electro-optical image which is a copy of a direct image, a second electro-optical image sensor or image generator arranged to generate a second electro-optical image, such as an infrared image, target or symbols, means for combining the electro-optically generated images, and a display device for displaying the combined images.

10. Arrangement according to Claim 9, characterized in that the display device consists of a line display in association with scanner glasses or a strapdown scanner device.

11. Arrangement according to Claim 10, characterized in that mirror optics in the form of a visor are arranged to image the line display.

12. Arrangement according to Claim 9, characterized in that the display device consists of a conventional display provided with mirror optics in the form of a visor for imaging the display.

13. Arrangement according to any one of Claims 9-12, in which the combined images are arranged to constitute a partial image in a system where a second included partial image consists of a direct visual image, characterized in that the display device is designed to be transparent to varying degrees at the transition with the direct visual image in order to create a soft transition.

14. Arrangement according to any one of Claims 9-13, characterized in that the arrangement comprises a cursor controlled by the viewer for moving the secondary image relative to the primary converted electro-optical image.

15. Arrangement according to any one of Claims 9-14, characterized in that the arrangement comprises sensors for detecting the viewer's head movements for controlling the movement of the secondary image relative to the primary converted electro-optical image on the basis of the movements of the viewer's head.