DE102020115055A1 - Device and method for displaying augmented reality information - Google Patents

Abstract

In order to provide a device for displaying augmented reality information, in which augmented reality information is displayed in such a way that it is coupled with the movement of an observer and perfectly adapted to the environment, so that an impression of depth is created for the observer A device (100) for displaying augmented reality information (23) is proposed, comprising a display device (12) with a degree of transparency greater than 0%, a first detection device (15) directed into a first half-space in front of the display device (12) , a second detection device (19) directed into a second half-space behind the display device (12), and a computing device (21), wherein the computing device (21) is designed to select an augmented reality information item (23) and an object the augmented reality -Associate reality information (23) and instruct the display device (12) that the object (20, 20a, 20b) associated augmented reality information (23), and the computing device (21) being designed to display a movement parallax of the movement parallax on the display device (17, 17a, 17b) resulting from a change in an observer's position and / or an object position for the observer ( 12) to determine the displayed augmented reality information (23) and to compensate for the parallax of movement.

Description

The present invention relates to a device for displaying augmented reality information according to the preamble of claim 1.

The present invention also relates to a means of transport comprising a device for displaying augmented reality information, an information system comprising a device for displaying augmented reality information and a method for displaying augmented reality information.

Technological background

The term “augmented reality” is understood to mean a computer-aided display of information in which the real world is expanded in real time by context-dependent virtual components. These can appear in various forms, such as two- or three-dimensional text information or two- or three-dimensional images. A user's perception is influenced in such a way that he gets the impression that the virtual components are actually in the real world.

From the US 2017/0294046 A1 a system and a method for expanding a reality viewed by a display device are known. The device includes a storage unit, a display position decision unit and a display device controller. The display position decision unit determines a display position of augmented reality information on the display device. The display device is designed as a transparent display. The display position is determined on the basis of an actual position of a user, an actual position of the display device and an actual position of an area visible by the display device to which the augmented reality information is assigned. The display device controller controls the display device to display the augmented reality information taking into account a spatial relationship between the actual position of the user and the actual position of the area visible through the display device to which the augmented reality information is assigned.

the WO 2013/170303 discloses a display system comprising a container having an inner compartment for a product and at least one screen, a portion of the screen enabling visual inspection of the product in the inner compartment through the screen. A camera is also provided which is positioned such that it detects a person in the vicinity of the display system.

the WO 2015/077766 A1 discloses systems and methods for augmenting reality. A traffic sign recognized along a travel route is recognized and identified, and it is determined whether the sign is relevant. On a display device, the viewed real-world scene, which contains the traffic sign, is expanded with augmented reality information in order to emphasize or minimize the meaning of the traffic sign.

From the DE 10 2017 214 225 B3 a method for operating a driver assistance system for a motor vehicle while the motor vehicle is in motion is known, in which the driver of the motor vehicle is shown a virtual vehicle on at least one display element, the virtual vehicle being used to support the driver with regard to its size and / or position and / or its appearance on the display element is varied while the virtual vehicle is displayed on the display element.

So-called augmented reality head-up displays (HUDs) are known from the prior art for use in motor vehicles, in which augmented reality information is displayed in the field of vision of a driver of a motor vehicle. Such head-up displays are intended to provide the driver with information while driving, without the driver having to take his eyes off the road. In the known head-up displays, the information is usually projected onto the windshield, as a result of which the information appears to float in front of the background viewed through the windshield. This type of presentation of information can be irritating for the driver and lead to an increased distraction.

Furthermore, digital devices for displaying advertisements which are set up in urban areas are known in the prior art. Images or short advertising films, for example, are played on such devices. Such devices can be perceived as annoying due to their size and the associated restriction of the field of view.

Presentation of the invention: task, solution, advantages

The present invention is based on the object of providing a device for displaying augmented reality information, in which augmented reality information is displayed in such a way that it changes perfectly, coupled with the movement of an observer can be adapted to the environment, so that the observer gets an impression of depth and preferably of multidimensionality of the augmented reality information presented.

Furthermore, the invention is based on the object of providing a means of transport with a device for displaying augmented reality information, an information system with a device for displaying augmented reality information and a method for displaying augmented reality information, with which the aforementioned problem is solved.

To solve the problem on which the invention is based, a device for displaying augmented reality information is proposed, the device being a display device with a degree of transparency greater than 0%, a first detection device directed into a first half-space in front of the display device, the first detection device for detection of data suitable for determining an observer position of an observer in the first half-space relative to the display device, a second detection device directed into a second half-space behind the display device, the second detection device for detecting for determining an object position of an object in the second half-space relative to the Display device suitable data is formed, and comprises a computing device, wherein the computing device is designed to determine the observer position relatively z u of the display device, wherein the computing device is designed to determine the object position relative to the display device from the data captured by the second detection device, the computing device being designed to select augmented reality information and the object the augmented reality information and to determine a display position of the augmented reality information on the display device based on the observer position and the object position, the computing device being designed to instruct the display device to display the augmented reality information assigned to the object at the display position, further provided is that the computing device is designed to determine a movement parallax of the augmented reality information displayed on the display device resulting from a change in the observer position and / or the object position for the observer and to determine the movement Compensate for parallax by updating the display position.

Another solution to the problem on which the invention is based is to provide a means of transport, in particular a motor vehicle, rail vehicle, watercraft or aircraft, comprising a device for displaying augmented reality information. Yet another solution to the object on which the invention is based is to provide an information system comprising a holding device and a device for displaying augmented reality information.

The device according to the invention comprises a display device with a degree of transparency greater than 0%. In particular, the device has a degree of transparency of greater than 0% in the optical spectrum, so that objects arranged behind the display device are at least partially visible to an observer in front of the display device through the display device.

According to the present invention, it is therefore provided that the representation of the augmented reality information takes place in a so-called optical-based see-through method, unlike in the video-based-see-through method known from the prior art. The video-based see-through approach is often used with smartphones or tablets where the user cannot see through the device. Instead, a camera integrated there records the environment and places a video of the environment covering the entire area on the display surface of the device. Augmented reality information is then inserted into this video depending on the position and orientation and also displayed. In contrast to this, in the case of the optical-based see-through method, an observer perceives the surroundings with his eyes through a transparent display device. Only the augmented reality information is shown on the display device as a function of position and orientation and is thus automatically integrated into the directly perceived outside world.

The display device has a front side and a rear side, with augmented reality information displayed on the display device being visible at least from the front side. An imaginary parting plane arranged between the front side and the rear side divides the space surrounding the display device into two half-spaces, a first half-space in front of the display device and a second half-space behind the display device. The first detection device is directed into the first half-space. This means that the first display device has data relating to the first half-space can capture. It can be provided that the first display device only records a partial area, a partial volume or a partial segment of the first half-space. The second detection device is directed into the second half-space. This means that the second display device can acquire data relating to the second half-space. It can be provided that the second display device only records a partial area, a partial volume or a partial segment of the second half-space.

Preferably, more than one first detection device and / or more than one second detection device can also be provided.

The computing device can be designed, for example, as a personal computer (PC) or as a mini-PC. The computing device can also be designed as a modular computing device with several, in particular physical, computing modules, the totality of the computing modules forming the computing device. For example, a first computation module of the computing device can be part of the first acquisition device and be designed to determine the observer's position of the observer relative to the display device from the data acquired by the first acquisition device. Furthermore, a second computation module of the computing device can be part of the second acquisition device and be designed to determine the object position of the observer relative to the display device from the data acquired by the second acquisition device. A third module of the computing device can be designed as a mini PC or as part of a mini PC and designed to select the augmented reality information, to determine the display position of the augmented reality information on the display device, to determine the parallax of movement and update the display position. It is also possible that parts of the computing device are designed as an external computing device, or are made available as computing power provided in a cloud.

A movement parallax arises when an observer or his surroundings move, with the distance and speed being reciprocally proportional to each other.

According to the invention it is now provided that an observer from the observer position in the first half space in front of the display device can view an object in the second half space behind the display device through the at least partially transparent display device and that augmented reality information assigned to the object is displayed on the display device. Since the augmented reality information is arranged in a different spatial plane than the object viewed by the display device, movement parallax can occur if the observer position or the object position changes relative to the display device. According to the invention, the computing device now determines the resulting movement parallax for the observer from a change in the observer position and / or the object position and compensates for the movement parallax by updating the display position of the augmented reality information on the display device.

The device according to the invention for displaying augmented reality information can be used in many technical areas. In the following, the invention is explained in a non-restrictive manner on the basis of the use of the device for displaying augmented reality information in a means of transport, in particular in a motor vehicle, a rail vehicle, a watercraft or an aircraft, and on the basis of the use in an information system with a holding device , wherein the holding device is a piece of street furniture, a display or a window pane of a building.

The invention can be used in any suitable means of transport, for example in cars, trucks, agricultural machines, motorized two- and three-wheelers, ships, trains, aircraft, construction machinery, etc., in order to provide a user of the means of transport with augmented reality information in a kind and Way to indicate that these appear seamlessly inserted into the environment viewed through the display device. It can thus be particularly preferably provided that the device is used to display extended reality information in a passenger vehicle. The display device of the device for displaying augmented reality information is then particularly preferably arranged in or on a windshield and / or in or on a side window of the passenger vehicle.

It is well known that distraction at the wheel is the leading cause of road accidents. Every second driver uses a navigation system while driving, the information of which is displayed either on an external device, such as a smartphone, or on an internal display inside the vehicle. In any case, the driver has to look away from the road to view the navigation information and focus on the display. In the time required for this, the vehicle can cover considerable distances, during which the driver cannot fully concentrate on the road traffic.

Head-up displays known in the prior art project vehicle information or navigation information onto the windshield. In such systems, however, the information appears to float in front of the street scene. When the driver moves inside the motor vehicle or when the motor vehicle itself is in motion for the information projected onto the windshield, a movement parallax which cannot be compensated for in known head-up displays. In addition, in known head-up displays, the solid angle from which the information projected onto the windshield is visible is very small. Therefore, when the driver changes, the usually very expensive lens optics have to be adjusted to the new driver. Furthermore, known head-up displays require a considerable amount of space in the vehicle.

When the device according to the invention is used in a passenger car, a driver can be protected from being distracted at the wheel by displaying important information, preferably all important information, for example navigation information, directly in the driver's field of vision, and an object in the outside environment as augmented reality. Information can be assigned. The navigation information displayed can be assigned to a specific lane, a motorway exit or a traffic sign, for example. Since movement parallax is compensated for, the information does not float in front of the background, but from the driver's point of view appears to be seamlessly integrated into the road. Since the information is also not projected onto a windshield, the augmented reality information is visible from a much larger angular range and preferably from all positions within the first half-space, in contrast to known head-up display systems. In particular, there is no need to adjust lens optics when the driver changes.

If the device is used to display augmented reality information in an information system comprising a holding device, the display device being attached to the holding device, the information system can be designed, for example, as a device for displaying advertising information. The holding device can preferably be designed as a shop window pane on or in which the display device is arranged.

It is known to use so-called digital out-of-home advertising (DOOH) in shopping centers, at bus stops, at train stations, airports, on streets, in cinemas or also on shop windows or within shops for customer loyalty and customer acquisition. It is estimated that 71% of DOOH advertisements are viewed by people, but only 13% of observers are encouraged to buy the products presented.

Different target groups also perceive advertising and the environment fundamentally differently, pay attention to different details or are sometimes stimulated to buy a product by different stimuli. For this reason, not all people can be reached with a clearly defined advertising.

More recently, digital displays have been used for outdoor advertising, on which images, video images in 2D format or animations can be displayed. Such displays often have larger dimensions and can obscure the view of local shops, which is disadvantageous for the shops.

When using the device according to the invention for displaying augmented reality information in an information system designed as a device for displaying advertising, advertising information tailored to an observer can be displayed on the display device. At the same time it is possible for the observer to look through the display device and to recognize objects behind the display device. The impairment of the viewing area is thus minimized. At the same time, the advertising information for the observer can be integrated into his field of vision in a visually appealing way, since a movement parallax of the advertising information that is caused by the movement of the observer is compensated for. The advertising information appears to the observer as augmented reality information inserted in the background.

A storage device is advantageously provided, a large number of augmented reality information items being stored in the storage device.

More preferably, the computing device is designed to download the selected augmented reality information from the storage device.

The augmented reality information displayed on the display device can be selected by the computing device from a large number of augmented reality information on an observer, context or situation-dependent basis and displayed on the display device. The storage device can be a local storage device, for example the hard drive of a personal computer, an external hard drive or a storage medium such as a USB stick. The storage device can, however, also be a decentralized storage device on the Internet, such as a so-called cloud storage device.

The augmented reality information can include texts, images, two-dimensional objects or 3D objects, as well as image sequences or animations. The augmented reality information can be static or moving, especially animated. The augmented reality information can be pictures or videos.

If the device is used in a means of transport such as a motor vehicle, the augmented reality information can be, for example, navigation symbols, arrows, instructions or information on traffic signs, information from an external device or multimedia system, vehicle information relating to the speed, the vehicle status or Act information on the formation of a rescue alley, etc.

If the device is used in an information system such as a device for displaying advertising, the augmented reality information can be videos, images, texts with advertising messages or two- or three-dimensional representations of products.

It is preferably provided that the augmented reality information from the observer position displayed on the display device has an essentially constant spatial relationship to the object.

In other words, it is preferred that, from the observer's point of view, the augmented reality information has an apparent spatial relationship to an object that is visible to the observer through the display device. The augmented reality information does not necessarily have to appear at the location of the object from the observer's point of view.

However, it is fundamentally preferred that, from the observer's point of view, the augmented reality information appears spatially close to the location of the object seen by the display device.

In the case of a means of transport with a device for displaying augmented reality information, the augmented reality information can be embodied as navigation symbols, for example, and displayed on the display device in such a way that they appear to float above the lane for the driver. For example, direction arrows can be displayed on a lane when the navigation system recommends a turning maneuver. In addition, it is possible that the augmented reality information is information about traffic signs. From the driver's point of view, the augmented reality information can then be displayed in the immediate vicinity next to the traffic sign visible by the display device, quasi at the location of the traffic sign. However, the augmented reality information can also have a constant spatial relationship with regard to the engine hood visible through the windshield and thus appear to float in front of the vehicle at a constant distance and appear to be seamlessly inserted into the surroundings.

In the case of an information system with a device for displaying augmented reality information, the augmented reality information can be, for example, a 3D representation of an advertised product. The object can be, for example, a base arranged behind the display device and visible through the display device. The 3D representation of the product can then be positioned on the display device from the point of view of the observer in such a way that the virtual representation of the product appears to be on the base.

With a further advantage it can be provided that the essentially constant spatial relationship of the displayed augmented reality information to the object comprises an essentially constant distance between the displayed augmented reality information and the object.

The essentially constant distance is then an essentially apparent constant distance.

In addition, it can be provided that the essentially constant spatial relationship of the displayed augmented reality information to the object comprises an essentially constant spatial orientation of the displayed augmented reality information to the object.

The essentially constant spatial alignment is then an essentially apparent constant spatial alignment.

In other words, it can be provided that the augmented reality information displayed on the display device is inserted into the scenery visible by the display device from the viewpoint of the observer, as if the augmented reality information had a fixed orientation with respect to an object behind the display device has. Furthermore, the displayed augmented reality information can be adjusted in perspective to the movement of the observer, so that the augmented reality information always appears to be in a seemingly fixed location.

If the device is used in a motor vehicle, the augmented reality information can be rotated or moved in perspective from the perspective of the observer while the motor vehicle is moving through the landscape. In this way, especially if the augmented reality information is a 3D object, the impression can be created that that the augmented reality information is a natural part of the environment.

If the device is used in an information system for displaying advertising, it can be provided that the augmented reality information is a 3D representation of a product. The three-dimensional virtual representation of the product can, for example, appear to be arranged for an observer on a base that is visible through the display device. If the three-dimensional representation of the product has a substantially constant spatial orientation to the base, the observer can view the advertised product from different angles by moving the observer relative to the display device.

With a further advantage it can be provided that the observer position comprises or is a head position of the observer.

If the observer's position is a head position of the observer, a movement parallax with respect to a head movement of the augmented reality information displayed on the display device can be compensated for.

In addition, it can be provided that the observer position includes an eye position and / or a viewing direction.

It is preferably provided that the computing device is designed to control the observer position and / or the change in the observer position and / or the object position and / or the change in the object position by means of artificial intelligence methods and / or machine learning methods and / or neural networks and / or to be determined by means of Haar cascades and / or Kalman filters and / or optical flow methods.

In particular, any suitable method that can be carried out by a computing device can be provided for determining the observer position and / or the change in the observer position and / or the object position and / or the change in the object position.

Furthermore, the display device can be a screen, in particular a display, further in particular a film display.

The display device is particularly preferably a film display with a degree of transparency of greater than 0%.

The film display can be designed to be flexible and can have different shapes. For example, the display can be circular, rectangular, square, oval or triangular.

When the device is used in a motor vehicle, the film display can be attached to a front or windshield or to a side window of the motor vehicle.

When used in an information system for displaying advertising, the film display can be arranged on a window pane or on a shop window pane of a building, in particular a shop.

It is preferably provided that the display device is an LED display device, in particular an OLED display device or an OLED display device.

With a further advantage it can be provided that the display device has a thickness of less than 10 mm, preferably less than 5 mm, particularly preferably less than 2 mm.

Such display devices, in particular designed as a film display, can be used particularly advantageously in means of transport and / or in information systems.

It is preferably provided that the display device has a degree of transparency of more than 10%, preferably more than 20%, particularly preferably more than 40%, more particularly preferably more than 50%.

In addition, it can advantageously be provided that the brightness of the display device can be set by the computing device, in particular as a function of the prevailing light and weather conditions.

Thus, the observer can see a clear image on the display device at all times.

The display device can preferably be arranged on a glass pane or a window pane.

When using a device for displaying augmented reality information in a means of transport such as in a motor vehicle, a display device configured as a film display can be integrated, for example, into the laminated glass of a windshield or a side window. The film display can be attached to the windshield inside the vehicle or outside the vehicle. The film display can either cover the entire area or only a partial area of the windshield or a side window. The membrane display can be used with the Installation by the windshield manufacturer during the manufacturing process of the windshield in the windshield itself or fixed by the automobile manufacturer on the inside of the windshield or on the outside of the windshield by the customer.

The foil display can be electrostatically attached, glued, pressed, treated with heat, chemically treated, screwed or connected with magnets for connection to the windshield or side window.

In the case of a stable display device provided with a special coating, the windshield or the side window can be dispensed with and the display device can be installed as a replacement for the windshield or for the side window.

When using the device for displaying augmented reality information in an information system such as a device for displaying advertising, a display device designed as a film display can be installed in a shop window by the window manufacturer or by a shop operator or center management from inside or outside attached to the shop window. The foil display can be electrostatically attached, glued, pressed, treated with heat, chemically treated, screwed or connected with magnets for connection to the shop window. In the case of a stable display device provided with a special coating, the shop window pane can be dispensed with and the display device can be installed as a replacement for the shop window pane.

It is particularly advantageous that the first detection device and / or the second detection device have a camera, in particular an infrared camera, a stereoscopic camera, a monochrome camera, a color camera, a depth camera, preferably a time-of-flight camera or a structured light -Camera, and / or that the first detection device and / or the second detection device comprises sensors, in particular ultrasonic sensors, position sensors or light barriers.

When using the device in a means of transport, in particular in a motor vehicle, the first detection device, in particular the first camera, is preferably directed inwards into the passenger compartment. With the first detection device, in particular the first camera, the observer position, in particular the head position, of the driver can be detected, in particular recorded. The observer position and in particular the head position of the driver can be determined from the data. The position of the first detection device, in particular the first camera, can be selected relatively freely in the passenger compartment. Depending on the customer's wishes, the vehicle manufacturer can install the first detection device in the interior of the means of transport, in particular the motor vehicle, as long as it is ensured that the head and in particular the face of the driver can be detected.

The current road situation is recorded with the second recording device, in particular the second camera, and road markings and traffic signs can be recorded, for example. The object position of the recorded markings and signs can be determined from the data. With the second detection device, in particular with the second camera, the positioning is relatively arbitrary when used in a means of transport, as long as the, preferably entire, street events can be detected.

When used in an information system, in particular in a device for displaying advertising, the current observer position, in particular the head position, of an observer can be continuously recorded, in particular recorded, with the first recording device, in particular the first camera. With the aid of the data supplied by the first detection device, in particular the first camera, the observer's position and in particular the observer's head position can be determined. When using the device for displaying augmented reality information in an information system, the position of the first detection device directed towards the observer, in particular the first camera, can be selected relatively freely. A shopkeeper or a center management can mount the camera or the first detection device as desired in such a way that it is ensured that the face of the observer can be recorded. When used in an information system, the current events in the environment are recorded, in particular recorded, with the second recording device, in particular the second camera, which is visible to an observer in front of the display device through the display device. The position for attaching the second detection device, in particular the second camera, is also relatively free as long as the, preferably entire, surroundings behind the display device can be detected.

The first detection device and / or the second detection device, in particular the first camera and / or the second camera, has a high clock frequency and / or a resolution of at least HD, preferably at least 1920 x 1080 pixels.

The computing device is preferably designed to use artificial intelligence methods and / or machine learning methods to determine observer information on the observer from the data of the first detection device and / or to determine object information on the object from the data of the second detection device, the Computing device is preferably designed to select an augmented reality information on the basis of the observer information and / or the object information.

The observer information is preferably determined and / or used as personal but anonymized data.

If the device is used, for example, in a means of transport such as a motor vehicle, information about the driver of a motor vehicle can be determined from the data from the first detection device, in particular the first camera. For example, the driver can be monitored for drowsiness. With the second detection device, in particular the second camera, object information on the road condition, on the lane held by the vehicle, on road signs and other objects in the area of the road can be determined. In particular, the significance of the objects for the driver is determined as object information. Using the object information on the objects, corresponding augmented reality information can then be selected by the computing device and displayed on the display device.

When the device is used in an information system, observer information on an observer, that is to say a potential customer, can be determined from the data determined by means of the first detection device, in particular the first camera. For example, it can be determined as observer information whether the observer is an animal or a person. Furthermore, the sex, age, stature, skin color, hairstyle and / or the mood of the observer can be determined. In addition, external characteristics such as wearing glasses, headgear or the presence of facial hair or the type of clothing can be determined as observer information. With the second detection device, in particular the second camera, data can be determined which are used by the computing device to determine object information. The object information can be, for example, objects such as logos, brand names or objects such as table, chairs, etc., which are visible through the display device.

Taking into account the object information and / or the observer information, suitable augmented reality information can then be selected by the computing device, which is displayed on the display device.

Particularly when used in an information system for displaying advertising information, suitable advertising information can be selected as augmented reality information, taking into account the observer information relating to gender, age, mood of the observer or external characteristics.

Another solution to the problem on which the invention is based consists in a means of transport, in particular a motor vehicle, rail vehicle, watercraft or aircraft, comprising a device as described above for displaying augmented reality information.

All of the features, configurations and functions explained for the device described above for displaying augmented reality information can also be transferred analogously to the means of transport according to the invention.

The means of transport is particularly preferably designed as a motor vehicle and more particularly as a passenger vehicle, an electric vehicle, a battery-electric vehicle or a hybrid electric vehicle.

In particular, the motor vehicle can be designed as an autonomous motor vehicle.

However, it is also possible for the means of transport to be designed as a ship, rail vehicle, aircraft, construction machine, agricultural machine, truck, etc.

However, the means of transport is particularly preferably designed as a motor vehicle. It is preferably provided that the display device of the device for displaying augmented reality information is arranged on a window, in particular on a windshield and / or a side window, of the motor vehicle.

Provision is particularly preferably made for the means of transport to include a navigation system, the computing device being designed to select augmented reality information on the basis of navigation information from the navigation system.

The augmented reality information preferably includes 3D objects, for example direction arrows, which are displayed on the display device attached to a windshield in such a way that, from the driver's point of view, they always remain in the same place with respect to the lane so that the driver can perceive the navigation information You don't have to take your eyes off the road.

Furthermore, it can preferably be provided that the augmented reality information relates to traffic signs.

For example, augmented reality information can be displayed on the display device in such a way that, from the perspective of the driver or observer, it appears to appear in the immediate vicinity and in a constant spatial relationship to a traffic sign perceived by the display device in the outside environment. The augmented reality information can relate to the meaning of the traffic signs and include, for example, explanations of the traffic signs. Furthermore, the augmented reality information can be designed in such a way that it emphasizes or lowers the importance of certain traffic signs for the driver. For example, the augmented reality information can be images of traffic signs. The images of the traffic signs can then be enlarged so that they appear at the position of the actual traffic signs. The meaning of the traffic signs can be emphasized by the enlargement. In addition, the meaning of the recognized traffic signs can be displayed in the form of two- or three-dimensional symbols or texts on the display device.

In addition, information on the vehicle speed, the vehicle status or information from a multimedia system can be displayed on the display device of the device installed in the motor vehicle for displaying augmented reality information.

It is provided with a further advantage that the augmented reality information relate to information for the formation of an emergency lane.

If, for example, a vehicle system, for example by integrating a navigation system with a TMC system or a real-time traffic information system and / or by analyzing the recorded data of the second detection device, determines the need to create an emergency lane, the driver can receive information on the display device in the form of are given by arrows, indicated driving lines or lanes or the like, how he has to behave in order to form an emergency lane.

The device for displaying augmented reality information of the means of transport particularly advantageously has an interface to a, preferably portable, data processing device and / or to a multimedia system.

The portable data processing device can be a cell phone, a tablet, a laptop or the like. In particular, navigation software can be executed on a cell phone, tablet, etc., the navigation information of which is transmitted via the interface of the computing device and on the basis of which suitable augmented reality information is selected and displayed on the display device.

The computing device is preferably designed to find an external portable data processing device and to automatically connect to the external portable data processing device.

If the motor vehicle is also an autonomously driving vehicle, it can be provided that on the display device, which is arranged, for example, on a windshield or a side window of the motor vehicle, information or data or images from a device connected to the device via the interface portable data processing device are displayed. This data can contain films, pictures, etc., for example. For example, a business woman or a businessman can create a presentation in the autonomous vehicle while driving on the display device or work with word processing software. The advantage is that the person can look straight ahead at the horizon while working, since the display device has a degree of transparency of greater than 0%.

Another solution to the problem on which the invention is based is to provide an information system comprising a holding device and a device as described above for displaying augmented reality information, the display device of the device for displaying augmented reality information being attached to the holding device, with It is further provided that the holding device is a piece of street furniture or a stand or a window pane of a building.

For example, the holding device can be a bus stop or a stand in a waiting area of a local public transport system. It is also possible that the Holding device is a stand such as an advertising column or the like. Furthermore, it can be provided that the holding device is a window pane in a building, in particular a shop window pane in a shop. In addition, it can be provided that the holding device is a display in a museum, at a concert venue, in a zoo, etc. Augmented reality information on the exhibits, the demonstration, the animals, etc. can be transmitted to a museum visitor, a concert visitor, a zoo visitor, etc. via the display device attached to the holding device.

However, it is particularly preferably provided that the holding device is a shop window pane.

All of the features, configurations and functions explained for the device described above for displaying augmented reality information can also be transferred analogously to the information system according to the invention.

Yet another solution to the problem on which the invention is based is to provide a method for displaying augmented reality information on a display device, the display device having a degree of transparency greater than 0%, with an observer being arranged in a first half-space in front of the display device, wherein data suitable for determining an observer's position in the first half-space relative to the display device are captured by a first detection device directed into the first half-space, an object being arranged in a second half-space behind the display device Object in the second half-space relative to the display device suitable data can be detected by a second detection device directed into the second half-space, the data detected by the first detection device being used by a computing device Observer position is determined relative to the display device, and wherein the object position relative to the display device is determined by the computing device from the data captured by the second detection device, and augmented reality information is selected by the computing device and assigned to the object, from The computing device determines a display position of the augmented reality information on the display device on the basis of the observer position and the object position, the display device being instructed by the computing device to display the augmented reality information assigned to the object at the display position, with further provision being made that the computing device determines a movement parallax of the augmented reality information displayed on the display device resulting from a change in the observer position and / or the object position for the observer, and the movement parallax e compensates for by updating the display position.

All of the features, configurations and functions explained in connection with the above-described device for displaying augmented reality information can also be transferred in an analogous manner to the method for displaying augmented reality information.

Figure list

• 1 eine Vorrichtung zur Darstellung von Erweiterte-Realität-Informationen in einem als Kraftfahrzeug,
• 2 einen erstes Anwendungsbeispiel einer Vorrichtung zur Darstellung von Erweiterte-Realität-Informationen in einem als Kraftfahrzeug,
• 3 einen zweites Anwendungsbeispiel einer Vorrichtung zur Darstellung von Erweiterte-Realität-Informationen in einem als Kraftfahrzeug,
• 4 einen drittes Anwendungsbeispiel einer Vorrichtung zur Darstellung von Erweiterte-Realität-Informationen in einem als Kraftfahrzeug,
• 5 eine erstes Informationssystem mit einer Vorrichtung zur Darstellung von Erweiterte-Realität-Informationen, und
• 6 eine zweites Informationssystem mit einer Vorrichtung zur Darstellung von Erweiterte-Realität-Informationen.

The invention is explained in more detail below with reference to the accompanying figures. Show it:

• 1 a device for displaying augmented reality information in a motor vehicle,
• 2 a first application example of a device for displaying augmented reality information in a motor vehicle,
• 3 a second application example of a device for displaying augmented reality information in a motor vehicle,
• 4th a third application example of a device for displaying augmented reality information in a motor vehicle,
• 5 a first information system with a device for displaying augmented reality information, and
• 6th a second information system with a device for displaying augmented reality information.

Detailed description of the figures

1 shows a device 100 for displaying augmented reality information in a motor vehicle 10 trained means of transport 200 . The device 100 has one as a foil display 11th trained display device 12th with a degree of transparency of more than 50%. The foil display 11th comprises a high-resolution pixel matrix and is on or in the windshield 13th of the motor vehicle 10 attached. The device further comprises 100 one as the first camera 14th trained first detection device 15th , which in the passenger compartment 16 in front of the foil display 11th is directed. Using the first camera 14th data are recorded which are used to determine the observer's position, in particular the head position and the eye position, of an observer 17th , in the case shown of a driver 17a , inside the motor vehicle 10 are suitable. Furthermore, the device 100 one as a second camera 18th trained second detection device 19th on which in the external environment of the motor vehicle 10 is directed, and collects data which are used to determine the object position of an object 20th , in the case shown here of a traffic sign 20a , outside the motor vehicle 10 are suitable. The first detection device 15th and / or the second detection device 19th can either be inside or outside the motor vehicle 10 or the passenger compartment 16 be arranged.

The device 100 also includes a mini PC 21a trained computing device 21 . The computing device 21 determined from the data from the first camera 14th the observer position relative to the foil display 11th and from the data from the second camera 18th the object position relative to the foil display 11th . The computing device is used to determine the observer position and the object position 21 running appropriate software. In addition, on the computing device 21 an artificial intelligence software is executed, which is based on the data from the first camera 14th additional observer information and from the data from the second camera 18th additional object information determined. The computing device selects on the basis of the observer information and the object information 21 from a plurality of in a storage device 22nd The stored augmented reality information is augmented reality information 23 and shows the membrane display 11th on, the augmented reality information 23 at a display position 24 to display. From the observer's point of view 17th or driver 17a seems the apparent picture 23a the augmented reality information in front of the motor vehicle 10 to float.

The device 100 further comprises an interface 25th , for example a Bluetooth interface, for an external data processing device 26th . The data processing device 26th is one of the driver in the exemplary embodiment 17a carried smartphone 26a that is running navigation software.

2 shows the driver's field of vision 17a from the passenger compartment 16 through the one on the windshield 13th of the motor vehicle 10 attached foil display 11th . The driver 17a can through the semi-transparent foil display 11th the street 27 and on the street 27 vehicles driving in front of him 28a , 28b , 28c , 28d recognize. There is also one on the side of the road 29 arranged traffic sign 20a through the foil display 11th for the driver 17a visible which of the computing device 21 as an object 20th is recognized. The computing device 21 the device 100 to display augmented reality information is initially determined from the data from the first camera 14th ( 1 ) the driver's head position or eye position 17a relative to the foil display 11th . In addition, the computing device determines 21 further observer information about the driver using the artificial intelligence software 17a . In the application example shown, the computing device could 21 for example, find that the driver 17th Shows signs of drowsiness. The computing device 21 also determined from the data from the second camera 18th ( 1 ) the object position of the second camera 18th detected traffic sign 20a relative to the foil display 11th . In addition, the computing device determines 21 the meaning of the traffic sign using the artificial intelligence software 20a , which concerns a speed limit in the present case. This object information is shared with the observer information from the computing device 21 further processed, which comes to the conclusion that there is a risk that the driver 17a the traffic sign for the prevailing traffic density 17a could be overlooked.

The computing device 21 therefore selects from the storage device 22nd ( 1 ) an augmented reality information 23 from which in the present case a framework 30a with a written notice 30b on the meaning of the traffic sign 20a is. The computing device 21 then determined on the basis of the previously determined head position and the object position, i.e. the position of the traffic sign 20th beside the road 29 , a display position 24 ( 1 ) on the membrane display 11th for augmented reality information 23 such that the frame 30a from the driver's point of view 17a , as in 2 shown the traffic sign 20a seems to border. Together with the written notice 30b thus becomes the traffic sign 20a for the driver 17a highlighted.

While the motor vehicle 10 the street 27 drives along, the object position of the traffic sign changes 20a relative to the foil display 11th of the motor vehicle 10 . The driver also moves 17a inside the motor vehicle 10 so that at least the head position or the eye position is relative to the foil display 11th can change. With a fixed display position 24 of augmented reality information 23 on the foil display 21 the changes in the object position and the head position from the driver's point of view 17a to a motion parallax for the augmented reality information 23 lead, that is, the framework 30a from the driver's point of view 17a from the traffic sign 20a move away and after a short time the traffic sign 20a would no longer frame. The computing device 21 is therefore designed to continuously determine this movement parallax resulting from the changes in the observer position and the object position and by updating display position 24 of augmented reality information 23 on the foil display 11th continuously balance so that the frame 30a from the driver's point of view 17a the traffic sign at all times 20a seems to border.

In 3 is a second example of application of the device 100 shown to represent augmented reality information. The device 100 received in the application example via the interface 25th from the one on the smartphone 26a ( 1 ) running navigation software navigation information. The computing device selects based on the navigation information 21 from the storage device 22nd an augmented reality information 23 which in the application shown is a directional arrow 31a with a written notice 31b , "5 km straight ahead". The computing device 21 determined on the basis of the determined head position of the driver 17a and the object position, which in the application shown is the position of the lane markings 32 is a display position 24 ( 1 ) on the membrane display 11th for augmented reality information 23 such that the direction arrow 31a from the driver's point of view 17a always in the middle of the traveled lane 27a the street 27 seems to be arranged. Relative movements of the motor vehicle 10 relative to the lane being traveled 27a or movements of the driver's head 17a which from the driver's point of view 17a to a movement parallax of the direction arrow 31a regarding the street 27 could result by updating the display position 24 of the direction arrow 31a on the foil display 11th by the computing device 21 balanced so that the direction arrow 31a from the driver's point of view 17a always in the middle between the lane markings 32 seems to be arranged.

Yet another example of the application of the device 100 for the representation of augmented reality information is in 4th shown. The computing device 21 determines from traffic information received from a real-time traffic information system that an emergency lane is to be formed. The computing device 21 therefore selects from those on the storage device 22nd The stored augmented reality information is augmented reality information 23 from what guidelines 33a , a direction arrow 33b and a written notice 33c include that of assisting the driver 17a serve in the formation of the rescue alley. As in the application examples described above, the computing device updates 21 the display position 24 of augmented reality information 23 on the foil display 11th such that the guidelines 33a , the direction arrow 33b and the written notice 33c from the driver's point of view 17a always in a fixed spatial relationship to the street 27 and especially to the road markings 32 seem to be arranged.

In the 5 becomes a further embodiment of a device 100 for displaying augmented reality information. The device 100 is as part of an information system 300 trained for urban areas. The information system 300 includes one as a frame 34 trained holding device 35 . The display device 12th with a degree of transparency of more than 50% is within the scope 34 held. One in front of the information system 300 standing observer 17th can through the semi-transparent display device 12th through objects 20th behind the display device 12th see. One on the observer 17th directed first camera 14th which at a front 36 of the frame 34 of the information system 300 is attached, represents a computing device 21 of the information system 100 Data available which are suitable for the observer position relative to the display device 12th to determine. One at a back 37 of the frame 34 attached second camera 18th collects data on objects 20th which from the observer's point of view 17th behind the information system 300 condition. In the example it is the object 20th around a pedestal 20b . The computing device 21 determined from the data from the first camera 14th and the second camera 18th the observer position relative to the display device 12th and the object position relative to the display device 12th . Furthermore, by means of a on the computing device 21 executed artificial intelligence software determines additional observer information and object information. For example, the computing device 21 using the artificial intelligence software to determine that it is the observer 17th a female observer 17b of a certain age group and that the observer 17b is in a positive mood. The computing device selects on the basis of this observer information 21 from a plurality of on a storage device 22nd stored augmented reality information about the determined mood and age group of the observer 17b appropriate extended reality information 23 which in the present example is advertising information. The advertising information is a 3D representation 38 of a product to be advertised. From the object position of the base 20b relative to the display device 12th and from the observer's position 17b relative to the display device 12th determines the computing device 21 a display position 24 for augmented reality information 23 on the display device 12th so that the apparent picture 23a the 3D representation 38 of the product from the observer's point of view 17b on which by the display device 12th visible base 20b is arranged. Moves the observer 17b relative to the display device 12th so this will change the Observer position, which leads to a movement parallax with regard to the 3D representation 38 of the product would result from the computing device 21 determined and by updating the display position 24 the 3D representation 38 of the product on the display device 12th balanced. From the observer's point of view, the apparent image remains 23a the 3D representation 38 of the product, regardless of the observer's position, always on the through the display device 12th visible base 20b arranged. When updating the display position 24 fits the computing device 21 also the 3D representation 38 in perspective, so that with the observer 17b the impression is given that the 3D representation 38 can be viewed from several sides if the observer 17b to the side of the display device 12th emotional. In addition, the 3D representation 38 of the advertised product according to the distance of the observer 17b from the display device 12th scaled so that the 3D representation 38 gets bigger when the observer 17b the display device 12th approaching.

Another application example is in 6th shown. The information system 300 the 6th include a device 100 for displaying augmented reality information essentially corresponds to the information system 300 the 5 , but with the difference that the holding device 35 of the information system 300 as a shop window 39 of a retail store 40 educated. The display device 12th is as a foil display 11th trained and on or in the shop window 39 attached. An observer 17b can through that on the shop window 39 attached semi-transparent foil display 11th the interior of the store 40 consider. From the data from a first camera 14th determines a computing device (not shown) 21 an observer position and further observer information about the observer 17b in front of the store 40 . From the data one into the interior of the shop 40 directional second camera, not shown 18th become the object positions of objects 20th inside the store 20th definitely. In addition, further object information can be determined. For example, the computing device 21 from the data from the second camera 18th in the store 40 Affixed logos of manufacturers or the product categories in the store 40 determine the products offered. Analogous to the embodiment of 5 definitely selects the computing device 21 from a plurality of on a storage device 22nd Stored augmented reality information is augmented reality information that matches the observer information and the object information 23 from, which is also a 3D representation 38 of a product can be. The computing device determines from the observer position and the object position or positions 21 a display position 24 of augmented reality information 23 such that for the observer 17b the impression arises that the 3D representation 38 the product is actually in the store 40 is located. By updating the display position 24 on the foil display 11th the parallax of movement is compensated. In addition, a spatial impression is created by a perspective adaptation and scaling of the 3D representation that is dependent on the change in the observer's position 38 of the product.

List of reference symbols

100

Device for displaying augmented reality information

200

Mode of Transport

300

Information system

10

Motor vehicle

11th

Foil display

12th

Display device

13th

Windshield

14th

First camera

15th

First detection device

16

Passenger compartment

17th

observer

17a

driver

17b

Observer

18th

Second camera

19th

Second detection device

20th

object

20a

Traffic signs

20b

base

21

Computing device

21a

Mini pc

22nd

Storage device

23

Augmented Reality Information

23a

Apparent picture

24

Display position

25th

interface

26th

External data processing device

26a

Smartphone 26a

27

Street

27a

track

28a

vehicle

28b

vehicle

28c

vehicle

28d

vehicle

29

Roadside

30a

framework

30b

Written notice

31a

Direction arrow

31b

Written notes

32

Lane marking

33a

Guideline

33b

Direction arrow

33c

Written notice

34

framework

35

Holding device

36

front

37

back

38

3d illustration

39

Shop window

40

Retail store

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• US 2017/0294046 A1 [0004]
• WO 2013/170303 [0005]
• WO 2015/077766 A1 [0006]
• DE 102017214225 B3 [0007]

Claims (10)

Device (100) for displaying augmented reality information (23), comprising a display device (12) with a degree of transparency greater than 0%, a first detection device (15) directed into a first half-space in front of the display device (12), the first Detection device (15) is designed to detect data suitable for determining an observer position of an observer (17, 17a, 17b) in the first half space relative to the display device (12), a second detection device (12) directed into a second half space behind the display device (12). 19), the second detection device (19) being designed to detect data suitable for determining an object position of an object (20, 20a, 20b) in the second half-space relative to the display device (12), a computing device (21), the computing device (21) is designed to use the data acquired by the first acquisition device (15) to determine the observer's position relative to the de r display device (12), wherein the computing device (21) is designed to determine the object position relative to the display device (12) from the data captured by the second detection device (19), wherein the computing device (21) is designed, a Select augmented reality information (23) and assign the augmented reality information (23) to the object and a display position (24) of the augmented reality information (23) on the display device (12) based on the observer position and the object position the computing device (21) being designed to instruct the display device (12) to display the augmented reality information (23) assigned to the object (20, 20a, 20b) at the display position (24), characterized in that the computing device (21) is designed, a movement parallax resulting from a change in the observer position and / or the object position for the observer (17, 17a, 17b) d to determine augmented reality information (23) displayed on the display device (12) and to compensate for the movement parallax by updating the display position (24). Device (100) according to Claim 1 , characterized in that a storage device (22) is provided, a large number of augmented reality information items (23) being stored in the storage device (22). Device (100) according to Claim 1 or 2 , characterized in that the augmented reality information (23) displayed on the display device (12) from the observer position has an essentially constant spatial relationship to the object (20, 20a, 20b), the essentially constant spatial relationship preferably the displayed augmented reality information (23) about the object (20, 20a, 20b) comprises a substantially constant distance of the displayed augmented reality information (23) from the object (20, 20a, 20b), and / or wherein the essentially constant spatial relationship of the displayed augmented reality information (23) to the object (20, 20a, 20b) preferably has an essentially constant spatial orientation of the displayed augmented reality information (23) to the object (20, 20a, 20b). Device (100) according to one of the preceding claims, characterized in that the observer position comprises or is a head position of the observer (17, 17a, 17b) and / or that the observer position comprises or is an eye position and / or a line of sight. Device (100) according to one of the preceding claims, characterized in that the computing device (21) is designed to control the observer position and / or the change in the observer position and / or the object position and / or the change in the object position by means of artificial intelligence methods and / or machine learning methods and / or neural networks and / or by means of Haar cascades and / or Kalman filters and / or optical flow methods. Device (100) according to one of the preceding claims, characterized in that the display device (12) is a screen, in particular a display, further in particular a film display (11), and / or that the display device (12) has a degree of transparency of more than 10 %, preferably more than 20%, particularly preferably more than 40%, more particularly preferably more than 50%. Means of transport (200), in particular motor vehicle (10), rail vehicle, watercraft or aircraft, comprising a device (100) for displaying augmented reality information (23) according to one of the preceding claims. Means of transport (200) according to Claim 7 , characterized in that a navigation system is provided, the computing device (21) being designed to select augmented reality information (23) based on navigation information, and / or that the augmented reality information (23) traffic signs (20a ) relate, and / or that the augmented reality information (23) relate to information on the formation of an escape lane. Information system (300) comprising a holding device (35) and a device (100) for Representation of augmented reality information (23) according to one of the Claims 1 until 6th , wherein the display device (22) is attached to the holding device (35), characterized in that the holding device (35) is a piece of street furniture or a display or a window pane of a building. Method for displaying augmented reality information (23) on a display device (12), the display device (12) having a degree of transparency greater than 0%, with an observer (17, 17a, 17b) in a first half-space in front of the display device ( 12) is arranged, with data suitable for determining an observer position of the observer (17, 17a, 17b) in the first half space relative to the display device (12) being detected by a first detection device (15) directed into the first half space, wherein an object (20, 20a, 20b) is arranged in a second half-space behind the display device (12), with suitable for determining an object position of the object (20, 20a, 20b) in the second half-space relative to the display device (12) Data are acquired by a second acquisition device (19) directed into the second half-space, the first acquisition device from a computing device (21) (15) the observer position relative to the display device (12) is determined, and the object position relative to the display device (12) is determined by the computing device (21) from the data captured by the second detection device (19), wherein augmented reality information (23) is selected by the computing device (21) and assigned to the object (20, 20a, 20b), a display position (24) of the augmented reality information (23 ) is determined on the display device (12) on the basis of the observer position and the object position, the display device (12) being instructed by the computing device (21) to read the augmented reality information (23 ) at the display position (24), characterized in that the computing device (21) a change in the observer position and / or the object position for the observer (17, 17a, 17b ) The resulting movement parallax of the augmented reality information (23) displayed on the display device (12) is determined and the movement parallax is compensated by updating the display position (24).

Images