DE102022116811A1 - APPARATUS, MOBILE DEVICE AND METHOD FOR CREATING AND DISPLAYING A PICTURE-IN-PICTURE - Google Patents

Abstract

The invention relates to a device for creating and displaying a picture-in-picture for a user in a motor vehicle, the device comprising: (i) a plurality of image sensors, an image being able to be captured by each of the image sensors; (ii) wherein a first image sensor of the image sensors, in particular an image sensor mounted on the outside of the motor vehicle, can capture a first image of an environment of the motor vehicle; (iii) wherein a second image sensor of the image sensors can capture a second image of at least a partial area of an interior of the motor vehicle; (iv) a communication device which is connected in terms of signals to the plurality of image sensors and which is configured to send the first and the second images to an externally arranged transmitting and receiving device which is connected in terms of signals to an externally arranged image processing device, wherein the externally arranged image processing device is configured to create a picture-in-picture, comprising the first image and the second image, (v) and wherein the externally arranged transmitting and receiving device is configured to send the created picture-in-picture to the communication device to send; (vi) a display device, in particular an LCD or an OLED display, which is configured to receive the picture-in-picture from the communication device and display it to the user in the motor vehicle.

Description

The present invention relates to a device, a mobile device and a method, in particular a computer-implemented method, for creating and displaying a picture-in-picture.

Newer motor vehicles can have measuring systems and/or cameras with which the surroundings of the motor vehicle can be recorded. One application for this can be assistance when parking. In this case, other vehicles and/or objects that are located in front of or behind the motor vehicle can be displayed on a display that is visible to a driver of the motor vehicle, so that the driver can avoid a collision with the other vehicles and/or the objects when parking. Furthermore, newer motor vehicles can have cameras in the interior of the motor vehicle, which enables recordings of the interior and the passengers inside. For example, suitable algorithms can be used to determine from the recordings whether a child has unbuckled themselves in a back seat. The driver can also be observed with such an interior camera. From recordings of the driver, suitable algorithms can be used to determine, for example, whether the driver is tired or distracted. If such a critical situation is detected, a warning message can be issued. From these examples it can be seen that areas of application of interior cameras and cameras for recording the surroundings of the motor vehicle can be different from one another.

The present invention is based on the object of providing an improved device for the use, in particular for use for a joint application, of interior cameras and cameras for detecting the surroundings of the motor vehicle.

A solution to this problem is achieved in accordance with the teaching of the independent claims. Various embodiments and developments of the invention are the subject of the subclaims.

A first aspect of the invention relates to an apparatus for creating and displaying a picture-in-picture to a user in a motor vehicle, the apparatus comprising: (i) a plurality of image sensors, an image being capable of being captured by each of the image sensors; (ii) wherein a first image sensor of the image sensors, in particular an image sensor mounted on the outside of the motor vehicle, can capture a first image of an environment of the motor vehicle; (iii) wherein a second image sensor of the image sensors can capture a second image of at least a partial area of an interior of the motor vehicle; (iv) a communication device which is connected in terms of signals to the plurality of image sensors and which is configured to send the first and the second images to an externally arranged transmitting and receiving device which is connected in terms of signals to an externally arranged image processing device, wherein the externally arranged image processing device is configured to create a picture-in-picture, comprising the first image and the second image, (v) and wherein the externally arranged transmitting and receiving device is configured to send the created picture-in-picture to the communication device to send; (vi) a display device, in particular an LCD or an OLED display, which is configured to receive the picture-in-picture from the communication device and display it to the user in the motor vehicle.

The terms “comprises,” “includes,” “includes,” “has,” “has,” “with,” or any other variation thereof, as may be used herein, are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or has a list of elements is not necessarily limited to those elements, but may include other elements that are not expressly listed or that are inherent to such a method or apparatus.

Furthermore, unless expressly stated to the contrary, “or” refers to an inclusive or and not an exclusive “or”. For example, a condition A or B is satisfied by one of the following conditions: A is true (or present) and B is false (or absent), A is false (or absent) and B is true (or present), and both A and B are true (or present).

The terms “a” or “an” as used herein are defined to mean “one or more”. The terms “another” and “another” as well as any other variant thereof are to be understood in the sense of “at least one further”.

The term “plural” as used herein is to be understood in the sense of “two or more”.

For the purposes of the invention, the term “configured” or “set up” to fulfill a specific function (and respective modifications thereof) means that the corresponding device is already in a configuration or setting in which it can carry out the function or it is at least so adjustable - ie configurable - that it can carry out the function after the appropriate setting. The configuration can be done, for example, by appropriately setting parameters of a process flow or switches or the like to activate or deactivate functionalities or settings. In particular, the device can have several predetermined configurations or operating modes, so that the configuration can be carried out by selecting one of these configurations or operating modes.

The term “image sensor” in the context of the invention is to be understood in particular as meaning a camera that is able to capture a scene using sensors, from which an image can be generated that can be viewed or perceived by a user.

The term “picture-in-picture” as used here means in particular an image in which at least two separately recorded images are displayed at the same time. Separate recording can mean that the two images were taken at different times, were taken by different cameras, and/or have different motifs. One image can be shown in a smaller view than the other image. The views can also be changed so that the other image is shown smaller. Furthermore, “picture-in-picture” can mean several still images or several moving images, i.e. a “video-in-video” or a mixed “picture-in-video”.

The term “mobile device” as used here means in particular a mobile phone, in particular a smartphone, a tablet, a laptop or the like.

The term “externally arranged” as used here means in particular that the devices, devices, or the like affected by it are not arranged as standard and continuously on the motor vehicle, and are also not standardly connected to electronic components of the motor vehicle in terms of signaling or electronic means are. However, it cannot be ruled out that the affected facilities, devices or the like will be installed on the motor vehicle subsequently, i.e. after the motor vehicle has been purchased. For example, when purchasing a motor vehicle, there is usually no cell phone installed in the motor vehicle. By means of an auxiliary structure such as a holder arranged on a dashboard, the mobile phone can be arranged at least temporarily on the motor vehicle.

The device according to the first aspect can ensure that the picture-in-picture is created by an externally arranged image processing device, in particular an image processing device that is integrated in a mobile device, in particular a mobile phone or smartphone. Because the picture-in-picture is created by the externally arranged image processing device, the computing power required for this can be saved in the motor vehicle and used for other applications.

Preferred embodiments of the device are described below, which can be combined in any way with each other and with the other aspects of the invention described, unless this is expressly excluded or is technically impossible.

In some embodiments, the at least first image sensor is arranged so that it can capture the first image from a first direction that is different from a second direction from which the second image sensor can capture the second image. This allows images from different directions to be captured, which can be displayed on a picture-in-picture, giving a user a quicker overview of an overall situation that has images from different directions.

In some embodiments, the device has an image evaluation device which is configured to determine whether the user is depicted in the second image and in which direction the user is pointing and/or looking, and wherein the image evaluation device is configured depending on the Adjust an image section of the first image in a certain direction. The determination of whether the user is depicted in the image and is pointing and/or looking in a certain direction can be done using face recognition and gesture recognition algorithms known from the prior art. Furthermore, this specific direction can be calibrated with reference objects in the interior, for example door handles, fittings or the like, so that the direction can be determined in relation to the interior. If a relative position of the first image sensor to the second image sensor and an opening angle of the first image sensor are known, an image section can be qualitatively assigned to the direction. By adjusting an image section depending on the direction, the image section can be selected so that it correlates with the direction, ie that the user's gaze or gesture shows, of a real scene that corresponds to the adjusted image section. This means that the direction shown can be shown on the picture-in-picture the user as well as the associated image section are displayed.

In some embodiments, the communication device is configured to wirelessly send the picture-in-picture to an externally arranged electronic device, in particular a mobile device, for displaying the picture-in-picture. This allows the picture-in-picture to be sent to other electronic devices and displayed on these devices.

In some embodiments, the display device is configured to display the picture-in-picture with augmented reality content, also known as “augmented reality.” This allows a realistic representation of the picture-in-picture to be achieved and the overall content of the picture-in-picture can be grasped more quickly by the user.

In some embodiments, the device has an externally arranged transmitting and receiving device. This allows signals to be exchanged between the communication device and the externally arranged transmitting and receiving device.

In some embodiments, the device has an externally arranged image processing device. This allows image processing to be carried out while avoiding the use of the motor vehicle's computing power.

A second aspect of the invention relates to a motor vehicle having a device according to the first aspect.

A third aspect of the invention relates to a mobile device for creating and displaying a picture-in-picture, comprising: (i) a transmitting and receiving device configured to receive a first image captured by an image sensor of a motor vehicle; (ii) a capture device capable of capturing a second image; (iii) an image processing device, which is connected in terms of signals to the transmitting and receiving device and the detection device, and which is configured to create a picture-in-picture, comprising the first image and the second image; (iv) a display device configured to receive and display the picture-in-picture from the image processing device (240). This allows a picture-in-picture to be created that has an image of an image sensor of a motor vehicle and an image of a mobile device.

Preferred embodiments of the device are described below, which can be combined in any way with each other and with the other aspects of the invention described, unless this is expressly excluded or is technically impossible.

In some embodiments, the image processing device is configured to create the picture-in-picture in real time and the display device is configured to display the picture-in-picture in real time. This allows changes that can be detected by the first image sensor and/or the second image sensor to be displayed directly to the user.

A fourth aspect of the invention relates to a method, in particular a computer-implemented method, for creating and displaying a picture-in-picture for a user in a motor vehicle, the method having the following steps: (i) capturing a first image of an environment of the motor vehicle a first image sensor; (ii) capturing a second image of an environment of the motor vehicle by a second image sensor; (iii) sending the first image and the second image by a communication device to an externally arranged transmitting and receiving device which is connected in terms of signals to an externally arranged image processing device; (iv) creating a picture-in-picture comprising the first image and the second image by the externally located image processing device; (v) sending the picture-in-picture by the externally arranged transmitting and receiving device to the communication device; (vi) receiving the picture-in-picture by a display device which is signal-coupled to the communication device; (vii) displaying the picture-in-picture by the display device to a user in the motor vehicle.

The features and advantages explained in relation to the first aspect of the invention also apply accordingly to the further aspects of the invention.

Further advantages, features and possible applications of the present invention result from the following detailed description in connection with the figures.

• 1 schematisch ein Kraftfahrzeug mit einer Vorrichtung gemäß einer Ausführungsform der Erfindung.
• 2A-2C schematisch Bildausschnitte, die durch am Kraftfahrzeug angebrachte Bildsensoren aufgenommen wurden; und
• 3 ein Flussdiagramm zur Veranschaulichung einer bevorzugten Ausführungsform des erfindungsgemäßen Verfahrens.

This shows

• 1 schematically a motor vehicle with a device according to an embodiment of the invention.
• 2A-2C schematic image sections that were recorded by image sensors attached to the motor vehicle; and
• 3 a flowchart to illustrate a preferred embodiment of the method according to the invention.

In the figures, the same reference numerals are used throughout for the same or corresponding elements of the invention.

In 1 A motor vehicle 100 is shown schematically with a device according to an embodiment of the invention.

A first image sensor 110 for capturing images of the surroundings of the motor vehicle is arranged at the front area of the motor vehicle 100. The area that can be detected by the first image sensor 110 is indicated schematically by the dashed arrows that point in the direction of the first image sensor 110. Likewise, the first image sensor 110 can be arranged at the rear area of the motor vehicle 100 or at a side area. Furthermore, several image sensors can be arranged on the outside of the motor vehicle to detect the surroundings of the motor vehicle 100.

A second image sensor 120 for capturing images of at least a portion of an interior of the motor vehicle is arranged between the front seats and rear seats of the motor vehicle 100. The second image sensor 120 is arranged on the headliner of the motor vehicle 100. The area to be captured is indicated schematically by the dashed arrows that point in the direction of the second image sensor 120. The second image sensor 120 can also be arranged at the upper area of the windshield, which adjoins the headliner, or at another location in the interior of the motor vehicle 100 for detecting a portion of the interior. It is also conceivable that several image sensors are arranged in the interior of the motor vehicle 100.

A communication device 130 is also arranged in the motor vehicle 100, with which the captured images of the first image sensor 110 and the second image sensor 120 can be received, and with which the images can then be sent to a transmitting and receiving device 210. The transmitting and receiving device 210 is connected to an image processing device 240 in terms of signals.

The image processing device 240 and the transmitting and receiving device 210 are integrated in a mobile device 200, for example a smartphone.

The first image and the second image are transmitted wirelessly by the communication device 130, i.e. via Bluetooth, via WLAN or via a cellular network according to a cellular standard such as 3G, LTE or 5G, to the transmitting and receiving device 210 of the mobile device 200. Likewise, the transmitting and receiving device 210 and the communication device 130 can be connected to one another in terms of signals via a cable connection. The signaling connection between the transmitting and receiving device 210 and the communication device 130 is indicated schematically by the dashed double arrow.

The image processing device 240 creates a picture-in-picture, comprising the first image and the second image. The picture-in-picture is sent to the communication device 130 by the transmitting and receiving device 210 and from there to a display device 140 in the motor vehicle 100. The picture-in-picture in the motor vehicle 100 can be displayed for the user by the display device 140.

Furthermore, an image evaluation device 150 is arranged in the motor vehicle, with which images captured by the second image sensor 120 can be evaluated. The image evaluation device 150 can be used to determine whether the user is shown in the second image and in which direction the user is looking or pointing. Once a direction has been determined, an image section of the first image can be adjusted depending on this direction. The determination of whether the user is depicted in the second image and is pointing and/or looking in a specific direction can be done using face recognition and gesture recognition algorithms known from the prior art.

In addition, the picture-in-picture can be displayed on a display device 230 of the mobile device 200. Likewise, the mobile device 200 can have a capture device 220, whereby images can be captured. These images can also be used to create a picture-in-picture. Images from the first image sensor 110 or the second image sensor 120 can also be combined with the images that were captured by the detection device 220.

In the 2A-2C Image sections are shown schematically, which were recorded by image sensors 110, 120 attached to the motor vehicle.

In 2A is an image section 260 of an interior of a motor vehicle 100 shown schematically. This image section 260 can, for example, have been recorded by a second image sensor 120, which is arranged in the front area of a headliner, adjacent to a windshield of the motor vehicle 100. A driver and three passengers of the motor vehicle are shown schematically in the image section. In 2A Another, smaller image section 270 is shown, which represents a road. This picture Section may have been recorded with a first image sensor 110, which is arranged at the front area of the motor vehicle 100. Through the in 1 Using the device described, both image sections 260, 270 can be displayed simultaneously as a picture-in-picture on a display device 140 in the motor vehicle 100. The smaller image section 270 can represent an image section that the driver and passengers can see through the windshield of the motor vehicle 100.

In 2 B are schematically the same image sections as in 2A shown, but with inverted sizes, so that the course of the road is shown to fill the screen, and the image detail of the interior with the driver and passengers is shown reduced in size. In relation to the first image sensor 110 and the second image sensor 120, this means that the device makes it possible to select which image sensor 110, 120 or which image section should be displayed with priority.

2C indicates the representation of 2A on, wherein an additional image section 280, which was recorded by a further image sensor (not shown here) which is arranged on a side area of the motor vehicle, is shown.

In 3 is a flowchart 300 to illustrate a preferred embodiment of the method according to the invention for creating and displaying a picture-in-picture for a user in a motor vehicle 100.

In a first step 310 of the method, a first image of an environment of the motor vehicle 100 is captured by a first image sensor 110.

In a further step 320 of the method, a second image of an environment of the motor vehicle 100 is captured by a second image sensor 120.

In a further step 330 of the method, the first image and the second image are sent by a communication device 130 to an externally arranged transmitting and receiving device 210, which is connected in terms of signals to an externally arranged image processing device 240.

In a further step 340 of the method, an image-in-image, comprising the first image and the second image, is created by the externally arranged image processing device 240;

In a further step 350 of the method, the picture-in-picture is sent by the externally arranged transmitting and receiving device 210 to the communication device 130;

In a further step 360 of the method, the picture-in-picture is received by a display device 140, which is coupled in terms of signals to the communication device 130;

In a further step 370 of the method, the picture-in-picture is displayed by the display device 140 for a user in the motor vehicle 100.

While at least one exemplary embodiment has been described above, it should be noted that a large number of variations exist. It should also be noted that the exemplary embodiments described are only non-limiting examples and are not intended to limit the scope, applicability, or configuration of the devices and methods described herein. Rather, the foregoing description will provide guidance to those skilled in the art for implementing at least one exemplary embodiment, with the understanding that various changes in the operation and arrangement of the elements described in an exemplary embodiment may be made without departing from that set forth in the appended claims the specified subject matter and its legal equivalents are deviated from.

REFERENCE SYMBOL LIST

100

motor vehicle

110

First image sensor

120

Second image sensor

130

Communication facility

140

Display device

150

Image evaluation device

200

Mobile device

210

Transmitting and receiving device

220

Detection device

230

Display facility

240

Image editing facility

260

Image detail from an indoor camera

270

Image detail of an external camera front/rear

280

Image detail from an external camera at the side

300

flow chart

310

Capturing a first image

320

Capturing a second image

330

Sending the first image and the second image

340

Creating a picture-in-picture by an externally located image processing unit

350

Sending the picture-in-picture to a communication device

360

Receiving the picture-in-picture by a display device

370

View the created picture-in-picture

Claims (11)

Device for creating and displaying a picture-in-picture for a user of a motor vehicle (100), the device comprising: a plurality of image sensors (110, 120), an image being capable of being captured by each of the image sensors (110, 120); wherein a first image sensor (110) of the image sensors (110, 120) can capture a first image of an environment of the motor vehicle; wherein a second image sensor (120) of the image sensors (110, 120) can capture a second image of at least a partial area of an interior of the motor vehicle (100); a communication device (130), which is connected in terms of signals to the plurality of image sensors (110, 120), and which is configured to send the first and second images to an externally arranged transmitting and receiving device (210), which is connected to an externally arranged image processing device (240) is connected in terms of signals to send, wherein the externally arranged image processing device (240) is configured to create a picture-in-picture, comprising the first image and the second image, and wherein the externally arranged transmitting and receiving device (210 ) is configured to send the created picture-in-picture to the communication device (130); a display device (140) configured to receive the picture-in-picture from the communication device (130) and display it to the user in the motor vehicle (100). Device according to Claim 1 , wherein the at least first image sensor (110) is arranged so that it can capture the first image from a first direction, which is different from a second direction from which the second image sensor (120) can capture the second image. Device according to one of the preceding claims, comprising an image evaluation device (150) which is configured to determine whether the user is shown in the second image and in which direction the user is pointing and/or looking, and wherein the image evaluation device (150) is configured to adapt an image section of the first image depending on the specific direction. Device according to one of the preceding claims, wherein the communication device (130) is configured to wirelessly send the picture-in-picture to an externally arranged electronic device (200) for displaying the picture-in-picture. Device according to one of the preceding claims, wherein the display device is configured to display the picture-in-picture with augmented reality content. Device according to one of the preceding claims, comprising an externally arranged transmitting and receiving device (210). Device according to one of the preceding claims, comprising an externally arranged image processing device (240). Motor vehicle (100), having a device according to one of Claims 1 until 7 . Mobile device (200) for creating and displaying a picture-in-picture, comprising: a transmitting and receiving device (210) configured to receive a first image captured by an image sensor (110, 120) of a motor vehicle (100); a capture device (220) through which a second image can be captured; an image processing device (240), which is connected in terms of signals to the transmitting and receiving device (210) and the detection device (220), and which is configured to create a picture-in-picture, comprising the first image and the second image; a display device (230) configured to receive and display the picture-in-picture from the image processing device (240). mobile device Claim 7 , wherein the image processing device (240) is configured to create the picture-in-picture in real time, and wherein the display device (230) is configured to display the picture-in-picture in real time. Method for creating and displaying a picture-in-picture for a user in a motor vehicle, the method having the following steps: capturing a first image of an environment of the motor vehicle (100) by a first image sensor (110); Capturing a second image of an environment of the motor vehicle (100) by a second image sensor (120); Sending the first image and the second image by a communication device (130) to an externally arranged transmitting and receiving device (210), which is connected in terms of signals to an externally arranged image processing device (240); creating a picture-in-picture comprising the first image and the second image by the externally located image processing device (240); Sending the picture-in-picture by the externally arranged transmitting and receiving device (210) to the communication device (130); Receiving the picture-in-picture by a display device (140) which is signal-coupled to the communication device (130); Displaying the picture-in-picture by the display device (140) for a user in the motor vehicle (100).

Images