EP4370962A1

EP4370962A1 - Video streaming from a vehicle to a remote virtual reality system

Info

Publication number: EP4370962A1
Application number: EP22750807.4A
Authority: EP
Inventors: Siav Kuong Kuoch
Original assignee: Valeo Comfort and Driving Assistance SAS
Current assignee: Valeo Comfort and Driving Assistance SAS
Priority date: 2021-07-12
Filing date: 2022-07-12
Publication date: 2024-05-22
Also published as: WO2023285409A1; DE102021117904A1

Abstract

According to a method for video-streaming from at least one camera (3a, 3b, 3c, 3d) mounted to a vehicle (2) to a virtual reality system (8), video data is generated by the at least one camera (3a, 3b, 3c, 3d), and a video-stream is generated depending on the video data. A texture map is generated depending on the video-stream, an image is rendered by mapping the texture map to a virtual surface, and the rendered image is displayed by a display device (7) of the virtual reality system (8). Sensor data is generated by at least one acceleration sensor (9) of the vehicle (2). The at least one video-stream is generated depending on the sensor data or the texture map is generated depending on the sensor data or the texture map is mapped onto the virtual surface depending on the sensor data.

Description

Video streaming from a vehicle to a remote virtual reality system

The present invention relates to a method to a video-streaming from at least one camera mounted on a vehicle to a virtual reality system located remotely from the vehicle, wherein video data, which represents an environment of the vehicle, is generated by the at least one camera, and at least one video-stream is generated depending on the video data, a texture map is generated depending on the at least one video-stream, and an image is rendered by mapping the texture map onto a virtual surface and the rendered image is displayed by means of a display device of the virtual reality system. The invention is further related to a corresponding video-streaming arrangement and a computer program product.

Vehicle cameras mounted on different positions of a motor vehicle may be streamed to a remote virtual reality system, for example for telecommunication applications, wherein the user of the virtual reality system may communicate with a passenger of the vehicle as if he or she were present in the vehicle. Furthermore, also for teleoperation purposes in the context of automated driving, such streaming may ease the interpretation of the situation for the tele-operator. However, when the motion in the virtual world is different from the actual motion of the user of the virtual reality system, this may cause motion sickness.

Known approaches to reduce the level of motion sickness try to induce the virtual motion into the real world experienced by the user. To this end, for example, glasses informing the user about the motion of the streaming cameras may be used or a chair moving according to the motion of the cameras may be used. These approaches have the disadvantage that they require additional hardware on the side of the virtual reality system.

It is therefore an object of the present invention to provide an improved concept for video streaming from at least one camera mounted to a vehicle to a virtual reality system located remotely from the vehicle, which reduces motion sickness for the user of the virtual reality system without requiring additional hardware of the virtual reality system. This object is achieved by the respective subject-matter of the independent claims.

Further implementations and preferred embodiments are subject-matter of the dependent claims.

The improved concept is based on the idea to compensate the motion of the at least one camera in one of the processing steps from generating a video-stream to rendering an image by taking into account acceleration sensor data of the vehicle.

According to the improved concept, a method for video-streaming from at least one camera mounted to a vehicle to a virtual reality system located remotely from the vehicle is provided. Therein, video data, which represents an environment of the vehicle, is generated by the at least one camera and at least one video-stream is generated depending on the video data. A texture map is generated depending on the at least video stream, and an image is rendered by mapping the texture map onto a virtual surface. The rendered image is displayed to a user of the virtual reality system by means of a display device of the virtual reality system. Sensor data is generated by at least one acceleration sensor of the vehicle depending on a motion of the vehicle. The at least one video-stream is generated depending on the sensor data or the texture map is generated depending on the sensor data or the texture map is mapped onto the virtual surface depending on the sensor data.

The method steps of generating the at least one video-stream, generating the texture map and mapping the texture map to the virtual surface for rendering the image may be carried out by at least one computing unit. The at least one computing unit may comprise a vehicle computing unit of the vehicle located in the vehicle, an external server computing unit located remotely to the vehicle and the virtual reality system and/or a computing unit of the virtual reality system. Depending on the implementation of the method, said method steps may be carried out by one of the at least one computing unit or the different method steps may be shared amongst two or all three of the at least one computing unit. In the following, the external server computing unit is also denoted as cloud.

For example, the at least one video-stream may be generated by the vehicle computing unit and transmitted to the virtual reality system, in particular to the computing unit of the virtual reality system, directly or via the external server computer system. In particular, the at least one video-stream may be generated by the vehicle computing unit of the vehicle and transmitted from the vehicle computing unit to the external computing unit.

In some implementations, the texture map is generated and mapped onto the virtual surface by the external computing unit and the rendered image is transmitted from the external server computing unit to the computing unit of the virtual reality system.

In other implementations, the video-stream is forwarded from the external server computing unit to the computing unit of the virtual reality system, and the texture map is generated and mapped onto the virtual surface by the computing unit of the virtual reality system.

In yet other implementations, the texture map is generated by the external server computing unit, and the texture map is transmitted from the external server computing unit to the computing unit of the virtual reality system, and the texture map is mapped onto the virtual surface by the computing unit of the virtual reality system.

In further implementations, the video data may be streamed by at least one intermediate video stream from the vehicle computing unit to the external server computing unit or directly to the virtual reality system computing unit. Then, the at least one video-stream may, for example, be generated by the external server computing unit or the computing unit of the virtual reality system, in particular depending on the sensor data.

The at least one acceleration sensor, which may also be denoted as G-sensor, is configured to generate at least one sensor signal representing at least one value of an acceleration of the vehicle. The accelerations of the vehicle may comprise a linear acceleration or an angular acceleration. The sensor data correspond to the sensor signals or are derived from the sensor signals, for example, by the vehicle computing unit.

The at least one camera is rigidly connected to the motor vehicle. Consequently, any motion of the motor vehicle directly translates to a corresponding motion of each of the at least one camera. In particular, any rotational motion of the vehicle with respect to the environment translates directly to a corresponding rotational motion of the at least one camera. The sensor data may quantify or may be used to quantify this motion of the at least one camera. The video data may, for example, comprise respective camera images from each of the at least one camera for each of a series of consecutive frames. In case the at least one camera comprises two or more cameras, the frames may be synchronized among the two or more cameras. The generation of the at least one video-stream, the texture map or the mapping of the texture map to the virtual surface may be carried out depending on the sensor data for each frame. In particular, the sensor data may be updated at a rate, which may be equal to or greater than a frame rate of the at least one camera.

By taking into account the sensor data for generating the at least one video-stream, for generating the texture map or for rendering the image, at least a part of the motion of the vehicle and therefore the at least one camera, may be compensated. In consequence, the video displayed by the display device to the user only contains a reduced amount of motion of the vehicle, in particular only a desired part of the motion of the vehicle. This desired part may, for example, correspond to change of the yaw angle of the vehicle, while pitch and roll angle changes of the vehicle may not be contained in the rendered video-stream by the compensation depending on the sensor data as described.

Consequently, the risk for the user to get motion sick because the motion in the virtual reality does not match the motion of the user in the real world is reduced without requiring additional hardware components such as specific seats or glasses apart from regular virtual reality equipment.

The at least one camera may, for example, comprise an omnidirectional camera or a plurality of cameras, that is two or more cameras. In the latter case, the cameras may, for example, be designed as non-gnomonic cameras, such as fisheye cameras, in order to achieve a large overall field of view of the at least one camera. For example, the at least one camera may comprise a front camera, a rear camera, a left camera and a right camera mounted on different sides of the vehicle. In this way, in particular by using non- gnomonic cameras or fisheye cameras having a particularly large field of view, a total field of view of the at least one camera of at least approximately 360° in the horizontal plane and 360° in a corresponding vertical plane may be achieved. In this context, one may also refer to a spherical or approximately spherical total field of view of the at least one camera or a 360° field of view of the at least one camera. In this way, a high level of immersion may be achieved for the user.

According to several implementations of the method, the pitch angle and the roll angle of a vehicle coordinate system of the vehicle with respect to a reference vehicle coordinate system is computed, in particular by the at least one computing unit, for example, by the vehicle computing unit, depending on the sensor data. An effect of the motion of the vehicle on the video data is compensated at least in part by generating the at least one video-stream depending on the pitch angle and the roll angle or by generating the texture map depending on the pitch angle and the roll angle or by mapping the texture map onto the virtual surface depending on the pitch angle and the roll angle.

In general, the vehicle coordinate system may be any three-dimensional Cartesian coordinate system rigidly connected to the vehicle, in particular to the vehicle body, and therefore rigidly connected to each of the at least one camera. For example, the vehicle coordinate system may have an x-axis, which is parallel to a longitudinal axis of the vehicle, a y-axis, which is parallel to the lateral axis of the vehicle and a z-axis which is parallel to a normal axis of the vehicle.

The reference vehicle coordinate system may be defined as the vehicle coordinate system at a reference time. Consequently, at the reference time, the pitch angle, the roll angle and a yaw angle of the vehicle coordinate system with respect to the reference vehicle coordinate system is zero. At any other time, the sensor data allow to determine the pitch angle, the roll angle and the yaw angle.

For compensating the effect of the motion on the video data at least in part, the temporal change of the pitch angle and the yaw angle may be compensated. In contrast, the temporal change of the yaw angle may, for example, not be compensated in order to allow the user to follow the direction of the vehicle travel.

In some implementations, the at least one video-stream comprises exactly one video stream, also denoted as combined video-stream. In other words, the combined video stream may, for example, be generated depending on the sensor data in respective embodiments. On the other hand, for each of the cameras, an individual video-stream of the at least one video-stream may be generated, in some implementations depending on the sensor data. Then, these individual video-streams may be combined to generate a combined video-stream, wherein the combination is, for example, carried out independent of the sensor data.

For generating the texture map, the at least one computing unit may map each pixel of a frame of the combined video-stream to a plane. The mapping may, for example, be carried out depending on a predefined look-up table stored in the at least one computing unit.

In implementations, in which the combined video-stream has been generated independent of the sensor data, the look-up table may, for example, be updated by the at least one computing unit depending on the sensor data, in particular depending on the pitch angle and the roll angle, in order to generate the texture map depending on the sensor data.

For rendering the image depending on the texture map, the image points of the texture map are mapped onto the virtual surface, wherein a virtual vehicle view is taken into account with respect to the virtual surface for mimicking a perspective view of the user with respect to the virtual surface in order to generate an immersive experience for the user as if he or she would be present in the vehicle.

In order to render the image depending on the sensor data or, in other words, map the texture map onto the virtual surface depending on the sensor data, the at least one computing unit may adapt a rotational position of the virtual surface depending on the sensor data, in particular depending on the pitch angle and the yaw angle, or the at least one computing unit may rotate the virtual camera view depending on the sensor data, in particular depending on the pitch angle and the yaw angle, accordingly.

According to several implementations, the at least one camera comprises two or more cameras with different respective fields of view. A respective part of the video data is generated by each of the two or more cameras.

According to several implementations, in particular implementations, in which the at least one video-stream is generated depending on the sensor data, the respective parts of the video data are combined and mapped onto a plane depending on the sensor data.

In such implementations, the at least one video-stream may consist of only one video stream, namely the combined video-stream. The combined video-stream may be generated depending on the video data being mapped on the plane.

In particular, the combination of the parts of the video data may be carried out by the vehicle computing unit. According to several implementations, the combination and mapping of the parts of the video data to the plane is carried out according to an equirectangular projection depending on the sensor data.

According to several implementations, in particular implementations, in which the texture map is generated depending on the sensor data or the texture map is mapped to the virtual surface depending on the sensor data, a data stream is generated by the vehicle computing unit depending on the sensor data. The texture map is generated depending on the data stream or the texture map is mapped onto the virtual surface depending on the data stream.

The data stream may, for example, be transmitted according to a data channel, which separate from the channel or channels to transmit the at least one video-stream.

For example, the data stream may be transmitted from the vehicle computing unit to the computing unit of the virtual reality system, which may then generate the texture map depending on the data stream and then map the texture map on the virtual surface. Alternatively, the computing unit of the virtual reality system may first generate the texture map independent of the data stream and then map the texture map on the virtual surface depending on the data stream.

In other implementations, the data stream is transmitted from the vehicle computing unit to the cloud and the cloud generates the texture map depending on the data stream and then maps the texture map on the virtual surface or first generates the texture map independent of the data stream and then maps the texture map on the virtual surface depending on the data stream.

In further implementations, the data stream is transmitted from the vehicle computing unit to the cloud and the cloud generates the texture map depending on the data stream and then transmits the texture map to the computing unit of the virtual reality system and the virtual reality system maps the texture map on the virtual surface. Alternatively, the cloud may generate the texture map independent of the data stream and transmits the texture map to the computing unit of the virtual reality system, and the virtual reality system maps the texture map on the virtual surface depending on the data stream.

According to several implementations, in particular implementations, in which the texture map is generated depending on the sensor data, a look-up table assigning video image pixels to respective positions on the texture map is updated depending on the data stream and the texture map is generated depending on the updated look-up table.

According to several implementations, in particular implementations, in which the texture map is mapped to the virtual surface depending on the sensor data, an initial rendering surface is rotated depending on the data stream and the virtual surface is given by the rotated initial rendering surface.

According to several implementations, in particular implementations, in which the texture map is mapped to the virtual surface depending on the sensor data, a rendering camera view is rotated depending on the data stream, and the texture map is mapped onto the virtual surface according to the rotated rendering camera view.

According to several implementations, at least two intermediate video-streams are generated depending on the respective parts of the video data by the vehicle computing unit of the vehicle and the at least two intermediate video-streams are transmitted from the vehicle computing unit to the external server computing unit.

According to several implementations, the at least two intermediate video streams are combined and mapped onto the plane by the external server computing unit to generate a combined video-stream. The texture map is generated and mapped onto the virtual surface depending on the combined video-stream. In particular, the at least one video stream consists of the combined video-stream in this case.

According to several implementations, the combined video-stream is transmitted from the external server computing unit to the computing unit of the virtual reality system, and the texture map is generated and mapped onto the virtual surface by the computing unit of the virtual reality system.

In alternative implementations, the texture map is generated by the external server computing unit and transmitted from the external server computing unit to the computing unit of the virtual reality system, and the texture map is mapped onto the virtual surface by the computing unit of the virtual reality system.

In further implementations, the texture map is generated and mapped onto the virtual surface by the external server computing unit, and the rendered image is transmitted from the external server computing unit to the computing unit of the virtual reality system. According to the improved concept, also a video-streaming arrangement is provided. The video-streaming arrangement comprises at least one camera mountable to a vehicle and the at least one camera is configured to generate video data, which represents an environment of the vehicle, when mounted to the vehicle. The video-streaming arrangement comprises at least one computing unit, which is configured to generate at least one video-stream depending on the video data to generate a texture map depending on the at least one video-stream and to render an image by mapping the texture map onto a virtual surface. The video-streaming arrangement also comprises a display device for a virtual reality system, which is configured to display the rendered image to a user of the virtual reality system. The video-streaming arrangement further comprises at least one acceleration sensor for the vehicle, which is configured to generate sensor data depending on a motion of the vehicle, when mounted to the vehicle. The at least one computing unit is configured to generate the at least one video-stream depending on the sensor data or to generate the texture map depending on the sensor data or to map the texture map onto the virtual surface depending on the sensor data.

Further implementations of the video-streaming arrangement according to the improved concept are readily derived from the various implementations of the method according to the improved concept and vice versa. In particular, a video-streaming arrangement according to the improved concept may be configured to carry out a method according to the improved concept or carries out such a method.

According to the improved concept, also a computer program product comprising at least one set of instructions is provided. When the at least one set of instructions is executed by the at least one computing unit of a video-streaming arrangement according to the improved concept, the at least one set of instructions cause the video-streaming arrangement to carry out a method according to the improved concept. In particular, the at least one set of instructions may comprise a first set of instructions for the vehicle computing unit, a second set of instructions for the external server computing unit and a third set of instructions for the computing unit of the virtual reality system. The respective computing units may then carry out the respective set of instructions to cause the video-streaming arrangement carrying out the method. Each of the set of instructions may be considered as a respective computer program. Each of the set of instructions or each of the computer programs, respectively, may be stored on a respective computer-readable storage medium.

Further features of the invention are apparent from the claims, the figures and the description of figures. The features and feature combinations mentioned above in the description as well as the features and feature combinations mentioned below in the description of figures and/or shown in the figures alone may not only be encompassed by the improved concept in the respectively specified combination, but also in other combinations. Thus, implementations of the improved concept are encompassed and disclosed, which may not explicitly be shown in the figures or explained, but arise from and can be generated by separated feature combinations from the explained implementations. Implementations and feature combinations, which do not have all features of an originally formulated claim, may be encompassed by the improved concept. Moreover, implementations and feature combinations, which extend beyond or deviate from the feature combinations set out in the relations of the claims, may be encompassed by the improved concept.

In the figures:

Fig. 1 shows schematically an exemplary implementation of a video-streaming arrangement according to the improved concept;

Fig. 2 shows schematically a vehicle and respective fields of view of cameras mounted to the vehicle;

Fig. 3 shows a flow diagram of an exemplary implementation of a method according to the improved concept;

Fig.4a shows a rear view of a vehicle

Fig. 4b shows a side view of a vehicle

Fig.4c shows schematically a roll angle of a vehicle;

Fig. 4d shows schematically a pitch angle of a vehicle; Fig. 5 shows a flow diagram of a further exemplary implementation of a method according to the improved concept; Fig. 6 shows the respective fields of view of Fig. 2 as well as a schematic representation of a roll angle and a pitch angle;

Fig. 7 shows a flow diagram of a further exemplary implementation of a method according to the improved concept;

Fig. 8 shows a flow diagram of a further exemplary implementation of a method according to the improved concept; and

Fig. 9 shows a flow diagram of a further exemplary implementation of a method according to the improved concept.

Fig. 1 shows schematically an exemplary implementation of a video-streaming arrangement 1 according to the improved concept, in particular for carrying out a method for video-streaming to the improved concept.

The video-streaming arrangement 1 may comprise a virtual reality system 8, which may, for example, be implemented as a virtual reality headset with a corresponding display device 7 and a computing unit 6. Furthermore, the video-streaming arrangement 1 may comprise a vehicle computing unit 4 of the vehicle 2 and at least one camera 3a, 3b, 3c, 3d mounted to the vehicle 2. For example, the at least one camera 3a, 3b, 3c, 3d may comprise a front camera 3c, a rear camera 3d, a left camera 3a and a right camera 3b.

The video-streaming arrangement 1 may also comprise an external server computing unit 5, which may also be denoted as cloud in the following. The vehicle computing unit 4 may, in some implementations be implemented as an electronic control unit, ECU, of the vehicle 2, for example as a telecommunications ECU, also denoted as TCU.

Furthermore, the video-streaming arrangement 1 comprises at least one acceleration sensor 9, also denoted as G-sensor, of the vehicle 2, which is able to generate respective sensor data concerning, for example, a yaw motion, a pitch motion and/or a roll motion of the vehicle 2. The corresponding roll angle F and pitch angle Q are depicted schematically in Fig. 4a to Fig. 4d. Fig. 4a shows the vehicle 2 and the cameras 3a, 3b,

3d from a rear view at a reference time and Fig. 4b shows a view from the left side of the vehicle 2 as well as the cameras 3a, 3c, 3d. At the reference time, the vehicle coordinate system corresponding to the longitudinal, lateral and normal axis of the vehicle 2, corresponds to a reference vehicle coordinate system. Therefore, the pitch angle Q and the roll angle F as well as the yaw angle (not shown) are zero in Fig. 4a and Fig. 4b. In Fig. 4c and Fig. 4d, a respective rear view and left view of the vehicle 2 is shown at a later time, where the vehicle coordinate system deviates from the reference vehicle coordinate system. In the exemplary illustration of Fig. 4c, the roll angle F deviates from zero and in the exemplary illustration of Fig. 4d, the pitch angle Q deviates from zero.

In Fig. 2, the vehicle 2 is shown schematically together with respective fields of view 10a, 10b, 10c, 10d of the at least one camera 3a, 3b, 3c, 3d. Although not shown in Fig. 2, the individual fields of view 10a, 10b, 10c, 10d may also partially overlap with neighboring fields of view.

The video-streaming arrangement 1 according to Fig. 1 is configured to carry out a method for video-streaming according to the improved concept. To this end, the at least one camera 3a, 3b, 3c, 3d generates video data, which represents the environment of the vehicle 2. At least one video-stream is generated, for example by the vehicle computing unit 4 or by the cloud 5. A texture map is generated depending on the at least one video stream, for example, by the cloud 5 or the computing unit 6 of the virtual reality device 8 and the texture map is mapped onto a virtual surface in order to render an image, for example, by the cloud 5 or the computing unit 6. Then, the rendered image is displayed by means of the display device 7.

Furthermore, sensor data is generated by the at least one acceleration sensor 9 of the vehicle 2 depending on a motion of the vehicle 2, in particular depending on a change of the pitch angle Q and the roll angle F. The at least one video-stream is generated depending on the sensor data or the texture map is generated depending on the sensor data or the texture map is mapped onto the virtual surface depending on the sensor data.

In this way, the motion of the vehicle 2 may be partially compensated by removing pitch and roll rotation from the image or the corresponding sequence of images displayed by the display device 7, while the yaw rotation of the vehicle 2 may be kept as the car is moving in the real world. As a result, regardless of the motion of the vehicle 2, rendering of the 360° surround view will kept a horizon horizontal at 360°. This matches a floor the user of the virtual reality system 8 is present on, which is in general not moving. In this way, motion sickness may be avoided.

Fig. 3 shows a schematic flow diagram of an exemplary implementation of a method according to the improved concept, comprising method steps S1 to S13. Therein, the method steps S1 to S6 may, for example, be carried out by the vehicle computing unit 4, method steps S7 and S8 may be carried out by the cloud 5, and method steps S9 to S13 may be carried out by the computing unit 6.

In step S1 , the sensor data are generated by the at least one acceleration sensor 9 and in step S2, the vehicle computing unit 4 may encode the sensor data and generate in step a corresponding data stream S3. The data stream is then transmitted to the cloud 5. In step S4, video data is acquired by the cameras 3a, 3b, 3c, 3d and the respective parts of the video data corresponding to the individual cameras 3a, 3b, 3c, 3d are compressed by the vehicle computing unit 4 in step S5. In step S6, the vehicle computing unit 4 generates a respective video-stream for each of the compressed parts of video data. The video stream is transmitted to the cloud 5 in a separate communication channel in parallel to the data stream.

In the implementation of Fig. 3, the cloud 5 may, for example, not perform computational steps and therefore forward the data stream in step S7 to the computing unit 6 and the video-streams in step S8 to the computing unit S6. In step S9, the computing unit 6 may decode the data stream and compute an actual roll angle F and actual pitch angle Q of the vehicle coordinate system. In step S10, the computing unit 6 may update look-up tables for texture mapping for each of the video streams depending on the actual pitch and roll angles q, F. In step S11 , the computing unit 6 may decompress the video streams received from the cloud 5 and perform a texture mapping for each of the video streams depending on the updated look-up tables in step S12. As a result, a respective texture map is generated for each of the cameras 3a, 3b, 3c, 3d. In step S13, the computing unit 6 may render a respective image by mapping the texture maps to a virtual rendering surface, which may for example be a spherical- or bowl-like surface in order to mimic a realistic view for the rendered images. Afterwards, the rendered images may be combined by the computing unit 6 and displayed by the display device 7.

Fig. 5 shows a flow diagram for a further exemplary implementation of a method according to the improved concept. The method may be considered as a variant of the method described with respect to Fig. 3. In particular, steps S1 to S6 are identical to steps S1 to S6 described with respect to Fig. 3. In contrast to the method of Fig. 3, the data stream is decoded, and the pitch angle and roll angle q, F are updated by the cloud 5 in step S9. Also the look-up tables are updated correspondingly in step S10 by the cloud 5. The video decompression in step S11 is also carried out by the cloud 5. Consequently, the steps S7 and S8 of forwarding the data and video-streams are no longer present. In step S12, the texture mapping processes described with respect to Fig. 3 are carried out by the cloud 5. In step S1 T, the resulting texture maps are compressed by the cloud 5, and the resulting streams are transmitted to the computing unit 6 in step S8’ by the cloud 5. In step S11”, the computing unit 6 decompresses the received video-streams. Step S13 corresponds to the step S13 of Fig. 3.

Fig. 7 shows a flow diagram of a further exemplary implementation of a method according to the improved concept. Also here, steps S1 to S6 as well as steps S7 and S8 correspond to the respective steps S1 to S8 of Fig. 3. Also, the decompression step of S11 is the same as described with respect to Fig. 3. Flowever, in the following step S12’, a set of static look-up tables, which is in particular not updated depending on the sensor data stream, is used by the computing unit 6 for the texture mapping steps. Instead, the data stream is decoded by the computing unit 6 in step S9’ and taken into account by the computing unit 6 for rendering the image and, in particular, for mapping the texture maps to the virtual surface in step S13’. To this end, the computing unit 6 may, for example, rotate the virtual surface to compensate for the roll angle F and the pitch angle Q, as depicted schematically in Fig. 6. As shown in Fig. 6, the rotation may be considered to consist of rotations of -F and -Q in order to compensate for the motion of the vehicle 2.

In Fig. 8 a flow diagram of a further exemplary implementation of a method according to the improved concept is shown. Therein, step T1 corresponds to step S1 of Fig. 3, Fig. 5 and Fig. 7. Analogously, steps T3a, T3b, T3c and T3d correspond to step S4 of Fig. 3,

Fig. 5 and Fig. 7, where the video data are acquired by the cameras 3a, 3b, 3c, 3d. In step T2, the vehicle computing unit 4 may determine the actual roll angle F and the actual pitch angle Q depending on the sensor data and compute an updated pose of the cameras correspondingly. In step T4, the vehicle computing unit 4 may combine the parts of the video data by means of an equirectangular projection taking into account the respective orientations of the cameras 3a, 3b, 3c, 3d in terms of the updated roll angle F and pitch angle Q obtained in step T2. Consequently, a single combined video-stream may be generated in step T5 and compressed and transmitted to the cloud 5. The cloud 5 forwards the video-stream in step T6 to the computing unit 6. The computing unit 6 decompresses the combined video-stream in step T7 and maps the equirectangular projection to the virtual surface in step T8 in order to render the image. In other words, the equirectangular projection serves as a texture map in this implementation.

In Fig. 9, a flow diagram of a further exemplary implementation of a method according to the improved concept is shown. The method is based on the method of Fig. 8, wherein the computation step of the equirectangular projection T4 is carried out by the cloud 5. Also the steps T2 and T5 are carried out by the cloud 5. Steps T7 and T8 are identical to those of Fig. 8 and carried out by the computing unit 6. Also steps T 1 and T3a to T3d are identical to the respective steps of Fig. 8. In step T9, the sensor data are encoded and the data stream is generated in step T10 by the vehicle computing unit 4.

In the implementation of Fig. 9, each part of the video data is compressed separately in respective steps T11 a, T11 b, T11 c, T11 d, and respective video-streams are generated by the vehicle computing unit 4 and forwarded to the cloud 5 in the respective steps T12a, T12b, T12c, T12d. Furthermore, in steps T13a, T13b, T13c, T 13d the transmitted video streams are decompressed by the cloud 5 before the equirectangular projection is carried out in step T4.

According to the improved concept, as described, a method for streaming video content from cameras mounted to a vehicle to a remote virtual reality system, which reduces motion sickness for the user, is provided.

According to various implementations, the data generated by the at least one acceleration sensor are streamed in a separately channel to be taken into account by the respective computing unit of the vehicle, the cloud or the virtual reality system for motion compensation.

Claims

1. Method for video-streaming from at least one camera (3a, 3b, 3c, 3d) mounted on a vehicle (2) to a virtual reality system (8) located remotely from the vehicle (2), wherein - video data, which represents an environment of the vehicle (2), is generated by the at least one camera (3a, 3b, 3c, 3d) and at least one video stream is generated depending on the video data; a texture map is generated depending on the at least one video stream and an image is rendered by mapping the texture map onto a virtual surface; and - the rendered image is displayed by means of a display device (7) of the virtual reality system (8); characterized in that sensor data is generated by at least one acceleration sensor (9) of the vehicle (2) depending on a motion of the vehicle (2); - the at least one video stream is generated depending on the sensor data or the texture map is generated depending on the sensor data or the texture map is mapped onto the virtual surface depending on the sensor data.

2. Method according to claim 1 , characterized in that a pitch angle and a roll angle of a vehicle coordinate system of the vehicle (2) with respect to a reference vehicle coordinate system is computed depending on the sensor data; and an effect of the motion of the vehicle (2) on the video data is compensated at least in part by generating the at least one video stream depending on the pitch angle and the roll angle or by generating the texture map depending on the pitch angle and the roll angle or by mapping the texture map onto the virtual surface depending on the pitch angle and the roll angle.

3. Method according to one of the preceding claims, characterized in that the at least one camera (3a, 3b, 3c, 3d) comprises two or more cameras (3a, 3b,

3c, 3d) with different respective fields of view; and a respective part of the video data is generated by each of the two or more cameras (3a, 3b, 3c, 3d).

4. Method according to claim 3, characterized in that the parts of the video data are combined and mapped onto a plane depending on the sensor data.

5. Method according to claim 4, characterized in that the combination and mapping of the parts of the video data to the plane is carried out according to an equirectangular projection depending on the sensor data.

6. Method according to one of the preceding claims, characterized in that a data stream is generated by a vehicle computing unit (4) of the vehicle (2) depending on the sensor data; and the texture map is generated depending on the data stream or the texture map is mapped onto the virtual surface depending on the data stream.

7. Method according to claim 6, characterized in that a look-up table assigning video image pixels to respective positions on the texture map is updated depending on the data stream; and the texture map is generated depending on the updated look-up table.

8. Method according to claim 6, characterized in that an initial rendering surface is rotated depending on the data stream and the virtual surface is given by the rotated initial rendering surface.

9. Method according to claim 6, characterized in that a rendering camera view is rotated depending on the data stream and the texture map is mapped onto the virtual surface according to the rotated rendering camera view.

10. Method according to the preceding claims, characterized in that the at least one video stream is generated by a vehicle computing unit (4) of the vehicle (2) and transmitted from the vehicle computing unit (4) to an external server computing unit (5).

11. Method according to claim 10, characterized in that the texture map is generated and mapped onto the virtual surface by the external server computing unit (5) and the rendered image is transmitted from the external server computing unit (5) to a computing unit (6) of the virtual reality system (8); or the video stream is forwarded from the external server computing unit (5) to the computing unit (6) of the virtual reality system (8) and the texture map is generated and mapped onto the virtual surface by the computing unit (6) of the virtual reality system (8); or the texture map is generated by the external server computing unit (5) and the texture map is transmitted from the external server computing unit (5) to the computing unit (6) of the virtual reality system (8) and the texture map is mapped onto the virtual surface by the computing unit (6) of the virtual reality system (8).

12. Method according to one of claims 4 or 5, characterized in that at least two intermediate video streams are generated depending on the respective parts of the video data by a vehicle computing unit (4) of the vehicle (2); and the at least two intermediate video streams are transmitted from the vehicle computing unit (4) to an external server computing unit (5).

13. Method according to claim 12, characterized in that the at least two intermediate video streams are combined and mapped onto the plane by the external server computing unit (5) to generate a combined video stream; and the texture map is generated and mapped onto the virtual surface depending on the combined video stream.

14. Method according to claim 13, characterized in that the combined video stream is transmitted from the external server computing unit (5) to a computing unit (6) of the virtual reality system (8) and the texture map is generated and mapped onto the virtual surface by the computing unit (6) of the virtual reality system (8); or the texture map is generated by the external server computing unit (5) and transmitted from the external server computing unit (5) to the computing unit (6) of the virtual reality system (8) and the texture map is mapped onto the virtual surface by the computing unit (6) of the virtual reality system (8); or the texture map is generated and mapped onto the virtual surface by the external server computing unit (5) and the rendered image is transmitted from the external server computing unit (5) to the computing unit (6) of the virtual reality system (8).

15. Video-streaming arrangement (1 ) comprising at least one camera (3a, 3b, 3c, 3d) mountable on a vehicle (2), which is configured to generate video data, which represents an environment of the vehicle (2); at least one computing unit (4, 5, 6), which is configured to generate at least one video stream depending on the video data, to generate a texture map depending on the at least one video stream and the render an image by mapping the texture map onto a virtual surface; and a display device (7) for a virtual reality system (8), configured to display the rendered image; characterized in that the video-streaming arrangement (1) comprises at least one acceleration sensor (9) for the vehicle (2), which is configured to generate sensor data depending on a motion of the vehicle (2); and the at least one computing unit (4, 5, 6) is configured to generate the at least one video stream depending on the sensor data or to generate the texture map depending on the sensor data or to map the texture map onto the virtual surface depending on the sensor data.