WO2018184735A1

WO2018184735A1 - Concept for virtual participation in live events

Info

Publication number: WO2018184735A1
Application number: PCT/EP2018/025091
Authority: WO
Inventors: Karl-Heinz Charly GRAF; Christian Weissig; Arne FINN; Rodrigo DIAZ
Original assignee: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.; Fans United Gmbh
Priority date: 2017-04-06
Filing date: 2018-04-04
Publication date: 2018-10-11

Abstract

The invention relates, for a more authentic virtual participation of a user in a live event, to the recording of the live event during same by means of a camera device from a predetermined position within a viewing region of the live event, furthermore with the simultaneous possibility for the user to have playback signals output via his/her terminal device to one or more viewers in the viewing region and/or a device having a camera device for recording the surroundings of the camera device and an output interface for streaming at least one excerpt of the video material obtained by the camera device additionally has an information output apparatus, which outputs a playback signal, which depends on an information signal received via an input interface of the device from the terminal device of the user, to one or more people in the surroundings of the device.

Description

Concept for virtual participation in live events

The present invention relates to a concept for virtual participation in live events.

Live broadcasts of live events, such as Football matches or the like, exist in a variety of formats. However, typical live broadcasts on radio and television do not allow the recipients of the live broadcasts to virtually participate in the particular live event, in the sense that they could interact with anyone on site. Although there are already live events in which it is enabled viewers to share their euphoria, opinion or the like via text messages live the persons on site of the live event on scoreboards, but also this possibility does not cause the switched Viewers would get an impression or a feeling that they were present at the live event. In other words, there is currently a lack of a concept for allowing external viewers to get a sense of actually being present at the live event. The object of the present invention is to provide such a concept, namely a concept for a virtual participation of a user in a live event, so that the user gets the chance to experience the live event more authentically.

This object is solved by the subject matter of the independent patent claims of the present application.

According to a first aspect of the present application, it has been recognized that one can arrive at a more authentic virtual participation of a user in a live event, when during the live event recording the same by means of a camera device from a predetermined position within a spectator area of the live Event, while at the same time allowing the user to output via his / her terminal playback signals to one or more viewers in the audience area. The viewer's perspective provided by the streaming of at least a portion of the recorded video material thus gives the user the feeling of being "present" directly Having multiple viewers display in the audience area results in a more authentic experience of the live event for the user. The authenticity increases by recording the live event from perspectives that are characterized not just by spectacular telephoto shots or otherwise unavailable to viewers on site locations or recordings from the air, but perspectives in which viewers on site a non-vanishing share of the recorded Make out visual field.

According to another aspect of the present application, which can be connected to the first aspect but can also be implemented independently from the first aspect, a more authentic virtual participation of a user in a live event is made possible by using a camera device for recording an environment of the camera device and an output interface for streaming at least a portion of the video material obtained by the camera apparatus, is additionally provided with an information output device which transmits a reproduction signal, which depends on an information signal received via an input interface of the device from the user's terminal, to one or more persons the environment of the device outputs. In this case, the output of the reproduction signal may be such that the output of the reproduction signal and the camera position from the environment or for the one or more persons, at least in projection along a spatial direction, such as in vertical projection, from the same direction. For example, a field of view of the camera device covers more than 180 ° in a predetermined plane, such as in the horizontal plane. and the playback signal is output so that it seems to originate from the camera position in this plane or a position on a vertical to that plane through the camera position. The coincidence between camera position and perceived position of the output of the user-initiated playback signal increases the authenticity of the virtual participation in the live event for the user, because the on-site viewers who respond to the user's playback signal become highly true In this case, the position corresponds to the perceived output location of the playback signal, and this position coincides at least from the perspective of the external user in his perspective with his own virtual position in the live event, namely the camera position. The user thus gets the feeling that his actions, ie the playback signal, and reactions of the persons, such as spectators, back to each other locally and the same applies to the local people themselves. This increases the authenticity.

Advantageous embodiments of the present application are the subject of the dependent claims. Preferred embodiments of the present application will be described below with reference to the figures, among which

1 shows a schematic block diagram of a system for a virtual participation of a user in a live event, the system comprising a device for on-site use as well as a device implementable in a user's terminal according to an embodiment;

Fig. 2 is a block diagram of an exemplary external user terminal in which the user-side device is implementable;

3 is a schematic three-dimensional representation of a construction of an on-site device comprising the camera device and the information output device of the on-site device, which are installed in a columnar form in this embodiment;

Fig. 4 schematically shows an image of the field of view taken by the camera device in the case of full audience occupation when the camera device is positioned in the viewer area;

Fig. 5 is a schematic view showing a viewing direction-dependent control of the reproducing direction of the reproduced signal of the user according to an embodiment; Fig. 6 is a schematic representation of an alternative view direction dependent

Represents playback of the playback signal; and

7 is a schematic block diagram of a system for virtual participation of a user in a live event, such as a football event. Game, shows, with an exemplary representation of an embedding in infrastructures such as social networks.

The following description of preferred embodiments of the present application will begin by describing a system for virtual user participation in a live event using a user's terminal. It should be noted, however, that the entities belonging to the system, namely the device with the camera device to be positioned on site and the user's terminal or the device installable in the terminal, can be separately marketed or independent entities and thus independent embodiments of the invention present application. The same applies, of course, to the process flow which is presented in connection with the description. The system of FIG. 1 is indicated generally by the reference numeral 10 and includes an on-site device, ie, the live event, device 12 to be operated on and a device 18 installable in a terminal 14 of the user 16. The user 16 is thus a person who does not need to be in a location 20 where the live event takes place, as long as it can be somewhere else. The terminal device 14 can be, for example, a computer, a mobile telephone or another electronic device of the user 16. Fig. 2 shows an example of such a terminal 14 in more detail. It comprises a processor 20, a screen 22, one or more loudspeakers 24 and a user input interface 26. The user input interface 26 comprises for example a microphone 26a, a camera 26c, a touch function 26f of the screen 22, a keyboard 26b, a mouse 26g, a motion detection device 26d, a gesture interface 26e or combinations thereof. All units could be housed in a head-up display having the screen functionality of the display 22, the speaker functionality of the speaker 24, and motion detection functionality as the user input interface 26, as well as the processor 20. The processor 20 is connected to all the units 22, 24 and 26. The device 18 belonging to the system 10 could be, for example, an application running on the processor 20 or a computer program running on the processor 20. Alternatively, the terminal 14 could be a proprietary device dedicated to the system 10, in which case the device 18 would be fixedly installed in the terminal 14 and thus, as it were, the terminal device. guesses 14 itself part of the system 10 would be. The installability can thus designate a software or electrical installability.

As already mentioned, the user 16 is not himself at the location 20. He may rather be at home or elsewhere beyond the location of the event. The location 20 may be, for example, a football stadium, another stadium, a theater, an opera house or the like. In a manner that will be discussed in more detail below, the system 10 is able to ensure that the user 16 can at least virtually participate in the live event taking place at the location 20, such as e.g. a football game, another sporting event, a play, an opera or the like. Although the following description deals only with the relationship between a user 16 and the device 12, it is to be understood that in the manner described with respect to the user 16, a variety of users may be benefited in the same manner to have the feeling of experiencing the live event live, the device would interact with this variety.

As already mentioned, the device 12 is intended to be operated on-site at the live event. Strictly speaking, the device 12 comprises a camera device 28, an output interface 30, an input interface 32 and an information output device 34, the camera device 28 and the information output device 34 being intended to be positioned in situ. It should be noted that the device may also include a controller 80, with which and via which, for example, all devices of the device 28 are connected to one another. In part, functionalities of the devices of the device 28, e.g. the interfaces 30 and 32 and the optional controller 80 may be implemented on a processor, such as one or more programs.

Preferably, the positioning of the camera device 28 is performed such that the camera device 28 adapted to receive its surroundings makes the recording of the live event from a particular position 38 within a viewer area 36 of the live event location 20. The output interface streams at least a portion of the video material received by the camera device 28 through the camera device 28 to the terminal 14 of the user 16 or the device 18 to give the user 16 a visual impression of the captured environment. In the case of positioning the camera device 28 so that the mounting would happen from the audience area 36, the user 16 gets a fairly authentic impression, namely, as if he himself is among the spectators 40. In other words, the camera device 28 generates a video material 42 that is received by the output interface 30 due to its recording. For example, as described below, the camera apparatus 28 may include a plurality of cameras or a camera, and the video material may be a panoramic video formed by composing videos of different cameras of the camera apparatus 28, or a composite video of another type, eg, none Panoramic field of view covers, or a set of multiple videos of cameras of the camera device 28, looking in different directions and thus record different parts of the environment of the camera device 28 and also cover a field of view - with mutual overlap, against each other or with gaps. The camera device 28 thus has a specific field of view, which covers and captures it by capturing the environment with gaps or gaps in this field of view with one or more cameras, and the video material 42 represents this field of view. In Fig. 1, the visual field is representatively shown at 44 representatively. It is not shown in Fig. 1 simplistic as a panoramic field of view, although this is the case in the following embodiments. The output interface 30 may stream to the user 16 all of the video material 42 relative to the entire field of view 44, or only a portion 46 thereof, namely a portion of the field of view 44 having a limited spatial extent. The streaming itself takes place, for example, via the Internet 49, for example by means of HTTP. One possibility would be, for example, that the output interface 30 the Videomateria! 42 for download by the device 18, for example by means of a suitable streaming download protocol, provides, such as streaming or download using DASH. In this case, it would be possible, for example, for the device 18 to determine the cutout 46 to which the streaming is restricted: the cutout 46 is selected such that there is a viewing cutout 48 which is directly opposite the user 16 on the screen 22 of the terminal 14 is displayed. Alternatively, it would be possible for the output interface 30 to broadcast the video material 42 to the terminal 14 by broadcast or multicast signal, ie to the terminal 14 of the user 16 in one operation together with other other users, with the respective terminals, such as the terminal 14, then Extract and display the field of view 48 from the respective broadcast or multicast signal. The video material 42 streamed from the output interface 30 to the terminal 14, or the section 46 thereof, is illustrated by the arrow 50 in FIG. This direction is not the only one. Rather, the input interface 32 is adapted to receive from the terminal 14 an information signal 52. The transmission of the signal 52 as well as the transmission of the video material 50 via the Internet 49 could take place.

The information signal 52 is a signal generated by the device 16, which the user 16 has input via the user input interface 26, as previously mentioned, depending on user inputs of the user 16, for example, containing a message to one or more of the viewers 40, and possibly in the form of a text message, a video and / or a sound recording. In other words, in the information signal 52, a reproduction signal may be coded, e.g. a text message, an audio message or a video message or a combination of the same. On the device 12 side, the playback signal is then obtained, for example, by decoding. However, the information signal 52 could also encode gestures that the user 16 has entered. In this respect, the user 16 for generating the information signal 52 in the device 18 as a user input interface 26, a camera of the terminal 14, a keyboard of the terminal 14 and / or a microphone of the terminal 14 have used. The input interface 32 receives the information signal 52 from the terminal 14 and the device 18, respectively, and the information output device 34 outputs the reproduction signal, which depends on the information signal 52, to one or more persons in the vicinity of the camera device, i. to one or more viewers 40.

Before referring to the following figures, preferred embodiments and embodiments of the above-mentioned participating modules of the system and individual steps of the operation of the system will be described, it should be understood that it should be apparent that the device 12 is also used elsewhere could be as in a state where the camera device 28 is positioned in a viewer area 36. The functional capabilities would be the same.

Referring now to FIG. 3, an embodiment of the device 12 will be described, at least for that portion of the device 12 that includes the camera device 28 and the information output device 34. First, let's start with However, FIG. 3 illustrates that the information output device 34 and the camera device 28 are preferably arranged in a predetermined manner relative to each other, such as mechanically fixed relative to each other, disposed in a common housing, or supported by a common frame. 3 shows that the camera device 28 may possibly be configured as a camera system with a plurality of cameras with a frame of mirrors 60, the mirrors 60 being arranged, for example, like the lateral surface areas of a pyramid, via which cameras the camera device 28, 3, mirrored on the mirrors of an axis of symmetry 62 of the mirror gelgestells 60 look away, as in Fig. 3 by way of example with a dashed line 64 is indicated. In this way, the camera device 28 would be designed as a panoramic camera system, which would be able to cover as field of view 44 a solid angle range which covers 360 ° in the plane perpendicular to the axis of symmetry 62 of the structure 60 and perpendicular thereto, for example, covering an opening angle of greater than 30 ° , such as B. 30 ° to 150 °, or an opening angle in a range of 45 ° to 120 °.

As is now illustrated in FIG. 3, it is advantageous if the information output device 34 are arranged at the same location at least along the axial direction 62, ie the projections of the camera device 28 and information output device 34 coincide with one another along the axis 62. In other words, camera device 28 and information output device 34 are mounted on a common axis 62. For example, FIG. 3 shows that the information output device 34 may include a visual display 66 and / or a speaker array 68. Both the display 66 and the speaker array 68 are configured for omnidirectional display and audio reproduction, respectively. In other words, a display area of the display 66 is cylindrical or polygonal to allow display to all sides from the axis 62, and the speaker array 68 is an array of speakers having radiating directions in different, discrete directions are arranged from the axis 62. The loudspeakers of the array 68 are arranged, for example, along a circle or along a polygon about the axis 62, and also the display surface of the display 66 could be arranged symmetrically about the axis 62, such. Example, as the lateral surface of a cylinder, prism or a pyramid, wherein not necessarily the complete lateral surface must be designed as a display surface, but for example, a predetermined part, such as less than 40% or less than 20%, the cylindrical surface, prism surface or pyramid Silent surface not belong to the display, but belong to the housing, for example. But it is enough when visual display 66 and / or loudspeaker array 68 can emit in discrete directions that are projected along axis 62 across field of view 44. Omnidirectionality is not required for camera device 28, and for information output device 34, playback is in directions within the field of view stretched 44.

In Fig. 3, it is exemplarily shown that the display 66 is above the camera device 28 while the speaker array 68 is located below the camera device 28, but the mutual position along the axis 62 could also be different. The aligned arrangement along the axis 62, however, ensures that, from the environment captured by the camera device 28, the output of the playback signal is perceived by the information output device 34 from a position at least in projection along the spatial direction 62 with the camera position of the camera device 28 matches. More specifically, regardless of where a person is in the field of view 44 of the camera device 28, that person will perceive the playback signal coming from a direction intersecting the axis 62, i. from a position at least above or below the camera device 28, or even from the same direction, i. the direction in which the camera device 28 is spatially positioned for the person.

For a person, such as a viewer 40 in the field of view 44, this gives the feeling that the playback signal output from the information reproducing device 34 comes from the structure 70 comprising the information output device 34 and the camera device 28, and in which are installed. This in turn gives the person or persons 40 the feeling that they have a "counterpart with whom they can communicate, because the user 16, from whom the playback signal originates, can use the video material 50 to inform the person or persons concerned seen by the camera device 28, and the playback signal generated from the information signal 52 thus corresponds to a "response" to the recorded scene in the field of view 44. In this way, both users 16 and persons 40 in the Surrounding the camera device 28 the feeling of "having the other communication partner" in front of him. This increases the authenticity, as desired, especially on the part of the user 16, who is actually not on site. In FIG. 3, the assembly 70 housing the camera device 28 and the information output device 34 is designed as a frame or housing and has a substantially columnar shape and, more particularly, has a footprint 72 configured as a foot or anchorage of the assembly 70 is to set up the structure 70 on the floor, so that the axis 62, for example, corresponds to the vertical. The assembly 70 may be configured to be firmly anchored to the ground, such as in-set or the like. Alternatively, the structure 70 may be designed so that it can be reversibly placed and fixed on the base 72, such. B. by means of screws or the like. Of course, for the sake of completeness, it is pointed out that the standing surface 72 does not necessarily have to be present, so that the structure 70 could also be fixed to a ceiling or wire ropes, for example.

FIG. 3 further shows that optionally further components of the system 10 could be accommodated in the structure 70. For example, as an example of an audio pickup device 74, a microphone array or microphone could be housed in the assembly 70 to pick up the audio scene originating from the field of view 44 of the camera device 28 at a position substantially on the axis 62. The output interface 30 would thus receive an audio presentation from the audio capture device 74, namely the result of the audio capture of the environment of the camera device 28, and would send an audio signal 76 derived from the audio presentation to the terminal 14 and the device 18, respectively. Of course, this transmission could also take place via the Internet 49. The audio signal 76 could be, for example, a multi-channel or multi-object signal, such as an MPEG surround or SAOC signal. In particular, the audio signal! 76 may be configured to represent the spatial audio scene arriving from the field of view 44 toward the camera device 28. The device 18 could play the audio signal 76 to the user 16 via the one or more speakers 24. In this case, it would be possible for the device 18 to process the audio signal 76 so that the audio scene is adapted to the direction in which the viewing section 48 lies. The audio signal 76, like the video material 50, could be broadcast by broadcast or multi-user to a plurality of users. Alternatively, it would be possible for the audio signal 76 as well as the video material 50 to be downloaded from the device 18 by means of a streaming protocol. In this case, it would be possible that the audio signal 76 is already being selectively downloaded by the device 18 or so that the audio signal 76 already represents the audio scene adapted to the direction 48. In this case For example, the audio signal 76 could be a stereo signal corresponding to the audio scene originating from the field of view 44, and in which left and right channels are aligned relative to the direction of the field of view 48. FIG. 3 further shows that the assembly 70, as another component of the device 12, could include a gesture input interface 78 comprising, for example, a set of cameras looking in discrete, different directions away from the axis 62 and capable of recognizing gestures that originate from persons in the field of view 44. With this device 78, persons in the camera-forming device, such as the viewers 40, would be able to gesture-issue commands to an optional controller 80 of the device 12, such as commanding to set up a videoconference with a particular user 16 to end. Examples will follow. Finally, FIG. 3 shows, by way of example, that it may be possible that the assembly 70 also houses interfaces to terminals of persons in the camera designation device, such as viewers 40, such as wireless interfaces, such as a Wi-Fi interface, a Bluetooth Interface or the like. This would also allow the people 40 in the vicinity of the camera building device or assembly 70 to enter commands, such as the just-mentioned command to set up or end a videoconference with a particular user 16.

In order to grant a particularly authentic virtual participation of the user 16 to the event or event, it is advantageous if the camera device 28 is positioned at a certain height h above the ground, such as, for example, with respect to the ground level of the audience area. In the case of FIG. 3, the height h is determined, for example, by the distance of the camera device 28 from the standing surface 72. However, of course, there are other possibilities for arranging the camera device 28, as mentioned above. It is advantageous if the height h is, for example, between 1 to 2.5 m (in each case inclusive), since then for the user 16 the visual field 44 appears to be viewed from a height which also corresponds to the usual viewing heights or perspectives of persons or on-site spectators 40 seems to match. When the camera device 28 is positioned, for example, in the field of view 44 assuming a dense population of the viewer area 36 with spectators, an area ratio between view taken by spectators, represented in FIG. 4 by crosshatching, 84 to the area of the field of view 44 results in which the View is not blocked by viewers, more than 20%. In Fig. 4, more than 50% are shown. This gives a very authentic feeling for the user 16.

FIG. 5 details the possibility already mentioned above that the information output device 34 can be configured such that the reproduction of the reproduction signal can be output in a directionally selective manner, ie directionally selective in one direction within the field of view 44 of the camera device 28. FIG by way of example, the case of a 360 ° field of view 44, in which case the information output device 34 would be capable, for example, of outputting the playback signal in different directions from, for example, the axis 32. For example, the output could include displaying the playback signal optically in the particular direction, or outputting the playback signal via a speaker in the appropriate direction. As indicated in Fig. 5, the output is in a direction corresponding to an instant visual field 48 from the field of view 44 displayed to the user 16 via a terminal 14 via, for example, the screen 22 thereof. 5 shows, by way of example, a hatched area 86 of the information output device 34, which area 86 of the information output device 34 is activated in order to output the playback signal of the user 16 into the camera device environment. In order to ensure this, the device 18 is designed, for example, to determine via the user input interface 26, depending on the user 16, a virtual viewing direction 88, by means of which the user 16 looks into the scene in the field of view 44. The user input interface 26, which uses the device 18 for this purpose, is, for example, a motion sensor of the terminal 14. The device 18 generates a direction signal 90 depending on the virtual line of sight 88, which sends it to the device 12, where it is received by the input interface 32, for example , The transmission can happen again via the Internet 49. The direction signal 90 encodes, for example, the line of sight 88. It would also be possible for the direction signal 90 to be the signal that defines the section 46 of the video material sent by the device 12 to the device 18 or the terminal 14 of the user 16. The advantage of the raster-selective output of the playback signal in a direction corresponding to the virtual line of sight 88 of the user 16 has the advantage that persons or viewers 40 who are not in the field of view 48 of the user 16, the playback signal from the user 16 is not, or not so much perceive and therefore sometimes not so much can refer to wrongly, whereby the authenticity is improved. In addition, this approach allows a larger number of users simultaneously with the persons acting as spectators 40 in the visual field 44 of the camera. On the other hand, since the individual communications between the individual users on the one hand and the viewers 40 on the other hand can be "distributed" along the horizontal or transverse extension direction of the visual field 44, ie, projected along the direction 32. which look in the same viewing direction 88 compete with each other with their playback signals, and it can be provided for this purpose that, as will be mentioned below, a selection process is carried out to decide which user 16 can output his playback signal in the appropriate direction and which This selection process might be automatic, such as based on certain priority rights of the users 16 or other characteristics associated with the users 16, such as the number of likes in social networks or the like Additionally or alternatively, a selection manually by the operator of the system 10. Thus, Fig. 5 shows that the playback signal originated by a particular user 16 can be outputted via the information output device 34 in a certain direction. The directional selectivity is reflected, for example, in that the playback signal is output to one or more speakers of the information output device 34, which radiate substantially in the direction 88. Alternatively or additionally, the playback signal is displayed in a portion of a display of the information output device 34 which faces in the direction 88 or is most visible from the direction 88. In the case of the audio reproduction of the reproduction signal, it is of course possible to superimpose a plurality of reproduction signals on each other. In the case of an optical display, ie on a display or a screen of the information output device 34, a local distribution of the display area could be used instead of an overlay. In other words, parts or all of the available display area of the information output device 34 would be split to assign to a subset of one or more users 16 from all users 16 associated with the device 12, the assignment also being made in that users 16 are assigned to those parts of the display area which face the viewing direction 88 of this user. Due to the limited number of such display areas, a selection can be made which, as already mentioned, can be performed automatically and / or manually by the system operator. The playback signal displayed in such a portion 86 of a display could be a video recorded via a camera of the terminal 14 functioning as a user input interface 26, or a text message which the user 16 may transmit via a microphone, for example. rophon with speech recognition of the terminal 14 or a keyboard of the terminal 14 has entered as user input device 26. Also in the audio playback of the playback signal of the user 16, a selection can take place in order to limit the mutual superimposition of the audio playback signals in number. The latter playback signals may have been input via a microphone of the terminal 14 as user input device 26.

Before referring to the above figures concrete implementation possibilities are discussed, as well as advantageous embodiments, it should be noted that in the case of carrying out the method of virtual participation of the user 16 in a live event in which the camera device 28 is positioned such that it receives the live event from a position within the viewer area 36, optionally with regard to the output of the dependent of the information signal 52 of the user 16 playback signal relaxations can be made, such as with respect to the location where the information signal reproduction is made. This can e.g. take place elsewhere, i. not co-local or in the same location as the position of the camera device 28. For example, the information output device 34 could comprise speakers distributed in the camera designation device 34, and instead of a directionally selective audio output, a location-selective audio output would take place in the sense that those speakers or the loudspeaker for reproducing the reproduction signal of a specific user 16 would be activated, which, viewed from the camera device 28, lies in the viewing direction 88 or in the viewing section 48, which the user 16 is currently looking at. In this case, it would also be said that primarily persons or viewers 40 receive the playback signal of a user 16 who also stands or sits in the field of vision of the latter, although the feeling for the addressed person or viewer 40 is lost, the playback signal may come from the setup 70.

For the sake of completeness, the last-mentioned situation is again indicated in FIG. 6, where, among speakers of an array of loudspeakers, which are comprised of the information output comprised by the information output device 34, such a part coincides with the playback signal originating from the user 16 , which lies in the region in which the virtual viewing direction 88 points, or in the region which projects in the direction 32 overlaps with the viewing section 48. In Fig. 5, these loudspeakers are indicated by hatching. Instead of speaker channels could also be a other types of playback locations may be distributed, such as screens at each audience seat or the like.

The above embodiments can be used, for example, in the live broadcast of football matches. This will be illustrated in the following with reference to FIG. 7. In contrast to conventional football transmission, the embodiments described above are based on the typical perception habits and rites of a fan during a stadium visit. In order to provide the user 16 with such a virtual stadium visit, the above embodiments provide an integrated camera and communication system 10. As has been described above, it is advantageous if, according to the authentic perceptual habit, a camera position is selected that is in a viewer area 36 lies. It could be a specially selected fan block in which viewers who belong to a particular fan community 100 are located. Users, such as the user 16, is given the opportunity to participate in the fan community 100 via the communication and interaction option described above. As described above, for this purpose, the camera device 28 can be designed as a 360 ° panoramic recording system. Voice communication 76/52, text communication 52, video communication 52 and / or gesture control 102 may be used. For example, the external fan 16, which could also be referred to as a satellite fan, has the option of using the device 18, which is for example an app on his terminal 14, such. As a smart TV, PC, notebook, tablet, smartphone or a VR device, the ability in the video material that is generated by the camera device 28, such as a 360 ° video, if necessary, with an audio live Stream 76 is supplemented to navigate from a stadium position midway between the fans 40. He sees the surrounded fans as shown in Fig. 4 at 84 from close range and at the same time the acoustic scene 104 is optionally offered depending on the direction of view on the terminal 14, wherein the speaker 24, via which the playback of the audio scene takes place to headphones can act. The device 18 could simultaneously provide access to a chat platform 106 with text, voice and image interaction capabilities. Thus, chat comments, voice messages, pictures or short video clips could be sent from a fan 16 to the chat platform 106 and optionally also to one of the displays 66, e.g. For example, the display 66 in a setup 70 is shown in FIG. 3. The information signals 52 from various users 16 that define playback signals could be subjected to selection as described above. The selection could be in one two-stage procedure. An automatic control 108 could filter out unwanted comments by, for example, keyword search, which are then discarded. The number of likes or other parameters 109 from a social network associated with the users of the competing playback signals may also be used for such selection 108. From the remaining playback signals, such as the remaining chat history, which can be followed in the device 18 of all external users 16, then, for example, moderators or operator side manually 1 10 a selection of suitable comments, ie matching playback signals, for example, pictures, May be selected to be displayed on the display 66, for example, so that they may also be read by the local fans or seen in the fan community 100 or heard.

For example, all posts can be rated as Likes. For local users or viewers 40, this assessment could also be done by gestures that are captured via the gesture interface 78 mentioned above. These ratings can then be used as a decision support for the automatic selection, but also for the manual selection of subsequent playback signals from external users. Selected external users 16 may be invited to a videoconference-like connection 12 to a segment. In other words, the information signal 52 of a user 16 could comprise a video and audio signal, and the user 16 would be assigned a specific part 86 to the display 66 and his audio part of the video signal via the loudspeaker array 68 also in the area 68, namely areas facing substantially in the virtual line of sight 88, which together with the transmission of video 50 and audio recording 76 from device 12 to terminal 14 creates a type of video conference. The latter transmission from device 12 to device 18 may be accomplished differently during the time of a videoconferencing, such as via another channel. There is also the possibility of direct or non-public communication between the fans, ie both the on-site viewers 40 and the external users 16. For this private chat rooms can be opened or direct Skype or Skype-like connections are established. For the improvements of the video conferencing options just mentioned, it could be provided that the eye contact between the video conference participants, ie the user 16 on the one hand and the one or more viewers 40 on the other side of the videoconference within the field of view, such. As the 360 ° panorama, by additional cameras, the parts of the field of view 44 redundant, but with possibly higher density record, can be improved by the user 16 of the video conference participants virtually as his screen 22 is displayed as if he Look at user 16 directly. A multi-view video analysis or teleprompter can also be used.

To overcome language barriers, voice recognition, automatic translation and gesture control can be integrated. That is, text messages as playback signals derived from a user 16 via information signals 52 may be subjected to voice recognition and automatic translation prior to their display or playback. Conversely, gesture control could be enabled for both viewers 40 and a user 16 on a terminal 14 side.

As discussed further above, for a simplified access of local fans 40, it may be possible for the system 10 to be interfaced, such as a computer. B. a wireless or Bluetooth interface is equipped. A functionality that has not yet been described in the preceding embodiments, but which may optionally be integrated, relates to a Snapshof functionality 12 for the user 16. As the activation of a particular button, button or the like, the device 18 of the device 12 sends a command, after which the device 12 extract a time excerpt from the video material 42 and send to the user 16 the terminal 14 associated address. The address may be, for example, an email address of the user, a particular social media account of the user 16 in a social network 14, or the like. The command, which is sent by the device 18 to the device 12, for example, provided with a time stamp corresponding to a time stamp of the currently displayed field of view 48 at the terminal 14 of the user 16. In this way, it is irrelevant that the many users communicating with the device 12 may have different end-to-end latencies in the video playback of the video material or the section thereof. For this functionality, the device 12, for example, relies on a buffer in which the video material 42 is temporarily stored for some time. The time segment 1 16 extracted in this way can either be a single image of the visual field 44, or affect a Biidfoige the visual field 44 with a predetermined duration. Since the timestamp provided to the instruction is derived from the timestamps provided by the timeframes of the video material 42 itself, this functionality is end-to-end latency neutral.

Further, in the case of setting up a videoconference between a user 16 and one or more on-site viewers 40, it is of course possible that not only the user 16 of the videoconference participant is displayed wary so as to look directly at his eye, but such a display can of course also be performed in the playback section 86 to the one or more viewers 40 by recording the record of the user 16 via a camera as a user input interface 26 and projecting it in a virtual line of sight so that the user 16 selects one or more several viewers 44 seems to look directly. In addition to the options described with reference to FIG. 3, it may be the case that the on-site device 12 is provided with a haptic interface, e.g. As a touch screen.

As described above, users may be allowed view-gaze navigation 16, which gaze navigation not only navigates the viewport 48 within the field of view 44, i. H. the viewing direction of a virtual camera, but possibly also the listening direction of the microphone array 74, the playback direction of the loudspeaker array 68 and / or the display direction of the display 66, as described above.

By the above embodiments thus receive external viewers such. For example, the user 16 has a feeling of being there on-site during a live event such as B. a stadium feeling in the case of a football game. This will give viewers such. As the user 16, a sense of belonging to the fan community on site. This may result in a closer bond with the clubs, such. B. in the football clubs. The clubs could for example be operator of the system 10 and thereby get the opportunity to increase their fan community significantly. These new fans 16 are personally addressable, giving the club the opportunity to engage with personalized marketing. Although some aspects have been described in the context of a device, it will be understood that these aspects also constitute a description of the corresponding method, so that a block or a component of a device is also to be understood as a corresponding method step or as a feature of a method step , Similarly, aspects described in connection with or as a method step also represent a description of a corresponding block or detail or feature of a corresponding device. Some or all of the method steps may be performed by a hardware device (or using a hardware device). Apparatus), such as a microprocessor, a programmable computer or an electronic circuit. In some embodiments, some or more of the most important method steps may be performed by such an apparatus.

Depending on particular implementation requirements, embodiments of the invention may be implemented in hardware or in software. The implementation may be performed using a digital storage medium, such as a floppy disk, a DVD, a Blu-ray Disc, a CD, a ROM, a PROM, an EPROM, an EEPROM or FLASH memory, a hard disk, or other magnetic media or optical memory are stored on the electronically readable control signals, which can cooperate with a programmable computer system or cooperate such that the respective method is performed. Therefore, the digital storage medium can be computer readable.

Thus, some embodiments of the invention include a data carrier having electronically readable control signals capable of interacting with a programmable computer system such that one of the methods described herein is performed.

In general, embodiments of the present invention may be implemented as a computer program product having a program code, wherein the program code is operable to perform one of the methods when the computer program product runs on a computer.

The program code can also be stored, for example, on a machine-readable carrier. Other embodiments include the computer program for performing any of the methods described herein, wherein the computer program is stored on a machine-readable medium. In other words, an embodiment of the method according to the invention is thus a computer program which has a program code for performing one of the methods described herein when the computer program runs on a computer. A further embodiment of the inventive method is thus a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program is recorded for carrying out one of the methods described herein. The data carrier, the digital storage medium or the computer-readable medium are typically representational and / or non-transitory or non-transient.

A further embodiment of the method according to the invention is thus a data stream or a sequence of signals, which represent the computer program for performing one of the methods described herein. The data stream or the sequence of signals may be configured, for example, to be transferred via a data communication connection, for example via the Internet.

Another embodiment includes a processing device, such as a computer or a programmable logic device, that is configured or adapted to perform one of the methods described herein.

Another embodiment includes a computer on which the computer program is installed to perform one of the methods described herein.

Another embodiment according to the invention comprises a device or system adapted to transmit a computer program for performing at least one of the methods described herein to a receiver. The transmission can be done for example electronically or optically. The receiver may be, for example, a computer, a mobile device, a storage device, or a similar device. be direction. For example, the device or system may include a file server for transmitting the computer program to the recipient.

In some embodiments, a programmable logic device (eg, a field programmable gate array, an FPGA) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform one of the methods described herein. In general, in some embodiments, the methods are performed by any hardware device. This may be a universal hardware such as a computer processor (CPU) or hardware specific to the process, such as an ASIC.

The devices described herein may be implemented, for example, using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

The devices described herein, or any components of the devices described herein, may be implemented at least in part in hardware and / or software (computer program).

For example, the methods described herein may be implemented using a hardware device, or using a computer, or using a combination of a hardware device and a computer.

The methods described herein, or any components of the methods described herein, may be performed at least in part by hardware and / or by software. The embodiments described above are merely illustrative of the principles of the present invention. It will be understood that modifications and variations of the arrangements and details described herein will be apparent to others of ordinary skill in the art. Therefore, it is intended that the invention be limited only by the scope of the appended claims and not by the specific details presented with the description and explanation of the embodiments herein.

Claims

claims

1 . A method for virtual participation of a user (16) in a live event, with

Capturing the live event by means of a camera device (28) from a predetermined position (38) within a viewer area (36) of the live event to obtain a video material (42);

Streaming at least a portion (46) of the video material (42) to a terminal (14) of the user (16);

Receiving an information signal (52) from the terminal (14); and

Outputting a playback signal that depends on the information signal (52) to one or more viewers (40) in the viewer area (36).

A method according to claim 1, wherein outputting the reproduction signal to the one or more viewers (40) is performed at the predetermined position (38).

A method according to claim 1 or 2, wherein the outputting of the reproduction signal to the viewers is performed by audio, text and / or video output.

A method according to any one of claims 1 to 3, wherein the outputting of the reproduction signal to the viewers is directionally selective in a direction which depends on a direction signal (90) originating from the terminal or location-selectively arranged on a subset of distributed Playback positions is performed, which depends on a direction signal (90), which comes from the terminal.

5. The method according to claim 3, wherein a virtual line of sight (88) is determined via a user interface (26) on the user's terminal (14), the direction signal (90) being dependent on the virtual line of sight (88), and the user (16) on the terminal (14) a view section (48) of the video material (42) which is dependent on the virtual viewing direction (88) is displayed.

6. The method according to any one of claims 1 to 5, wherein the streaming is performed by means of a broadcast or multicast signal to terminals of a plurality of users or by means of a Streamingdownloadprotokol! S.

A method according to any one of claims 1 to 6, wherein the video material (42) comprises a panoramic video or a set of one or more videos.

8. The method of claim 1, wherein the recording comprises an audio recording.

9. The method according to claim 8, wherein a virtual line of sight (88) is determined via a user interface (26) on the user's terminal, and at the terminal a viewing section (48) of the video material dependent on the virtual line of sight (88) (42) accompanied by an audio signal output corresponding to a directional portion of a spatial audio scene obtained by the audio recording.

A method according to any one of claims 1 to 9, wherein the reproduced signal outputted to the one or more viewers (40) is selected from a plurality of reproduced signals depending on information signals which depend on terminals of a plurality of users to whom the video material (42) is at least partially streamed.

1 1. Method according to one of claims 1 to 10, wherein the recording of the live event by means of the camera device (28) from a height (h) of 1 to 2.5 meters from a floor of the audience area (36) and / or from a position in the spectator area (36).

A device for virtual participation of a user (16) at a live event, comprising: a camera device (28) for capturing an environment of the camera device (28) to obtain a video material (28); an output interface (30) for streaming at least a portion (46) of the video material (42) to a terminal (14) of the user (16); an input interface (32) for receiving an information signal (52) from the terminal (14); and information output means (34) for outputting a reproduction signal depending on the information signal (52) to one or more persons (40) in the vicinity of the camera device (28) so as to be out of the environment captured by the camera device (28) the output of the information and the camera device (28) is perceived in the same direction at least in projection along a spatial direction (62).

13. The apparatus of claim 12, wherein the information output means (34) is adapted to output the playback signal by audio, text and / or video output.

14. An apparatus according to claim 12 or 13, wherein said information output means (34) is adapted to supply said playback signal to said one or more persons. directionally output direction (40) in a direction dependent on a direction signal (90) originating from the terminal (14).

An apparatus according to any one of claims 12 to 14, wherein said camera device (28) has a field of view (44) horizontally covering at least 180 °, and said information output means (34) horizontally selecting one or more directions from a plurality of is controllable across the visual field distributed directions in which the playback signal is output.

16. Device according to one of claims 12 to 15, wherein the output interface is adapted to stream the video material (42) by means of a broadcast or multicast signal to a plurality of user terminals or by means of a streaming download protocol.

The apparatus of any one of claims 12 to 16, wherein the video material (42) comprises a panoramic video or a set of one or more videos.

18. The device according to claim 12, wherein the device comprises an audio pickup device (74) configured to perform an audio recording of the surroundings of the camera device to obtain an audio presentation, wherein the output interface is formed, one of Audio presentation (76) to the terminal (14).

19. The apparatus of claim 18, wherein the audio pickup device (74):

is formed so that the audio presentation represents a spatial audio scene.

20. Device according to one of claims 12 to 19, wherein the output interface (30) is adapted to stream at least a respective section (46) of the video material (42) to a plurality of user terminals, wherein the input interface (32) is formed To receive information signals from the plurality of user terminals. and further comprising means for automatically or operator-controlled selecting the output reproduction gamma from a plurality of reproduction signals that depend on the information signals of the plurality of user terminals.

21. Device according to one of claims 12 to 20, wherein the device has a standing surface (72), wherein the camera device (28) is fixed relative to the support surface (72) that the recording from a height of 1 to 2.5 Meters opposite the base (72).

Apparatus according to any one of claims 12 to 21, adapted to extract a time segment from the video material upon receipt of a time stamped instruction from the terminal (14) by means of the time stamp, and to a user assigned to the terminal Address to send.

23. A system for virtual participation of a user (16) at a live event by means of a terminal (14) of the user, comprising a device (12) with a camera device (28) for receiving an environment of the camera device (28) to a video material (42) to obtain; an output interface (30) for streaming at least a portion (46) of the video material (42) to a terminal (14) of the user; an input interface (32) for receiving an information signal (52) from the terminal; and an information output device (34) for outputting a reproduction signal, which depends on the information signal (52), to one or more persons (40) in the vicinity of the camera device (28), and to a device (18 ) for displaying the at least one visible section (48) of the video material (42) relative to the user, generating the information signal (52) via a user interface (26) of the terminal (14). from the user (16) and sending the information signal (52) to the device

(12).

24. A system according to claim 23, wherein said information output means (34) is adapted to output the reproduction signal by audio, text or video output.

A system according to claim 23 or 24, wherein the information output means is adapted to output the reproduction signal to the one or more persons (40) in a direction-selective direction, that of a direction signal

(90) depends on or outputs selectively to a subset of distributed playback locations that depends on a direction signal (90); and the device (18) which can be installed in the terminal is designed to determine via the user interface (26), depending on the user, a virtual line of sight (88) to generate the direction signal (90) in dependence on the virtual line of sight (88), the direction signal (90) to the device (12) to send and the user (16) dependent on the virtual line of sight (88) viewing section (48) of the video material (42).

26. A system according to any one of claims 23 to 25, wherein the camera device (28) has a field of view (44) horizontally covering at least 180 °, the information output device (34) for selecting one or more directions from a plurality of horizontal over the field of view distributed directions is controllable, in which the playback signal is output.

A system according to any one of claims 23 to 26, wherein the output interface (30) is adapted to stream the video material (42) to a plurality of user terminals or via a streaming download protocol by means of a broadcast or multicast signal.

A system according to any one of claims 23 to 27, wherein the video material (42) comprises a panoramic video or a set of one or more videos.

29. A system according to any one of claims 23 to 28, wherein the device (12) comprises an audio pickup device (74) adapted to perform an audio recording of the surroundings of the camera device (28) to obtain an audio presentation, the output interface (30) is adapted to stream an audio signal (76) derived from the audio presentation to the terminal.

The system of claim 29, wherein the device (18) installable in the terminal is adapted to determine a virtual line of sight (88) via the user interface (26), the direction signal (90) depending on the virtual line of sight (88) and to the user a viewing section dependent on the virtual line of sight (88)

(48) of the video material (42) accompanied by an audio signal output corresponding to a directional excerpt from a spatial audio scene represented by the audio presentation.

31. A system according to any one of claims 23 to 30, wherein the output interface (30) is adapted to stream the video material at least partially to a plurality of user terminals, wherein the input interface (32) is adapted to receive information signals from the plurality of user terminals, and further comprising means for automatically or operator controlled selecting the output display gaiety from a plurality of playback signals depending on the information signals of the plurality of user terminals.

32. System according to one of claims 23 to 31, wherein the device (12) has a standing surface (72), wherein the camera device (28) is mounted opposite the support surface (72) such that the receptacle from a height of 1 to 2.5 meters opposite the base (72).

A system according to any of claims 23 to 32, wherein the apparatus (12) is adapted to extract and extract a time slice from the video material (42) upon receipt of a time stamped instruction from the terminal by means of the time stamp to send an address assigned to the user of the terminal.

34. The system of claim 33, wherein the device installable in the terminal is adapted, upon user input by the user, to derive the time stamp provided to the command from timestamps of currently displayed time frames of the video material and send the command to the device ( 12).

35. Computer program with a program code for carrying out the method according to one of claims 1 to 1 1, when the program runs on a computer.