WO2023149835A1 - System and method for producing a video stream - Google Patents

System and method for producing a video stream Download PDF

Info

Publication number
WO2023149835A1
WO2023149835A1 PCT/SE2023/050090 SE2023050090W WO2023149835A1 WO 2023149835 A1 WO2023149835 A1 WO 2023149835A1 SE 2023050090 W SE2023050090 W SE 2023050090W WO 2023149835 A1 WO2023149835 A1 WO 2023149835A1
Authority
WO
WIPO (PCT)
Prior art keywords
digital video
video stream
participant
shared
stream
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/SE2023/050090
Other languages
English (en)
French (fr)
Inventor
Andreas BJÖRKMAN
Anders Nilsson
Lars ERLMAN
Per BLANKENS
Maxx SÖRQVIST
Marcus RAGNARSSON
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
LiveArena Technologies AB
Original Assignee
LiveArena Technologies AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to EP23750040.0A priority Critical patent/EP4473736A4/en
Priority to JP2024545896A priority patent/JP2025506378A/ja
Priority to CN202380020400.6A priority patent/CN118743225A/zh
Priority to AU2023216645A priority patent/AU2023216645A1/en
Priority to CA3249860A priority patent/CA3249860A1/en
Priority to KR1020257041491A priority patent/KR20260006042A/ko
Application filed by LiveArena Technologies AB filed Critical LiveArena Technologies AB
Priority to KR1020247027325A priority patent/KR20240144941A/ko
Publication of WO2023149835A1 publication Critical patent/WO2023149835A1/en
Priority to US18/434,868 priority patent/US12262069B2/en
Anticipated expiration legal-status Critical
Priority to JP2026016587A priority patent/JP2026082966A/ja
Ceased legal-status Critical Current

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/4302—Content synchronisation processes, e.g. decoder synchronisation
    • H04N21/4307—Synchronising the rendering of multiple content streams or additional data on devices, e.g. synchronisation of audio on a mobile phone with the video output on the TV screen
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23424—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G06F3/013—Eye tracking input arrangements
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/017—Gesture based interaction, e.g. based on a set of recognized hand gestures
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00—Data switching networks
    • H04L12/02—Details
    • H04L12/16—Arrangements for providing special services to substations
    • H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00—Data switching networks
    • H04L12/02—Details
    • H04L12/16—Arrangements for providing special services to substations
    • H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
    • H04L12/1813—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast for computer conferences, e.g. chat rooms
    • H04L12/1827—Network arrangements for conference optimisation or adaptation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L65/00—Network arrangements, protocols or services for supporting real-time applications in data packet communication
    • H04L65/40—Support for services or applications
    • H04L65/401—Support for services or applications wherein the services involve a main real-time session and one or more additional parallel real-time or time sensitive sessions, e.g. white board sharing or spawning of a subconference
    • H04L65/4015—Support for services or applications wherein the services involve a main real-time session and one or more additional parallel real-time or time sensitive sessions, e.g. white board sharing or spawning of a subconference where at least one of the additional parallel sessions is real time or time sensitive, e.g. white board sharing, collaboration or spawning of a subconference
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/21805—Source of audio or video content, e.g. local disk arrays enabling multiple viewpoints, e.g. using a plurality of cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/21—Server components or server architectures
    • H04N21/218—Source of audio or video content, e.g. local disk arrays
    • H04N21/2187—Live feed
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/234—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs
    • H04N21/23418—Processing of video elementary streams, e.g. splicing of video streams or manipulating encoded video stream scene graphs involving operations for analysing video streams, e.g. detecting features or characteristics
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
    • H04N21/23—Processing of content or additional data; Elementary server operations; Server middleware
    • H04N21/242—Synchronisation processes, e.g. processing of PCR [Programme Clock References]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/4223—Cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/434—Disassembling of a multiplex stream, e.g. demultiplexing audio and video streams, extraction of additional data from a video stream; Remultiplexing of multiplex streams; Extraction or processing of SI; Disassembling of packetised elementary stream
    • H04N21/4347—Demultiplexing of several video streams
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44213—Monitoring of end-user related data
    • H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV programme
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/478—Supplemental services, e.g. displaying phone caller identification, shopping application
    • H04N21/4788—Supplemental services, e.g. displaying phone caller identification, shopping application communicating with other users, e.g. chatting
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/141—Systems for two-way working between two video terminals, e.g. videophone
    • H04N7/147—Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems
    • H04N7/152—Multipoint control units therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L12/00—Data switching networks
    • H04L12/02—Details
    • H04L12/16—Arrangements for providing special services to substations
    • H04L12/18—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast
    • H04L12/1886—Arrangements for providing special services to substations for broadcast or conference, e.g. multicast with traffic restrictions for efficiency improvement, e.g. involving subnets or subdomains

Definitions

  • the present invention relates to a system, computer software product and method for producing a digital video stream, and in particular for producing a digital video stream based on two or more different digital input video streams.
  • the digital video stream is produced in the context of a digital video conference or a digital video conference or meeting system, particularly involving a plurality of different concurrent users.
  • the produced digital video stream may be published externally or within a digital video conference or digital video conference system.
  • the present invention is applied in contexts that are not digital video conferences, but where several digital video input streams are handled concurrently and combined into a produced digital video stream.
  • contexts may be educational or instructional.
  • participants may experience various challenges in terms of variable connectivity, leaving/reconnecting etc., posing further challenges in automatically producing a well- formed user experience.
  • Swedish application 2151461-7 which also not been published at the effective date of the present application, discloses various solutions specific to the handling of latency in multiparticipant digital video environments, such as when different groups of participants are associated with different general latency.
  • the invention relates to a method for providing a shared digital video stream, the method comprising in a collecting step, collecting from a first digital video source a first digital video stream and from a second digital video source a second digital stream; in a first production step, producing the shared digital video stream as an out-put digital video 5 stream based on consecutively considered frames of said first digital video stream so that image information from said first digital video source is visible in the shared digital video stream, but so that image information from said second digital video source is not visible in the shared digital video stream; in a trigger detection step, automatically detecting a trigger by performing at least one of a) digitally analysing said first digital video stream to automatzo ically detect the trigger in the form of a participant depicted in the first digital video stream gazing towards, or making a predetermined gesture in relation to, an object depicted in said second digital video stream, the detection being based on information about a relative orientation of the first camera, said participant and said object, the detection
  • the invention relates to a computer software product for providing a shared digital video stream, the computer software function being arranged to, when executing, perform a collecting step, wherein a first digital video stream is collected from a first digital video source and a second digital stream is collected from a second digital video source; a first production step, wherein the shared digital video stream is produced as an output digital video stream based on consecutively considered frames of said first digital video stream so that image information from said first digital video source is visible in the shared digital video stream, but so that image information from said second digital video source is not visible in the shared digital video stream; a trigger detection step, wherein a trigger is automatically detected by performing at least one of a) digitally analysing said first digital video stream to automatically detect the trigger in the form of a participant depicted in the first digital video stream gazing towards, or making a predetermined gesture in relation to, an object depicted in said second digital video stream, the detection being based on information about a relative orientation of the first camera, said participant and said object, the detection further being a
  • the invention relates to a system for providing a shared digital video stream, the system comprising a central server in turn comprising a collecting function, arranged to collect from a first digital video source a first digital video stream and from a second digital video source a second digital stream; a first production function, arranged to produce the shared digital video stream as an output digital video stream based on consecutively considered frames of said first digital video stream so that image information from said first digital video source is visible in the shared digital video stream, but so that image information from said second digital video source is not visible in the shared digital video stream; a trigger detection function, arranged to automatically detect a trigger by performing at least one of a) digitally analysing said first digital video stream to automatically detect the trigger in the form of a participant depicted in the first digital video stream gazing towards, or making a predetermined gesture in relation to, an object depicted in said second digital video stream, the detection being based on information about a relative orientation of the first camera, said participant and said object, the detection further being based on a digital
  • Figure 1 illustrates a first exemplifying system
  • Figure 2 illustrates a second exemplifying system
  • Figure 3 illustrates a third exemplifying system
  • Figure 4 illustrates a central server
  • Figure 5 illustrates a first method
  • Figures 6a-6f illustrate subsequent states in relation to the different method steps in the method illustrated in Figure 5;
  • Figure 7 illustrates, conceptually, a common protocol
  • Figure 8 illustrates a fourth exemplifying system
  • Figure 9 illustrates a second method; and is Figures lOa-llb illustrate, from above and in various configurations, cameras and participants.
  • FIG. 1 illustrates a system 100 according to the present invention, arranged to perform a method according to the invention for providing a digital video stream, such as a shared digital video stream.
  • the system 100 may comprise a video communication service 110, but the video communi-5 cation service 110 may also be external to the system 100 in some embodiments. As will be discussed, there may be more than one video communication service 110.
  • the system 100 may comprise one or several participant clients 121, but one, some or all participant clients 121 may also be external to the system 100 in some embodiments.0
  • the system 100 may comprise a central server 130.
  • central server is a computer-implemented functionality that is arranged to be accessed in a logically centralised manner, such as via a well-defined API (Application Programming Interface).
  • API Application Programming Interface
  • the functionality of such a central server may be im- plemented purely in computer software, or in a combination of software with virtual and/or physical hardware. It may be implemented on a standalone physical or virtual server computer or be distributed across several interconnected physical and/or virtual server computers.
  • the physical or virtual hardware that the central server 130 runs on may comprise a per se conventional CPU, a per se conventional GPU, a per se conventional RAM/ROM memory, a per se conventional computer bus, and a per se conventional external communication functionality such as an internet connection.
  • Each video communication service 110 is also a central server in said sense, that may be a different central server than the central server 130 or a part of the central server 130.
  • each of said participant clients 121 may be a central server in said sense, with the corresponding interpretation, and physical or virtual hardware that each participant client 121 runs on, in other words that computer software defining the functionality of the participant client 121, may also comprise a per se conventional CPU/GPU, a per se conventional RAM/ROM memory, a per se conventional computer bus, and a per se con- ventional external communication functionality such as an internet connection.
  • Each participant client 121 also typically comprises or is in communication with a computer screen, arranged to display video content provided to the participant client 121 as a part of an ongoing video communication; a loudspeaker, arranged to emit sound content provided to the participant client 121 as a part of said video communication; a video camera; and a microphone, arranged to record sound locally to a human participant 122 to said video communication, the participant 122 using the participant client 121 in question to participate in said video communication.
  • each of the participating clients 121 comprises a respective input means 123, w that may comprise said video camera; said microphone; a keyboard; a computer mouse or trackpad; and/or an API to receive a digital video stream, a digital audio stream and/or other digital data.
  • the input means 123 is specifically arranged to receive a video stream and/or an audio stream from a central server, such as the video communication service 110 and/or the central server 130, such a video stream and/or audio stream being provided as a part of is a video communication and preferably being produced based on corresponding digital data input streams provided to said central server from at least two sources of such digital data input streams, for instance participant clients 121 and/or external sources (see below).
  • each of the participating clients 121 comprises a respective output means0 124, that may comprise said computer screen; said loudspeaker; and an API to emit a digital video and/or audio stream, such stream being representative of a captured video and/or audio locally to the participant 122 using the participant client 121 in question.
  • each participant client 121 may be a mobile device, such as a mobile phone,5 arranged with a screen, a loudspeaker, a microphone and an internet connection, the mobile device executing computer software locally or accessing remotely executed computer software to perform the functionality of the participant client 121 in question.
  • the participant client 121 may also be a thick or thin laptop or stationary computer, executing a locally installed application, using a remotely accessed functionality via a web0 browser, and so forth, as the case may be.
  • Each of the participating clients may be allocated to such a respective group.
  • the groups may reflect different roles of the participating clients, different virtual or physical locations of the participating clients and/or different interaction rights of the participating clients.
  • Various available such physical locations may be, for instance, "on the stage”, “in the panel”, “in the physically present audience” or “in the physically remote audience”.
  • virtual location may be defined in terms of the physical location, but may also involve a virtual grouping that may partly overlap with said physical locations. For instance, a physically present audience may be divided into a first and a second virtual group, and some physically present audience participants may be grouped together with some physically distant audience participants in one and the same virtual group.
  • interaction rights may be, for instance, “full interaction” (no restrictions), “can talk but only after requesting the microphone” (such as raising a virtual hand in a video conference service), “cannot talk but write in common chat” or “view/listen only”.
  • each role defined and/or physical/virtual location may be defined in terms of certain predetermined interaction rights.
  • all participants having the same interaction rights form a group.
  • any defined roles, locations and/or interaction rights may reflect various group allocations, and different groups may be disjoint or overlapping, as the case may be. This will be exemplified below.
  • the video communication may be provided at least partly by the video communication service 110 and at least partly by the central server 130, as will be described and exemplified 5 herein.
  • a "video communication" is an interactive, digital communication session involving at least two, preferably at least three or even at least four, video streams, and preferably also matching audio streams that are used to produce one or sev- w eral mixed or joint digital video/audio streams that in turn is or are consumed by one or several consumers (such as participant clients of the discussed type), that may or may not also be contributing to the video communication via video and/or audio.
  • Such a video communication is real-time, with or without a certain latency or delay.
  • At least one, preferably at least two, or even at least four, participants 122 to such a video communication is in- 75 volved in the video communication in an interactive manner, both providing and consuming video/audio information.
  • At least one of the participant clients 121, or all of the participant clients 121, may comprise a local synchronisation software function 125, that will be described in closer detail below.
  • the video communication service 110 may comprise or have access to a common time reference, as will also be described in closer detail below.
  • Each of the at least one central server 130 may comprise a respective API 137, for digitally5 communicating with entities external to the central server 130 in question. Such communication may involve both input and output.
  • the system 100 may furthermore be arranged to digitally communicate with, and in particular to receive digital information, such as audio and/or0 video stream data, from, an external information source 300, such as an externally provided video stream.
  • an external information source 300 such as an externally provided video stream.
  • the information source 300 is "external" means that it is not provided from or as a part of the central server 130.
  • the digital data provided by the external information source 300 is independent of the central server 130, and the central server 130 cannot affect the information contents thereof.
  • the external information source 130 may be live captured video and/or audio, such as of a public sporting s event or an ongoing news event or reporting.
  • the external information source 130 may also be captured by a web camera or similar, but not by any one of the participating clients 121.
  • Such captured video may hence depict the same locality as any one of the participant clients 121, but not be captured as a part of the activity of the participant client 121 per se.
  • an externally provided information source 300 and an inter- w nally provided information source 120 are that internally provided information sources may be provided as, and in their capacity as, participants to a video communication of the abovedefined type, whereas an externally provided information source 300 is not, but is instead provided as a part of a context that is external to said video conference is There may also be several external information sources 300, that provide digital information of said type, such as audio and/or video streams, to the central server 130 in parallel.
  • each of the participating clients 121 may constitute the source of a respective information (video and/or audio) stream 120, provided to the video communi-0 cation service 110 by the participating client 121 in question as described.
  • the system 100 may be further arranged to digitally communicate with, and in particular to emit digital information to, an external consumer 150.
  • a digital video and/or audio stream produced by the central server 130 may5 be provided continuously, in real-time or near real-time, to one or several external consumers 150 via said API 137.
  • the consumer 150 is "external" means that the consumer 150 is not provided as a part of the central server 130, and/or that it is not a party to the said video communication.
  • all functionality and communication herein is provided digitally and electronically, effected by computer software executing on suitable computer hardware and communicated over a digital communication network or channel such as the internet.
  • a number of participant cli- 5 ents 121 take part in a digital video communication provided by the video communication service 110.
  • Each participant client 121 may hence have an ongoing login, session or similar to the video communication service 110, and may take part in one and the same ongoing video communication provided by the video communication service 110.
  • the video communication is "shared" among the participant clients 121 and therefore also w by corresponding human participants 122.
  • the central server 130 comprises an automatic participant client 140, being an automated client corresponding to participant clients 121 but not associated with a human participant 122. Instead, the automatic participant client 140 is added as a participant client is to the video communication service 110 to take part in the same shared video communication as participant clients 121. As such a participant client, the automatic participant client 140 is granted access to continuously produced digital video and/or audio stream(s) provided as a part of the ongoing video communication by the video communication service 110, and can be consumed by the central server 130 via the automatic participant client0 140.
  • the automatic participant client 140 receives, from the video communication service 110, a common video and/or audio stream that is or may be distributed to each participant client 121; a respective video and/or audio stream provided to the video communication service 110 from each of one or several of the participant clients 121 and relayed, in raw or modified form, by the video communication service 110 to all or requesting5 participant clients 121; and/or a common time reference.
  • the central server 130 may comprise a collecting function 131 arranged to receive video and/or audio streams of said type from the automatic participant client 140, and possibly also from said external information source(s) 300, for processing as described below, and0 then to provide a produced, such as shared, video stream via the API 137. For instance, this produced video stream may be consumed by the external consumer 150 and/or by the video communication service 110 to in turn be distributed by the video communication service 110 to all or any requesting one of the participant clients 121.
  • a collecting function 131 arranged to receive video and/or audio streams of said type from the automatic participant client 140, and possibly also from said external information source(s) 300, for processing as described below, and0 then to provide a produced, such as shared, video stream via the API 137. For instance, this produced video stream may be consumed by the external consumer 150 and/or by the video communication service 110 to in turn be distributed by the video communication service 110 to all or any requesting one of the participant clients 121.
  • Figure 2 is similar to Figure 1, but instead of using the automatic client participant 140 the s central server 130 receives video and/or audio stream data from the ongoing video communication via an API 112 of the video communication service 110.
  • Figure 3 is also similar to Figure 1, but shows no video communication service 110.
  • the participant clients 121 communicate directly with the API 137 of the central server w 130, for instance providing video and/or audio stream data to the central server 130 and/or receiving video and/or audio stream data from the central server 130.
  • the produced shared stream may be provided to the external consumer 150 and/or to one or several of the client participants 121.
  • Figure 4 illustrates the central server 130 in closer detail.
  • said collecting function 131 may comprise one or, preferably, several, format-specific collecting functions 131a.
  • Each one of said format-specific collecting functions 131a may be arranged to receive a video and/or audio stream having a predetermined format, such as a predetermined binary encoding format and/or a predetermined stream data container, and may be specifically0 arranged to parse binary video and/or audio data of said format into individual video frames, sequences of video frames and/or time slots.
  • a predetermined format such as a predetermined binary encoding format and/or a predetermined stream data container
  • the central server 130 may further comprise an event detection function 132, arranged to receive video and/or audio stream data, such as binary stream data, from the collecting5 function 131 and to perform a respective event detection on each individual one of the received data streams.
  • the event detection function 132 may comprise an Al (Artificial Intelligence) component 132a for performing said event detection. The event detection may take place without first time-synchronising the individual collected streams.
  • the central server 130 further comprises a synchronising function 133, arranged to time- synchronise the data streams provided by the collecting function 131 and that may have been processed by the event detection function 132.
  • the synchronising function 133 may comprise an Al component 133a for performing said time-synchronisation.
  • the central server 130 may further comprise a pattern detection function 134, arranged to s perform a pattern detection based on the combination of at least one, but in many cases at least two, such as at least three or even at least four, such as all, of the received data streams.
  • the pattern detection may be further based on one, or in some cases at least two or more, events detected for each individual one of said data streams by the event detection function 132. Such detected events taking into consideration by said pattern detection w function 134 may be distributed across time with respect to each individual collected stream.
  • the pattern detection function 134 may comprise an Al component 134a for performing said pattern detection.
  • the pattern detection may further be based on the abovediscussed grouping, and in particular be arranged to detect a particular pattern occurring only with respect to one group; with respect to only some but not all groups; or with respect is to all groups.
  • the central server 130 further comprises a production function 135, arranged to produce a produced digital video stream, such as a shared digital video stream, based on the data streams provided from the collecting function 131, and possibly further based on any de-0 tected events and/or patterns.
  • a produced video stream may at least comprise a video stream produced to comprise one or several of video streams provided by the collecting function 131, raw, reformatted or transformed, and may also comprise corresponding audio stream data.
  • there may be several produced video streams where one such produced video stream may be produced in the above-discussed way but5 further based on a another already produced video stream.
  • the central server 130 may further comprise a publishing function 136, arranged to publish the produced digital video stream in question, such as via API 137 as described above. It is noted that Figures 1, 2 and 3 illustrate three different examples of how the central server 130 can be used to implement the principles described herein, and in particular to provide a method according to the present invention, but that other configurations, with or without using one or several video communication services 110, are also possible.
  • Figure 5 illustrates a method for providing said produced digital video stream.
  • Figures 6a-6f illustrates different digital video/audio data stream states resulting from the method steps illustrated in Figure 5.
  • the method starts.
  • respective primary digital video streams 210, 301 are collected, such as by said collecting function 131, from at least two of said digital video sources 120, 300.
  • Each such primary data stream 210, 301 may comprise an audio part 214 and/or a video part 215.
  • video in this context, refers to moving and/or still image contents of such a data stream.
  • Each primary data stream 210, 301 may be encoded according to any video/audio encoding specification (using a respective codec used by the entity providing the primary stream 210, 301 in question), and the encoding formats may be different across different ones of said primary streams 210, 301 concurrently used in one and the same video communication.
  • At least one, such as all, of the primary data streams 210, 301 is provided as a stream of binary data, possibly provided in a per se conventional data container data structure. It is preferred that at least one, such as at least two, or even all of the primary data streams 210, 301 are provided as respective live video recordings.
  • the primary streams 210, 301 may be unsynchronised in terms of time when they are received by the collecting function 131. This may mean that they are associated with different latencies or delays in relation to each other. For instance, in case two primary video streams 210, 301 are live recordings, this may imply that they are associated, when received by the collecting function 131, with different latencies with respect to the time of recording.
  • the primary streams 210, 301 may themselves be a respective live cam- s era feed from a web camera; a currently shared screen or presentation; a viewed film clip or similar; or any combination of these arranged in various ways in one and the same screen.
  • the collecting step is shown in Figures 6a and 6b.
  • the collecting function 131 can store each primary video stream 210, 301 as bundled au- w dio/video information or as audio stream data separated from associated video stream data.
  • Figure 6b illustrates how the primary video stream 210, 301 data is stored as individual frames 213 or collections/clusters of frames, "frames" here referring to time-limited parts of image data and/or any associated audio data, such as each frame being an individual still image or a consecutive series of images (such as such a series constituting at the most 1 is second of moving images) together forming moving-image video content.
  • said primary digital video streams 210, 301 are analysed, such as by said event detection function 132 and in particular said Al component 132a, to detect at least one event 211 selected0 from a first set of events. This is illustrated in Figure 6c.
  • this event detections step may be performed for at least one, such as at least two, such as all, primary video streams 210, 301, and that it may be performed individually for each such primary video stream 210, 301.
  • the event detection5 step preferably takes place for said individual primary video stream 210, 301 only taking into consideration information contained as a part of that particular primary video stream 210, 301 in question, and particularly without taking into consideration information contained as a part of other primary video streams.
  • the event detection preferably takes place without taking into consideration any common time reference 260 associ-0 ated with the several primary video streams 210, 301.
  • the event detection preferably takes into consideration information contained as a part of the individually analysed primary video stream in question across a certain time interval, such as a historic time interval of the primary video stream that is longer than 0 seconds, such as at least 0.1 seconds, such as at least 1 second.
  • the event detection may take into consideration information contained in audio and/or video data contained as a part of said primary video stream 210, 301.
  • Said first set of events may contain any number of types of events, such as a change of slides in a slide presentation constituting or being a part of the primary video stream 210, 301 in question; a change in connectivity quality of the source 120, 300 providing the primary video stream 210, 301 in question, resulting in an image quality change, a loss of image data or a regain of image data; and a detected movement physical event in the primary video stream 210, 301 in question, such as the movement of a person or object in the video, a change of lighting in the video, a sudden sharp noise in the audio or a change of audio quality. It is realised that this is not intended to be an exhaustive list, but that these examples are provided in order to understand the applicability of the presently described principles.
  • the primary digital video streams 210 are time-synchronised.
  • This time-synchronisation may be with respect to a common time reference 260.
  • the time-synchronisation may involve aligning the primary video streams 210, 301 in relation to each other, for instance using said common time reference 260, so that they can be combined to form a time-synchronised context.
  • the common time reference 260 may be a stream of data, a heartbeat signal or other pulsed data, or a time anchor applicable to each of the individual primary video streams 210, 301.
  • the common time reference can be applied to each of the individual primary video streams 210, 301 in a way so that the informational contents of the primary video stream 210, 301 in question can be unambiguously related to the common time reference with respect to a common time axis.
  • the common time reference may allow the primary video streams 210, 301 to be aligned, via time shifting, so as to be time-synchronised in the present sense.
  • the time- synchronisation may be based on known information about a time difference between the primary video streams 210, 301 in question, such as based on measurements.
  • the time-synchronisation may comprise determining, for each 5 primary video streams 210, 301, one or several timestamps 261, such as in relation to the common time reference 260 or for each video stream 210, 301 in relation to another video stream 210, 301 or to other video streams 210, 301.
  • w the hence time-synchronised primary digital video streams 210, 301 are analysed to detect at least one pattern 212 selected from a first set of patterns. This is illustrated in Figure 6e.
  • the pattern detection step may preferably be performed based on video and/or audio information contained as a part of at least two of the is time-synchronised primary video streams 210, 301 considered jointly.
  • Said first set of patterns may contain any number of types of patterns, such as several participants talking interchangeably or concurrently; or a presentation slide change occurring concurrently as a different event, such as a different participant talking. This list is not ex-0 haustive, but illustrative.
  • detected patterns 212 may relate not to information contained in several of said primary video streams 210, 301 but only in one of said primary video streams 210, 301. In such cases, it is preferred that such pattern 212 is detected based on5 video and/or audio information contained in that single primary video stream 210, 301 spanning across at least two detected events 211, for instance two or more consecutive detected presentation slide changes or connection quality changes. As an example, several consecutive slide changes that follow on each other rapidly over time may be detected as one single slide change pattern, as opposed to one individual slide change pattern for each0 detected slide change event.
  • the first set of events and said first set of patterns may comprise events/patterns being of predetermined types, defined using respective sets of parameters and parameter intervals. As will be explained below, the events/patterns in said sets may also, or additionally, be defined and detected using various Al tools.
  • the shared digital video stream is produced as an output digital video stream 230 based on consecutively considered frames 213 of the time-synchronised primary digital video streams 210, 301 and said detected patterns 212.
  • the present invention allows for the completely automatic production of video streams, such as of the output digital video stream 230.
  • such production may involve the selection of what video and/or audio information from what primary video stream 210, 301 to use to what extent in such output video stream 230; a video screen layout of an output video stream 230; a switching pattern between different such uses or layouts across time; and so forth.
  • Figure 6f shows one or several additional pieces of time-related (that may be related to the common time reference 260) digital video information 220, such as an additional digital video information stream, that can be time-synchronised (such as to said common time reference 260) and used in concert with the time-synchronised primary video streams 210, 301 in the production of the output video stream 230.
  • the additional stream 220 may comprise information with respect to any video and/or audio special effects to use, such as dynamically based on detected patterns; a planned time schedule for the video communication; and so forth.
  • the produced0 output digital video stream 230 is continuously provided to a consumer 110, 150 of the produced digital video stream as described above.
  • the produced digital video stream may be provided to one or several participant clients 121, such as via the video communication service 110.
  • the method ends.
  • the method may iterate any number 5 of times, as illustrated in Figure 5, to produce the output video stream 230 as a continuously provided stream.
  • the output video stream 230 is produced to be consumed in real-time or near real-time (taking into consideration a total latency added by all steps along the way), and continuously (publishing taking place immediately when more information is available, however not counting the below-described deliberately added latencies or de- w lays).
  • the output video stream 230 may be consumed in an interactive manner, so that the output video stream 230 may be fed back into the video communication service 110 or into any other context forming a basis for the production of a primary video stream 210 again being fed to the collection function 131 so as to form a closed feedback loop; or so that the output video stream 230 may be consumed into a different (external to system is 100 or at least external to the central server 130) context but there forming the basis of a real-time, interactive video communication.
  • the collecting step may comprise collecting at least one of said primary digital video streams 210 from the shared digital video communication service 110 itself, such as via an5 automatic participant client 140 in turn being granted access to video and/or audio stream data from within the video communication service 110 in question; and/or via an API 112 of the video communication service 110.
  • the collecting step may comprise collecting at least one0 of said primary digital video streams 210, 301 as a respective external digital video stream 301, collected from an information source 300 being external to the shared digital video communication service 110. It is noted that one or several used such external video sources 300 may also be external to the central server 130.
  • the primary video streams 210, 301 are not formatted in the same s manner. Such different formatting can be in the form of them being delivered to the collecting function 131 in different types of data containers (such as AVI or MPEG), but in preferred embodiments at least one of the primary video streams 210, 301 is formatted according to a deviating format (as compared to at least one other of said primary video streams 210, 301) in terms of said deviating primary digital video stream 210, 301 having a w deviating video encoding; a deviating fixed or variable frame rate; a deviating aspect ratio; a deviating video resolution; and/or a deviating audio sample rate.
  • a deviating format as compared to at least one other of said primary video streams 210, 301
  • the collecting function 131 is preconfigured to read and interpret all encoding formats, container standards, etc. that occur in all collected primary video streams is 210, 301. This makes it possible to perform the processing as described herein, not requiring any decoding until relatively late in the process (such as not until after the primary stream in question is put in a respective buffer; not until after the event detection step; or even not until after the event detection step).
  • the collecting function 131 may be arranged to perform a decoding and analysis of such primary video stream 210, 301, followed by a conversion into a format that can be handled by, for instance, the event detection function. It is noted that, even in this case, it is preferred not to perform any reencoding at this stage. 5
  • primary video streams 220 being fetched from multi-party video events such as one provided by the video communication service 110, typically have requirements on low latency and are therefore typically associated with variable framerate and variable pixel resolution to enable participants 122 to have an effective communication. In other words, overall video and audio quality will be decreased as necessary for the sake of low latency.
  • External video feeds 301 will typically have a more stable framerate, higher quality but therefore possibly higher latency.
  • the video communication service 110 may, at each moment in time, use a different encoding and/or container than the external video source 300.
  • the analysis and video production process described herein in this case therefore needs to combine these streams 210, 301 of different formats into a new one for the combined experience.
  • the collecting function 131 may comprise a set of format-specific col- lecting functions 131a, each one arranged to process a primary video stream 210, 301 of a particular type of format.
  • each one of these format-specific collecting functions 131a may be arranged to process primary video streams 210, 301 having been encoded using a different video respective encoding method/codec, such as Windows® Media® or DivX®.
  • the collecting step comprises converting at least two, such as all, of the primary digital video streams 210, 301 into a common protocol 240.
  • the term "protocol” refers to an information-structuring standard or data structure specifying how to store information contained in a digital video/audio stream.
  • the common protocol preferably does not, however, specify how to store the digital video and/or audio information as such on a binary level (i.e. the encoded/compressed data instructive of the sounds and images themselves), but instead forms a structure of predetermined format for storing such data.
  • the common protocol pre-scribes storing digital video data in raw, binary form without performing any digital video decoding or digital video encoding in connection to such storing, possibly by not at all amending the existing binary form apart from possibly concatenating and/or splitting apart the binary form byte sequence.
  • the raw (encoded/compressed) binary data contents of the primary video stream 210, 301 in question is kept, while repacking this raw binary data in the data structure defined by the protocol.
  • the common protocol defines a video file container format.
  • Figure 7 illustrates, as an example, the primary video streams 210, 301 shown in Figure 6a, restructured by the respective format-specific collecting function 131a and using said common protocol 240.
  • the common protocol 240 prescribes storing digital video and/or audio data in data sets 241, preferably divided into discreet, consecutive sets of data along a time line pertaining to the primary video stream 210, 301 in question.
  • Each such data set may include one or several video frames, and also associated audio data.
  • the common protocol 240 may also prescribe storing metadata 242 associated with specified time points in relation to the stored digital video and/or audio data sets 241.
  • the metadata 242 may comprise information about the raw binary format of the primary 15 digital video stream 210 in question, such as regarding a digital video encoding method or codec used to produce said raw binary data; a resolution of the video data; a video frame rate; a frame rate variability flag; a video resolution; a video aspect ratio; an audio compression algorithm; or an audio sampling rate.
  • the metadata 242 may also comprise information on a timestamp of the stored data, such as in relation to a time reference of the primary0 video stream 210, 301 in question as such or to a different video stream as discussed above.
  • the collecting step may comprise using different ones of said format-specific collecting functions 131a for collecting primary digital video streams 210, 301 being encoded using different binary video and/or audio encoding formats, in order to parse the primary video0 stream 210, 301 in question and store the parsed, raw and binary data in a data structure using the common protocol, together with any relevant metadata.
  • the determination as to what format-specific collecting function 131a to use for what primary video stream 210, 301 may be performed by the collecting function 131 based on predetermined and/or dynamically detected properties of each primary video stream 210, 301 in question.
  • Each hence collected primary video stream 210, 301 may be stored in its own separate memory buffer, such as a RAM memory buffer, in the central server 130.
  • the converting of the primary video streams 210, 301 performed by each format-specific collecting function 131a may hence comprise splitting raw, binary data of each thus converted primary digital video stream 210, 301 into an ordered set of said smaller sets of data 241.
  • the converting may also comprise associating each (or a subset, such as a regu- larly distributed subset along a respective time line of the primary stream 210, 301 in question) of said smaller sets 241 with a respective time along a shared time line, such as in relation to said common time reference 260.
  • This associating may be performed by analysis of the raw binary video and/or audio data in any of the principle ways described below, or in other ways, and may be performed in order to be able to perform the subsequent time- synchronising of the primary video streams 210, 301.
  • at least part of this association of each of the data sets 241 may also or instead be performed by the synchronisation function 133.
  • the collecting step may instead comprise associating each, or a subset, of the smaller sets 241 with a respective time of a time line specific for the primary stream 210, 301 in question.
  • the collecting step also comprises converting the raw binary video and/or audio data collected from the primary video streams 210, 301 into a uniform quality and/or updating frequency. This may involve down-sampling or up-sampling of said raw, binary digital video and/or audio data of the primary digital video streams 210, 301, as nec- essary, to a common video frame rate; a common video resolution; or a common audio sampling rate. It is noted that such re-sampling can be performed without performing a full decoding/reencoding, or even without performing any decoding at all, since the formatspecific collecting function 131a in question can process the raw binary data directly according to the correct binary encoding target format.
  • Each of said primary digital video streams 210, 301 may be stored in an individual data storage buffer 250, as individual frames 213 or sequences of frames 213 as described above, and also each associated with a corresponding time stamp in turn associated with said common time reference 260.
  • the video communication service 110 is Microsoft® Teams®, running a video conference involving concurrent participants 122.
  • the automatic participant client 140 is registered as a meeting participant in the Teems® meeting.
  • the primary video input signals 210 are available to and obtained by the collecting function 130 via the automatic participant client 140. These are raw signals in H264 format and contain timestamp information for every video frame.
  • the relevant format-specific collecting function 131a picks up the raw data over IP (LAN0 network in the cloud) on a configurable predefined TCP port. Every Teems® meeting participant, as well as associated audio data, are associated with a separate port. The collecting function 131 then uses the timestamps from the audio signal (which is in 50 Hz) and downsamples the video data to a fixed output signal of 25 Hz before storing the video stream 220 in its respective individual buffer 250. 5
  • the common protocol 240 may store the data in raw binary form. It can be designed to be very low-level, and to handle the raw bits and bytes of the video/audio data.
  • the data is stored in the common protocol 240 as a simple byte array or corresponding data structure (such as a slice). This means that the data does not0 need to be put in a conventional video container at all (said common protocol 240 not constituting such conventional container in this context). Also, encoding and decoding video is computationally heavy, which means it causes delays and requires expensive hardware.
  • Video frame pixel width (int) 0 4 bytes 25.0 Video frame rate
  • the "Detected event in, if any" data is included as a part of the common protocol 260 specification.
  • this information may instead be put in a separate memory buffer.
  • said at least one additional piece of digital video information 220 that may be an overlay or an effect, is also stored in a respective individual buffer 250, as individual frames or sequences of frames each associated with a corresponding time stamp in turn associated with said common time reference 260.
  • the event detection step may comprise storing, using said common protocol 240, metadata 242 descriptive of a detected event 211, associated with the primary digital video stream 210, 301 in which the event 211 in question was detected.
  • the event detection step comprises a first trained neural network or other machine learning component analysing at least one, such as several or even all, of said primary digital video streams 210, 301 individually in order to automatically detect any of said events 211. This may involve the Al component 132a classifying, in a managed classification, the primary video stream 210, 301 data into a set of predefined events and/or, in an unmanaged classification, into a dynamically determined set of events.
  • the detected event 211 is a change of presentation slides in a presentation being or being comprised in the primary video stream 210, 301 in question.
  • the event detection step may comprise at least one of the following:
  • the event 211 can be detected based on an image analysis of a difference between a first image of a detected slide and a subsequent second image of a detected slide.
  • the nature of the primary video stream 220, 301 being that of showing a slide can be automat- ically determined using per se conventional digital image processing, such as using motion detection in combination with OCR (Optical Character Recognition).
  • This ap- proach also allows full configurability - for instance, sometimes it is desired to be able to capture computer mouse movement, for instance when the presenter wishes to present something in detail using the computer mouse to point to different things.
  • the event 211 may be detected based on an image analysis of an informational complexity of said second image itself, to determine the type of event with greater specificity.
  • This may, for instance, involve assessing a total amount of textual information on the slide in question, as well as associated font sizes. This may be done by using conventional OCR methods, such as deep learning-based character recognition techniques.
  • the event detection function 132 may call the relevant format- specific collecting function for image interpreting services, or the event detection function 132 may itself include functionality for assessing image information, such as on individual pixel level, for a number of different supported raw binary video data formats.
  • the detected event 211 is a loss of communication connection of a participant client 121 to a digital video communication service 110.
  • the detection step may comprise detecting that said participant client 121 has lost communication connection based on an image analysis of a series of subsequent video frames 213 of a primary digital video stream 210 corresponding to the participant client 121 in question. Because participant clients 121 are associated with different physical locations and different internet connections, it can happen that someone will lose connection to the video communication service 110 or to the central server 130. In that situation, it is desirable to avoid showing a black or empty screen in the produced output video stream 230.
  • connection loss can be detected as an event by the event detection function s 132, such as by applying a 2-class classification algorithm where the 2 classes used are connected / not connected (no data).
  • no data differs from the presenter sending out a black screen intentionally. Because a brief black screen, such as of only 1 or 2 frames, may not be noticeable in the end production stream 230, one may apply said 2-class classification algorithm over time to create a time series. Then, a threshold w value, specifying a minimum length for a connection interruption, can be used to decide whether there is a lost connection or not.
  • detected events of these exemplified types may be used by the pattern detection function 134 to take various actions, as suitable and desired.
  • the individual primary video streams 210, 301 may each be related to the common time reference 260 or to each other in the time domain, making it possible for the synchronisation function 133 to time-synchronise them in relation to each other.
  • the common time reference 260 is based on or comprises a common audio signal 111 (see Figures 1-3), the common audio signal 111 being common for the shared digital video communication service 110 involving at least two remotely connected participant clients 121 as described above, each providing a respective one of said primary digital video streams 210. 5
  • a common audio signal is produced and can be captured by the central server 130 via the automatic participant client 140 and/or via the API 112.
  • a common audio signal may be used as a heartbeat signal to time-synchronise the individual primary video streams 220 by0 binding each of these to specific time points based on this heartbeat signal.
  • Such a common audio signal may be provided as a separate (in relation to each of the other primary video streams 210) signal, whereby the other primary video streams 210 may each be individually time-correlated to the common audio signal, based on audio contained in the other primary video stream 210 in question or even based on image information contained therein (such as using automatic image processing-based lip syncing techniques).
  • such a common audio signal is used as a heartbeat for all primary video streams 210in the central server 130 (but perhaps not external primary video streams 301).
  • all other signals are mapped to this common audio time heartbeat to make sure that everything is in time sync.
  • the time-synchronisation is achieved using a time synchronisation element 231 introduced into the output digital video stream 230 and detected by a respec- tive local time-synchronising software function 125 provided as a part of one or several individual ones of the participant clients 121, the local software function 125 being arranged to detect a time of arrival of the time synchronisation element 231 in the output video stream 230.
  • the output video stream 230 is fed back into the video communication service 110 or otherwise made available to each participant client 121 and the local software function 125 in question.
  • the time synchronisation element 231 may be a visual marker, such as a pixel changing colours in a predetermined sequence or manner, placed or updated in the output video 230 at regular time intervals; a visual clock updated and displayed in the output video 230; a sound signal (that may be designed to be non-audible to participants 122 by, for instance, having low enough amplitude and/or high enough frequency) and added to an audio forming part of the output video stream 230.
  • the local software function 125 is arranged to, using suitable image and/or audio processing, automatically detect respective times of arrival of each of the (or each of the) time synchronisation element(s) 231. Then, the common time reference 260 may be determined at least party based on said detected times of arrival.
  • each of the local software functions 125 may communicate to the central server 130 respective information signifying said detected time of arrival. s Such communication may take place via a direct communication link between the participant client 121 in question and the central server 130. However, the communication may also take place via the primary video stream 210 associated with the participant client 121 in question. For instance, the participating client 121 may introduce a visual or audible code, such as of the above discussed type, in the primary video stream 210 produced by that w participant client 121 in question, for automatic detection by the central server 130 and used to determine the common time reference 260.
  • each participant client 121 may perform an image detection in a common video stream available for viewing by all participant clients 121 to the video com- 75 munication service 110 and relay the results of such image detection to the central server 130, in a way corresponding to the ones discussed above, to there be used to over time determine respective offsets of each participant client 121 in relation to each other.
  • a common time reference 260 may be determined as a set of individual relative offsets. For instance, a selected reference pixel of a commonly available video stream may be mon-0 itored by several, or all, participating clients 121, such as by said local software function 125, and a current colour of that pixel may be communicated to the central server 130.
  • the central server 130 may calculate a respective time series based on consecutively received such colour values from each of a number of (or all) the participant clients 121, and perform a cross-correlation resulting in an estimated set of relative time offsets across the different5 participant clients 121.
  • the output video stream 230 fed into the video communication service 110 may be included as a part of a shared screen every participant client of the video communication in question, and may therefore be used to assess such time offset associated with the par-0 ticipant clients 121.
  • the output video stream 230 fed to the video communication service 110 may be available again to the central server via the automatic participant client 140 and/or the API 112.
  • a common time reference 260 may be determined at least partly s based on a detected discrepancy between an audio part 214 of a first one of said primary digital video streams 210, 301 and an image part 215 of said first primary digital video streams 210, 301.
  • Such discrepancy may, for instance, be based on a digital lip sync video image analysis of a talking participant 122 viewed in said first primary digital video stream 210, 301 in question.
  • lip sync analysis is conventional as such, and may for instance w use a trained neural network.
  • the synchronisation step comprises deliberately introducing a delay (in this context the terms "delay” and "latency” are intended to mean the same thing) of at the most 30 seconds, such as at the most 5 seconds, such as at the most 1 seconds, such as at the most 0.5 seconds, but longer than 0 s, so that the output digital video stream 230 is provided at least with said delay.
  • a delay in this context the terms "delay” and "latency” are intended to mean the same thing
  • the deliberately introduced delay is at least0 several video frames, such as at least three, or even at least five or even 10, video frames, such as this number of frames (or individual images) stored after any resampling in the collecting step.
  • the term "deliberately” means that the delay is introduced irrespective of any need for introducing such a delay based on synchronisation issues or similar.
  • the deliberately introduced delay is introduced in addition to any5 delay introduced as a part of the synchronisation of the primary video streams 210, 301 in order to time-synchronise them one in relation to the other.
  • the deliberately introduced delay may be predetermined, fixed or variable in relation to the common time reference 260.
  • the delay time may be measured in relation to a least latent one of the primary video streams 210, 301, so that more latent ones of these streams 210, 301 as a result of said0 time-synchronisation are associated with a relatively smaller deliberately added delay.
  • a relatively small delay is introduced, such as of 0.5 seconds or less. This delay will barely be noticeable by participants to a video communication service 110 using the output video stream 230.
  • a larger delay may be introduced.
  • This deliberately introduced delay may be enough so as to achieve sufficient time for the synchronisation function 133 to map the collected individual primary stream 210, 301 video frames onto the correct common time reference 260 timestamp 261. It may also be enough w so as to allow sufficient time to perform the event detection described above, in order to detect lost primary stream 210, 301 signals, slide changes, resolution changes, and so forth. Furthermore, deliberately introducing said delay may be enough so as to allow for an improved pattern detection function 134, as will be described in the following. is It is realized that the introduction of said delay may involve buffering 250 each of the collected and time-synchronised primary video streams 210, 301 before publishing the output video stream 230 using the buffered frames 213 in question.
  • video and/or audio data of at least one, several or even all of the primary video streams 210, 301 may then be present in the central server 130 in a buffered manner, much like a cache but not0 (like a conventional cache buffer) used with the intention of being able to handle varying bandwidth situations but for the above reasons, and in particular to be used by the pattern detection function 134.
  • said pattern detection step comprises taking into considera-5 tion certain information of at least one, such as several, such as at least four, or even all, of the primary digital video streams 210, 301, the certain information being present in a later frame 213 than a frame of a time-synchronised primary digital video stream 210 yet to be used in the production of the output digital video stream 230.
  • a newly added frame 213 will exist in the buffer 250 in question during a particular latency time before forming0 part of (or basis for) the output video stream 230.
  • the information in the frame 213 in question will constitute information in the "future" in relation to a currently used frame to produce a current frame of the output video stream 230.
  • the output video stream 230 timeline reaches the frame in question 213, it will be used for the production of the corresponding frame of the output video stream 230, and may thereafter be discarded.
  • the pattern detection function 134 has at its disposal a set of video/audio frames 213 that have still not been used to produce the output video stream 230, and may use this data to detect said patterns.
  • the pattern detection can be performed in different ways.
  • the pattern detection step comprises a second trained neural network or other machine learning component analysing at least two, such as at least three, such as at least four, or even all, of said primary digital video streams 120, 301 in concert to automatically detect said pattern 212.
  • the detected pattern 212 comprises a speaking pattern involving at least two, such as at least three, such as at least four, different speaking participants 122, each associated with a respective participant client 121, to the shared video communication service 110, each of said speaking participants 122 being viewed visually in a respective one of said primary digital video streams 210, 301.
  • the production step preferably comprises determining, keeping track of and updating a current production state of the output video stream 230.
  • a state can dictate what, if any, participants 122 are visible in the output video stream 230, and where on the screen; if any external video stream 300 is visible in the output video stream 230, and where on the screen; if any slides or shared screens are shown in full-screen mode or in combination with any live video streams; and so on.
  • the production function 135 can be viewed as a state machine with respect to the produced output video stream 230.
  • the central server 130 To generate the output video stream 230 as a combined video experience to be viewed by, for instance, an end consumer 150, it is advantageous for the central server 130 to be able to understand what happens on a deeper level than merely detecting individual events associated with individual primary video streams 210, 301.
  • a presenting participant client 121 is changing a currently viewed slide. s This slide change is detected by the event detection function 132 as described above, and metadata 242 is added to the frame in question indicative of a slide change having happened. This happens a number of times, since the presenting participating client 121 turns out to skip a number of slides forward in rapid succession, resulting in a series of "slide change" events detected by the even detection function 132 and stored with corresponding w metadata 242 in the individual buffer 250 for the primary video stream 210 in question. In practice, each such rapidly forward skipped slide may be visible for only a fraction of a second.
  • the pattern detection function 134 looking at the information in the buffer 250 in question, is spanning across several of these detected slide changes, will detect a pattern corresponding to one single slide change (that is, to the last slide in the forward-skipping, the slide remining visible once the rapid skipping is finished), rather than a number or rapidly performed slide changes.
  • the pattern detection function 134 will note that there are, for instance, ten slide changes in a very short period of time, why they will be handled as a0 detected pattern signifying one single slide change.
  • the production function 135, having access to the patterns detected by the pattern detection function 134, may choose to show the final slide in full-screen mode in the output video stream 230 for a couple of seconds, since it determines this slide to be potentially important in said state machine. It may also choose not to show the intermediately viewed slides at all in the output stream5 230.
  • the detection of the pattern with several rapidly changing slides may be detected by a simple rule-based algorithm, but may alternatively be detected using a trained neural network designed and trained to detect such patterns in moving images by classification.
  • a trained neural network designed and trained to detect such patterns in moving images by classification.
  • the event detection function 132 can continuously analyse each primary video stream 210, 301 to at all times determine whether or not a person being viewed in that particular primary video stream 210, 301 is currently speaking or not.
  • the pattern detection function 134 may be operable to detect w particular overall patterns, involving several of said primary video streams 210, 301, said patterns being useful for producing a smooth output video stream 230.
  • the pattern detection function 134 may detect a pattern of very frequent switches between a current speaker and/or patterns involving several concurrent speakers.
  • the production function 135 can take such detected patterns into consideration when taking automated decisions in relation to said production state, for instance by not automatically switching visual focus to a speaker who only speaks for half a second before again going silent, or to switch to a state where several speakers are displayed side by side during a certain time period when both are speaking interchangeably or concurrently.
  • This state0 decision process may in itself be performed using time series pattern recognition techniques, or using a trained neural network, but can also be based at least partly on a predetermined set of rules.
  • Such multiple patterns may be used by different Al components, computer vision detecting algorithms, and so forth, by the production function 135.
  • permanent slide changes can be detected while concurrently detecting unstable connections of some participant clients 121, while other patterns detect a current main speaking participant 122.
  • a classifier neural network can be trained, and/or a set of rules can be developed, for analysis of a time series of such pattern data.
  • Such a classification may be at least partly, such as completely, supervised to result in determined desired state changes to be used in said production.
  • different such predetermined classifiers can be produced, specifically arranged to automatically produce the output video stream 230 according to various and different production styles and desires.
  • Training may be based on known production state change sequences as desired outputs and known pattern time series data as training data.
  • a Bayesian model can be used to produce such classifiers.
  • information can be a priori gleaned from an experienced producer, providing input such as "in a talkshow I never switch from speaker A to Speaker B directly but always first show an overview before I focus on the other speaker, unless that the other speaker is very dominant and speaking loud.”
  • This production logic then be represented as a Bayesian model on the general form "if X is true
  • the production step may comprise producing the output digital video 5 stream 230 based on a set of predetermined and/or dynamically variable parameters regarding visibility of individual ones of said primary digital video streams 210, 301 in said output digital video stream 230; visual and/or audial video content arrangement; used visual or audio effects; and/or modes of output of the output digital video stream 230.
  • Such parameters may be automatically determined by said production function 135 state maze chine and/or be set by an operator controlling the production (making it semi-automatic) and/or be predetermined based on certain a priori configuration desires (such as a shortest time between output video stream 230 layout changes or state changes of the above-exemplified types).
  • the state machine may support a set of predetermined standard layouts that may be applied to the output video stream 230, such as a full-screen presenter view (showing a current speaking participant 122 in full-screen); a slide view (showing a currently shared presentation slide in full-screen); "butterfly view", showing both a currently speaking participant 122 together with a currently shared presentation slide, in a0 side-by-side view; a multi-speaker view, showing all or a selected subset of participants 122 side-by-side or in a matrix layout; and so forth.
  • Various available production formats can be defined by a set of state machine state changing rules (as exemplified above) together with an available set of states (such as said set of standard layouts).
  • one such production format may be "panel discussion", another "presentation”, and so forth.
  • select-5 ing a particular production format via a GUI or other interface to the central server 130 an operator of the system 100 may quickly select one of a set of predefined such production formats, and then allow the central server 130 to, completely automatically, produce the output video stream 230 according to the production format in question, based on available information as described above.
  • a respective in-memory buffer is created and maintained, as described above, for each meeting participant client 121 or external video source 300. These buffers can easily be removed, added, and changed on the fly.
  • the central server 130 can then be arranged to receive information, during the production of the output video stream 230, regarding added/dropped-off participant clients 121 and participants 122 scheduled for delivering speeches; planned or unexpected pauses/resumes of presentations; desired changes to the currently used production format, and so forth. Such information may, for instance, be fed to the central server 130 via an operator GUI or interface, as described above.
  • At least one of the primary digital video streams 210, 301 is provided to the digital video communication service 110, and the publishing step may then comprise providing said output digital video stream 230 to that same communication service 110.
  • the output video stream 230 may be provided to a participant client 121 of the video communication service 110, or be provided, via API 112 as an external video stream to the video communication service 110. This way, the output video stream 230 may be made available to several or all of the participants to the video communication event currently being achieved by the video communication service 110.
  • the output video stream 230 may be provided to one or several external consumers 150.
  • the production step may be performed by the central server 130, providing said output digital video stream 230 to one or several concurrent consumers as a live video stream via the API 137.
  • the participating clients 121 may be organised into two or more groups of participating clients 121.
  • Figure 8 is a simplified view of the system 100 in a configuration to perform automatic production of an output video stream in cases where there are such groups.
  • the central server 130 comprises the collecting function 131, that may be as described above.
  • the central server 130 also comprises a first production function 135', a second production s function 135” and a third production function 135'”.
  • Each such production function 135', 135”, 135'” corresponds to the production function 135, and what has been said above in relation to production function 135 applies equally to production functions 135', 135” and 135'”.
  • the production functions 135', 135”, 135'” may be distinct or co-arranged in one single logical function with several functions, and there may also be more than three pro- w duction functions, depending on the detailed configuration of the central server 130.
  • the production functions 135', 135”, 135'” may in some cases be different functional aspects of one and the same production function 135, as the case may be.
  • the various communications between the production functions 135', 135”, 135'” and other entities may take place via suitable APIs.
  • the central server 130 comprises a first publishing function 136', a second publishing function 136” and a third publishing function 136'”.
  • Each such publishing function 136', 136”, 136'” corresponds to the publishing function 136, and what has been said above in relation to publishing function 136 applies equally to publishing functions 136', 136” and5 136'”.
  • the publishing functions 136', 136”, 136'” may be distinct or co-arranged in one single logical function with several functions, and there may also be more than three publishing functions, depending on the detailed configuration of the central server 130.
  • the publishing functions 136', 136”, 136'” may in some cases be different functional aspects of one and the same publication function 136, as the case may be.
  • each of these groups may comprise one or, preferably, at least two, participating clients. There may be only two such groups, or more than three such groups, depending on the detailed configuration.
  • the allocation between groups 121', 121”, 121'” may be exclusive in the sense that each participant client 121 is allocated to maximum one such group 121', 121”, 121'”. In alternative configurations, at least one participant client 121 may be allocated to more than one such group 121', 121”, 121'” at the same time.
  • Figure 8 also shows an external consumer 150, and it is realised that there may be more than one such external consumer 150 as described above.
  • Figure 8 does not, for reasons of simplicity, show the video communication service 110, but it is realised that a video communication service of the above-discussed general type may be used with the central server 130, such as providing a shared video communication service to each participating client 121 using the central server 130 in the way discussed above.
  • Respective primary video streams may be collected, by collecting function 131, from the respective participating clients 121, such as participating clients of said groups 121', 121”, 121'”. Based on the provided primary video streams, production functions 135', 135”, 135'” may produce respective digital video output streams.
  • one or several such produced output streams may be provided as respective input digital video streams from one or several respective production functions 135', 135'” to another production function 135”, that may in turn produce a secondary digital output video stream for publication by publication function 136”, the secondary digital output video stream thus being produced based on one or several input primary digital video streams as well as on one or several pre-produced digital input digital video streams.
  • the two or more different production steps 135', 135”, 135'” may comprise the introduction of a respective time-delay.
  • one or several of the respective produced output digital video streams from these production steps 135', 135”, 135'” may be time-unsynchronised, due to said time-delay introduction, with any other of the video s streams that may be provided to other participant clients in said publishing step.
  • Such timedelays may be deliberately added and/or be a direct consequence of the production of the produced digital video stream in question, in any of the ways described herein.
  • any participating client consuming said time-unsynchronised produced output digital video stream will do so in a "time zone" which is slightly offset (time-wise) in relation to a video w stream consumption "time zone" of said other participating clients.
  • one of the groups 121', 121”, 121'” of participating clients 121 may consume a respective produced video stream in a first such "time zone”, whereas participating client 121 of another one of said groups 121', 121”, 121'” may consume a respective produced is video stream in a second such "time zone”. Since both these respective produced video streams may be produced based at least partly on the same primary video streams, all such participant clients 121 will be active in the same video communication, but in different "time zones" in relation to each other. In other words, a respective timeline for produced video stream consumption may be time-wise offset between different groups 121', 121”,0 121'”.
  • each participating client 121 in one or several of said groups 121', 121”, 121'” may be able to interact with each other at the same perceived time-delay.
  • groups being associated with a larger respective time-delay may use a produced video stream from a group with a smaller time-delay as an input video stream to be used when producing an output video stream that the larger time-delay group views in their later "time-zone".
  • the result of this first larger time-delay production is hence a produced digital video stream of the above-described type, that may for instance visually comprise one or several of the primary video streams in question, in processed or non-processed form, as sub-parts.
  • This produced video stream in question may comprise live-captured video streams, slides, externally provided video or imagery, and so forth, as generally described above in relation to video output streams produced by the central server 130.
  • the produced video stream in question may also be produced based on detected events and/or patterns of deliberately delayed or real-time input primary video streams, provided by participant clients 121, in the general way described above.
  • the first group 121' participant clients are part of a debate panel, communicating using the video communication service 110 at a relative low latency, each one of these participant clients continuously being fed produced video stream from publishing step 136' (or the respective primary video stream of each other).
  • An audience to the debate panel is constituted by the second group 121” participant clients, continuously being fed the produced video stream from production step 135”, in turn being associated with a slightly higher latency.
  • the produced video stream from production step 135” may be automatically produced, in the general manner discussed above, to automatically shift between views of individual debate panel speakers (participant clients allocated to the first group 121', such views being provided directly from the collection function 131) and a produced view showing all debate panel speakers (this view being the first produced video stream).
  • the delay deliberately added to the respective primary video streams used in the production step 136” may be at least 0.1 s, such as at least 0.2 s, such as least 0.5 s; and may be at the most 5 s, such as at the most 2 s, such as at the most 1 s. It may also depend on an inherit latency associated with each respective primary video stream, so as to achieve a full time-synchronisation between each of the used primary video streams and also the produced video stream incoming to production step 135” from production step 135'.
  • Figure 8 furthermore illustrates a number of alternative or concurrent ways of publishing the various produced video streams produced by the central server 130.
  • a publishing step performed by a first publishing function 136' arranged to 15 receive first produced video stream from the first production function 135', said first produced video stream may be continuously provided to at least one of a first participating client 121 and a second participating client 121.
  • this first participating client may be a participating client from the group 121' providing a respective primary digital video stream to the first production function 135'.
  • one or several of the participating clients of the group 121' may also receive a second produced video stream, by the second publishing function 136”, in turn being arranged to receive a second produced video stream from the second production function 135”. 5
  • each of the primary video stream-providing participating clients being allocated to the first group 121' may be provided with the first produced video stream, involving a certain delay or latency due to the synchronisation between said primary video streams and also a possibly0 deliberately added delay or latency added in order to allow sufficient time for an event and/or pattern detection, as described above.
  • each of the participating clients being allocated to the second group 121” may be provided with the second produced video stream, also including said deliberately added delay in connection to the second production step, added with the purpose of time- 5 synchronising the first produced video stream with said first and second primary video streams.
  • first group 121' participant clients form a sub-group of all participating clients 121 currently participating in the video communication service 110 in question, being present in and using the service in question in a "time zone" that is slightly ahead (such as 1-3 seconds ahead) of any participating client instead being continuously provided a produced video stream, such as the first or the second produced video stream.
  • participat- 75 ing clients (not allocated to the first group 121' but instead to the second group 121”) will be continuously provided the second produced video stream, which is produced based on (and may at each point in time contain either or both of) at least one of the primary video streams which the second produced video stream is produced based on, but then in a slightly later "time zone". Since the first produced video stream is produced directly based0 on these at least one primary video streams, without any delay or latency added to time- synchronise them to an already-produced video stream based on the primary video streams themselves, a more direct, low-latency video communication service 110 experience is then made available to these participating clients 121. 5 Again, this may also mean that the participating clients 121 allocated to the first group 121' are not provided access to the second produced video stream.
  • the second produced video stream may also be produced as a digital video stream based additionally on a produced output video (a third produced output0 digital video stream) of a third production step 135'”.
  • Each of the production steps 135', 135”, 135'” may hence be produced based on at least partly overlapping primary input digital video streams, but using different respective deliberately added latencies ("time zones") in relation to said primary video streams and provided to different respective groups 121', 121”, 121'” of participant clients 121.
  • the participant clients 121 allocated to the third group 121” may have less strict latency requirements than the participant clients 121 allocated to the first group 121'.
  • the first group 121' participating clients 121 may be the members of the above-discussed debate panel (interacting with each other in real-time hence requiring low latency), whereas the third group 121” participating clients 121 may constitute a panel of experts or w similar, that interact with the panel but in a more structured way (such as using clear ques- tions/answers), thus being able to tolerate higher latency than the first group 121'.
  • Both the first produced video stream and the third produced video stream may be fed to the second production function 135” to base the production of the second produced video is stream on, as the case may be.
  • the first production step 135' may involve introducing a deliberate delay or latency of the type discussed above, introduced in addition to any delay introduced as a part of the synchronisation of the first and second primary video streams, and introduced so as to0 achieve sufficient time, for instance, to perform an efficient event and/or pattern detection.
  • Such introduction of a deliberate delay or latency may take place as a part of a synchronising performed by said synchronising function 133 (not shown in Figure 8 for reasons of simplicity).
  • the corresponding may be true for the third production step 135'”, but introducing a deliberate delay or latency that is different from the deliberately introduced delay or la-5 tency for the first production step 135'.
  • the deliberately introduced delays or latencies result in a time-unsynchronisa- tion between the first produced video stream and the third produced video stream.
  • the first and third produced video streams do not follow a common timeline in0 case they are both immediately and continuously published upon production of each individual frame.
  • three separate produced video streams may be produced and consumed/published simultaneously, but in different "time zones". Even though they are based on at least partly the same primary video material, the produced video streams are published with different 5 latency.
  • the first group 121' requiring the lowest latency, can interact using the first produced video stream, offering very low latency; the third group 121'”, willing to accept slightly larger latency, can interact using the second produced video stream, offering more latency but on the other hand greater flexibility in terms of deliberately added delays so as to achieve a better automatic production as described elsewhere herein; whereas the sec- w ond group 121”, that are not so sensitive to latency, can enjoy interaction using the second produced video stream that can both incorporate material from the first 121' and third 121'” groups and also be automatically produced in a very flexible manner.
  • respective participating clients 121 allocated to each of said groups0 121', 121”, 121'” may participate in one and the same video communication service 110 within which the second produced video stream is continuously published.
  • different ones of said groups 121', 121”, 121'” may be associated with different participator interaction rights in said video communication service 110.
  • different ones of said groups 121', 121”, 121'” may also be associated with different maximum time-delays (latencies) used for producing a respective produced video stream published to participant clients 121 allocated to the group 121', 121”, 121'” in question.
  • the first group 121' of panel debating participant clients may be associated with full interaction rights, and can speak whenever they wish to.
  • the third group 121'” of participant clients may be associated with slightly restricted interaction rights, for instance that they need to request the floor before being able to speak by the video communication service 110 unmuting their mics.
  • the second group 121” of audience participant users may be associated with even more restricted interaction rights, such as only being able to pose 5 questions in writing in a common chat room but not being able to speak.
  • various groups of participant users may be associated with different interaction rights and different latencies for the respective produced video stream that is published to them, in a way so that latency is an increasing function of decreasing interaction rights.
  • the w freer the participant user 121 in question is allowed, by the video communication service 110, to interact with other users, the lower the accepted latency.
  • the lower the accepted latency the smaller the possibilities for the corresponding automatic production function to take into considerations things such as detected events or patterns. is
  • a group having a largest latency may be a viewer-only group, with no interaction rights apart from passive participation in the video communication service.
  • a respective maximum time-delay (latency) for each of said groups 121', 121”, 121”' may be determined as a largest latency difference across all primary video streams0 and any produced video streams that are continuously published to participating clients in the group in question. To this sum may be added any additional time-delay added deliberately with the purpose of detecting events and/or patterns as described above.
  • the term "production” and "produced digital video stream” may refer to5 different types of production.
  • a single, well-defined digital video stream is produced by a central entity, such as the central server 130, to form the produced digital video stream in question, for provision to and publication at each of a particular set of participating clients 121 that are to consume the produced digital video stream in question.
  • different individual such participating clients 121 may view a slightly different0 version of the produced digital video stream in question.
  • the produced digital video stream may comprise several individual or combined digital video streams that the participating client 121 local software function 125 can allow the user 122 in question to switch between; arrange on the screen 124; or to configure or process in any other way.
  • time zone i.e. at what latency
  • participant clients 120 of the first group 121' are provided each other's primary video streams, described above in connection to Figure 8
  • first produced digital video stream being provided to these participant clients (in the sense that the time-synchronised set of raw or processed first and second primary digital video streams are made available to both the first and second participant clients).
  • participant clients groups 121', 121'', 121''' are experiencing interaction with each other in realtime, or at least near real-time (depending on unavoidable hardware and software latencies). These participant clients are provided video, including sound, from each other to achieve such interaction and communication between the users 122 in question.
  • the first group 121' may serve users 122 at the core of the meeting, the interaction of which may be of interest for other (non-first group 121') participant clients to take part of.
  • a second group of such other participant clients 121'' participate in the same meeting, but in a different "time zone", being further from real-time than the first group participant clients 121'.
  • the second group 121'' may be an audience with interaction privi- privileges such as the possibility to pose questions to the first group 121'.
  • the "time zone" of the second group 121'' may have a delay in relation to the "time zone" of the first group 121' such that posed questions and answers are associated with a noticeable but short delay.
  • this slightly larger delay allows this second group 121'' participant clients to experience a produced digital video stream that was automatically produced in a more complex manner, providing a more agreeable user experience.
  • a third group of such other participant clients 121'” also participate in the same meeting, but as viewers only.
  • This third group 121'” consume a produced digital video stream that can be automatically produced in an even more elaborate and complex manner, being consumed in a third "time zone” having even more delay than the second "time zone”.
  • the third group 121'” will experience the meeting as being performed in "real-time", with an agreeable production.
  • Figure 9 illustrates a method according to the present invention, the method being for providing a shared digital video stream.
  • a first step SI the method starts.
  • a first digital video stream is collected from a first digital video source
  • a second digital stream is collected from a second digital video source, in0 the way generally described above.
  • the first and/or second digital video stream may each be collected from a respective participating client 121 or from an external source 300, and may be performed by the collecting function 131 of the central server 130.
  • said shared digital video stream is produced as an5 output digital video stream.
  • This production may be performed as generally described above, such as by production step 135, 135', 135”, 135'”.
  • the shared digital video stream is produced based on consecutively considered frames of said first digital video stream so that image information0 from said first digital video source is visible in the shared digital video stream, but so that image information from said second digital video source is not visible in the shared digital video stream.
  • the shared video stream will to at least some extent contain visual material originating from the first digital video stream, but will not contain any visual material originating from the second digital video stream.
  • a subsequent trigger detection step S5 said first and/or second digital video stream is digitally analysed so as to detect at least one trigger.
  • This analysis and detection may be performed by the same production step 135, 135', 135”, 135'” that produces the shared video stream in question, and is based on the automatic w detection of an image and/or audio pattern of a predetermined type.
  • the trigger may be an event or pattern of the types described and exemplified above (performed by the event detection function 132 and pattern detection function 134, respectively), and the detection thereof will typically take place using automatic digital processing is of audio and/or image/video data contained in the digital video stream(s) in question.
  • automatic image processing algorithms such as using a trained neural network or other machine learning tools, may be employed to automatically detect the presence of a certain trigger based on imagery contained in said first and/or second video feed.
  • conventional automatic audio processing algo-0 rithm of corresponding type may be used to detect the presence of a certain trigger based on audio contained in said first and/or second video feed.
  • That the image and/or audio pattern is of a "predetermined type" means that the pattern in question can be characterised in terms of a set of one or several absolute or relative5 parameter values that are defined ahead of the detection in question. This will be exemplified below.
  • the existence of said audio or image pattern constitutes the corresponding trigger in question.
  • the trigger is predetermined specifically for being instructive, to0 the automatic production step 135, 135', 135”, 135'” producing the shared video stream, according to a predetermined production rule, to alter the production mode (rule) of the shared digital video stream upon the detection of the trigger in question.
  • the production step 135, 135', 135”, 135'” may comprise or have access to a data- s base defining one or several triggers in terms of corresponding parameter values characterising a corresponding image and/or audio pattern, instantaneous or over time.
  • the shared digital video stream is then again produced, by the same (or differ- w ent) production step 135, 135', 135”, 135'”, but not in the same way as in the first production step S4.
  • the shared digital video stream is produced as an output digital video stream based on consecutively considered frames of said second is digital video stream, so that image information from said second digital video source is visible in the shared digital video stream.
  • the output digital video stream may in this case furthermore be produced based on, or be produced not based on, consecutively considered frames of the first digital video stream, so that image information from the first digital video source is visible in the shared digital video stream.
  • the shared video stream may go from displaying content from the first but not the second video stream, to either displaying content from the second but not the first video stream, or to display content from both the first and the second video streams.
  • the shared digital video stream is produced as an output digital video stream based on consecutively considered frames of said first digital video source but with at least one of a different cropping, a different zooming, a different panning and a the selection of a different focus plane of said first digital video stream as compared to the first production step S4.
  • the contents of the0 video stream being displayed in the shared video stream are cropped, de-cropped, zoomedin, zoomed-out, vertically and/or horizontally panned, and/or a focus plane of the video stream in question is shifted, in relation to a current crop/zoom/pan/focus plane status of the video stream as used in the first production step S4.
  • crop/pan/zoom/focus plane may be performed by the production 5 step 135, 135', 135”, 135'” in question based on an existing video stream (that may then itself contain multiple possible focus planes with different image information on pixel level), and/or by the production step 135, 135', 135”, 135'” communicating a command to a video source (such as a digital video camera) capturing the video stream in question) to modify the corresponding capture parameter accordingly.
  • a video source such as a digital video camera
  • this may then entail a cor- w responding camera capturing the video stream in question zooming, panning and/or shifting its focus plane in accordance with the instructions provided by the production step 135, 135', 135”, 135”'.
  • said output digital video stream is continuously provided is to a consumer of the shared digital video stream, such as to a participating client 121 and/or to an external consumer 150, in the general way described above.
  • step S9 the method ends.
  • the first digital video stream may be continuously captured by a first digital camera
  • the second digital video stream may be continuously captured by a second, different digital camera (hence constituting primary video streams using the terminology used previously5 herein).
  • the first and/or second digital video stream may constitute a respective previously produced digital video stream, in turn being produced with different latency ("time zone") as compared to the presently produced shared video stream (see above for0 details regarding such "time zones").
  • the first video stream is an already-produced video stream
  • an automatic production may be achieved of the shared video stream that can offer a more intuitive and natural experience to consumers of the produced shared video, since the actual audio/video contents of the respective video streams are used to detect triggers that in turn result in the automatic production moving from one automatic production format to a different such format.
  • Triggers may be predefined, using suitable parameters, to cater for various needs. For instance, behaviour of individual persons being depicted in the first and/or second video stream may be automatically assessed with respect to such triggers and allowed to change the production format depending on how such behaviour is conducted.
  • particular predefined triggers may be used as manual cues, delivered by a human being depicted in the first and/or second video stream, to change the production format in question on the fly, during an ongoing production. In the following, a number of examples of such triggers and corresponding production format changes will be described.
  • the predetermined pattern comprises a first (human or, for instance mechanical machine) participant 430 depicted in said first digital video stream, in Figure 10a illustrated as captured by a first digital video camera 410.
  • the first participant 430 gazes towards an object 440, in the example illustrated in Figure 10a a second (human or machine) participant, the second participant 440 in turn being depicted in the second digital video stream.
  • the second digital video stream is, in this example, captured by a second digital video camera 420.
  • the second object 440 may be something else, such as a group of human participants or any physical object of general interest for the ongoing communication.
  • the shared video stream may be a documentation of a medical procedure, whereby the object 440 may be a part of a patient.
  • the object 440 may s be the object of a teaching session or sales presentation.
  • the object 440 may also be, for instance, a whiteboard or slide presentation screen.
  • the first and second video streams may be of any of the types described hereinabove.
  • the predetermined image and/or audio pattern is detected based on information about a relative orientation of said first camera 410, said participant 430 and said object 440. This information may be present in the system 100 (and in particular in the central server 130), such as previously provided during setup/con- figuration and/or automatically detected during ongoing production.
  • respective locations of the first 410 and second cameras 420 may be fed to the central server 130 from the respective camera 410, 420 in question, by the cameras comprising measurement means, such as a MEMS circuit comprising an accelerometer and a gyro, or other per se conventional localisation means, such as stepping motors, arranged0 to continuously or intermittently measure a current location of the camera 410, 420 in question based on some suitable geometric reference.
  • the orientation of the cameras 410, 420 in question may be detected by a third camera (not shown), using a suitable automatic digital image processing algorithm, the third camera capturing at least one of the cameras 410, 420 in an image and determining, using digital image processing, the5 relative orientations in question based on this captured image information.
  • an "orientation" may encompass both a location and an orientation component.
  • the location of the first participant 430 and the object 440, in relation to some suitable reference system may be determined using digital image processing based on the video stream(s) captured by the first camera 410 and/or the second camera 420.
  • the first participant 430 gazes downwards in the Figure, not s towards the second participant 430.
  • the predetermined image and/or audio pattern is further detected based on a digital image-based determination of at least one of a body direction, a head direction and a gaze direction of the first participant 430, based on the first digital video w stream.
  • the first participant 430 has turned towards the second participant 440, facing (and gazing) towards the second participant 440.
  • Body and head direction of the first participant 430 may be determined based on digital image processing of the first video stream, captured by the first camera 410.
  • Such algorithms are convention as such, and can, for instance, use a priori knowledge about an expected shape of the first participant 430 in the video stream captured by the first camera 410 when turned in various directions. This may be accomplished using a trained neural0 network or other machine learning components.
  • the central server 130 can determine if the first participant 430 is turned (head or body) towards the object 440 in question or not. 5
  • Gaze direction of the first participant 430 may be accomplished in a similar manner, such as based on the images captured by the first video camera 410. Such gaze tracking technology is well-known in itself, and may for instance be based on discerning the position of pupils and light reflections visible in the eyes of the first participant 430.
  • the trigger in question may be defined as the detection of a transition pattern, such as the transition by the first participant 430 from not being directed or gazing towards the object 440 to the first participant 430 in fact facing and/or gazing towards the object 440.
  • the central server 130 may continuously monitor for such transition, based on suitably set s corresponding absolute or relative parameter values, and the trigger may be detected once such a transition occurs.
  • any such detected pre- w determined image and/or audio pattern will then constitute a detected corresponding trigger, in turn leading to the automatic production switching to a different respective corresponding mode, according to a predetermined parameterised production logic.
  • Figure 10c illustrates a second example, being similar to the one shown in Figure 10b, but is wherein the predetermined image and/or audio pattern corresponding to the trigger in question is not that the first participant 430 turns or gazes to the object 440. Instead, the predetermined image and/or audio pattern comprises a participant (such as the first participant 430) depicted in the first digital video stream (such as by the first camera 410) making a predetermined gesture. 0
  • the gesture may be any gesture of predetermined, parameterised type, such as a gesture in geometric relation to said object 440 depicted in said second digital video stream.
  • the gesture may be the first participant 430 pointing towards said object 440 (as depicted in Figure 10c showing an arm 431 of the first participant 430 pointing towards the5 object 440).
  • the gesture may, however, also be based on, for instance, only a hand or finger of the first participant 430.
  • This gesture-type predetermined pattern (and in particular its direction in space, if any) may be detected in a way corresponding to the situation described in connection to Figure 10b,0 hence based on information about a relative orientation of said first camera 410, said first 430 participant and said object 440, and further being detected based on a digital image- based determination of a gesture direction of the first participant 430 based on the first digital video stream.
  • the orien- s tation of the second camera 420 may also be detected and used in the determination of if the trigger is detected or not. For instance, it can be used to determine that the object 440 is visible in the second camera 420, which may constitute a condition for the trigger to be detected in case the trigger is associated with a switching to the second camera 420 upon the detected trigger in question.
  • the second production step S6 may w comprise determining one second camera 420 (among several possible second cameras) currently showing the object 440 and selecting that second camera 420 as the one providing the second video stream in question in the second production step S6.
  • the first camera 410 captures both the first participant 430 and the object 440.
  • this detected image pattern constitutes the detected trigger.
  • the second production step may comprise the video stream captured0 by the first camera 410 being panned and/or zoomed and/or cropped in the produced output video stream, so as to focus the attention of a viewer of the produced output video stream in question to the object 440.
  • the detected attention of the first participant 430 causes the automatic production (in said second production step) to highlight or shift focus in some manner, based on the existing video streams or by instructing units producing the primary video streams in question, so as to provide increased visual focus to the object 440 in question.
  • the predetermined image and/or audio pattern comprises the first participant 430 depicted in said second digital video stream gazing towards the second camera 420 continuously capturing the second digital video stream.
  • the predetermined image and/or audio pattern comprises a relative w change of movement in said first digital video stream and/or said second digital video stream.
  • the amount of general movement shown in the first video stream may be parameterised and a zoom of the first video stream and/or the second video stream may then be increased as a function of decreasing general movement or vice versa; or the second production step may switch to the second image stream in case of a detected increased is general movement, the second video stream being captured by the second camera 420 showing a more wide-angle or more distant view of a scene depicted by the first camera 410.
  • a recorded sound of the participant's 430 speech may be used to determine such a zoom-in/zoom-out/camera switch performance. For instance, if it is recorded that the participant 430 talks louder, there may be a zoom-out of the first video0 stream in the output shared video stream and vice versa.
  • the predetermined image and/or audio pattern comprises a predetermined sound pattern in turn being characterised by its frequency and/or amplitude and/or amplitude time derivative and/or absolute amplitude change and/or a sound loca-5 tion as determined for instance by a relative microphone volume of a particular sound-capturing microphone.
  • a microphone may be a part of, for instance, the first camera 410, or may be a standalone microphone. Such microphone is then arranged to record sound arising at or in direct connection to a scene being shown in the first video stream.
  • the predetermined sound pattern may comprise a predetermined phrase containing at least one verbally uttered word.
  • the sound may be recorded and be provided as a part of the video streams fed to the first production step, and the predetermined pattern may be detected by the central server 130 and then, when the corresponding trigger is detected, the production may switch to the second production step.
  • the audio analysis may use a suitable digital audio processing algorithm, such as a rule-based decision engine using 5 various sound information (such as pitch, amplitude and pattern matching) to determine if the predetermined sound patter is detected or not; a trained neural network; and so forth.
  • the present method may also comprise a delaying step (see Figure 9), wherein a latency is deliberately introduced with respect to at least the first and w the second digital video stream, the latency being present in the shared digital video stream.
  • the trigger detection step may be performed based on the first and/or second digital video streams before introducing said latency.
  • the latency may be at the most 30 seconds, such as at the most 5 seconds, such as at the is most 1 seconds, such as at the most 0.5 seconds.
  • the automatic production may for instance plan an automatic switch from a first camera 410 to a second camera 420 based on a detected trigger (such as the participant 430 looking into the second camera 420), the planning taking0 place a certain time (such as 0.5 s) before the event is to take part in the produced shared video stream.
  • a switch can be performed at precisely the time of the trigger actually taking place, or at a time fitting optimally with other parameterised production parameters, such as being timed with a speech rhythm of the participant 430 in question.
  • such planning5 may take place with different time horizons with respect to produced output video streams produced with respect to (for consumption by) respective participant clients 121 of different ones of such groups 121', 121”, 121'”.
  • the predetermined image and/or audio pattern may constitute (and be determined as) an0 "event” and/or a "pattern” of the type described above in connection to the event detection function 132 and the pattern detection function 132.
  • the following are a number of examples of how the present invention can be embodied in practise:
  • different cameras can present different camera angles of a presenter.
  • the automatic production function can then be configured to automatically choose between the different camera angles depending on which one of the cameras the presenter is looking at, and/or triggered by gestures or audio cues.
  • the automatic production function can be configured to automatically switch between different cameras depending on what camera a current speaker is facing.
  • the automatic production function can be configured to switch to a camera pointed at an audience for adding input or questions from attendees. This could be triggered by monitoring an audio feed associated with that camera and when it reaches a certain level that camera goes live or by a voice command, for example "question from the audience".
  • the automatic production function can be configured to automatically switch to a camera pointed at a product when it senses motion in that source, or based on another trigger.
  • a video stream captured by a robotic camera capturing the surgery may be replaced by a normal information presentation upon detecting predetermined gestures, audio cues, or upon recognizing that the surgeon is not using a surgeon console or looking up from it.
  • a camera could be configured to be pointed at a regular white- board or chalkboard, and the automatic production function can be configured to switch to that camera when the teacher is making gestures toward the board or by voice commands.
  • the automatic production function may be configured to switch between cameras pointed at a singer, bandmembers or an orchestra. This can be done by either gestures or triggered by which camera the talent is looking at.
  • the automatic production function may be configured to cut between different camera angles depending on who is talking, either by face-tracking, audio cues, gestures or by following a predetermined schedule (rundown).
  • the automatic production may also switch from one format (production rule) to another one based on predetermined schedules (rundowns), and of course also manual override, as the case may be.
  • the present invention also relates to a computer software function for providing the shared digital video stream according to what has been described above. Such a computer software function may then be arranged to, when executing, perform at least some of the abovedescribed collecting, delay, first production, trigger detect, second production and publishing steps described above.
  • the computer software function may be arranged to execute on physical or virtual hardware of the central server 130, as described above.
  • the present invention also relates to the system 100 as such, being a system for providing the shared digital video stream, and in turn comprising the central server 130.
  • the central server 130 may be arranged to perform at least some of said collecting, delay, first production, trigger detect, second production and publishing steps described above. For instance, these steps may be performed by the central server 130 executing said computer software function to perform said steps as described above.
  • the collection may be performed by the collecting function 131.
  • the detection of the predetermined image and/or audio pattern and triggers described above may be performed by the event- and pattern detection function 132, 134, or by the production function 135, of the central server 130.
  • Any deliberate delay may be performed in the way generally described above, such as by the collecting function 131.
  • the publishing may be performed by the publishing function 136.
  • the central server 130 may be arranged to control group 121', 121”, 121'” allocation to individual participant clients 121. For instance, changing group allocation for a particular such participant client dynamically during the course of the live video communication service session may be a part of the automatic production of said video communication service by the central server 130. Such reallocation may be triggered based on a predetermined time-table or dynamically, for instance as a function of parameter data that may change dynamically over time, for instance upon the request (provided via the client 121 in question) of individual participant client users 122.
  • the central server 130 may be arranged to dynamically change the group structure during the course of the video communication service, such as only using a certain group for a predetermined time slot (such as during a planned panel debate).
  • Changing group allocation in a predetermined manner may be the automatic result of the detection of a particular trigger, in a way corresponding to what has been described above in connection to Figures 9-llb.
  • the present invention may further comprise an interaction step, in which at least one participant client of a first group, the first group being associated with a first latency, interacts in a two-way (bi-directional) manner with at least one participant client of the same first group or of a second group, the second group then being associated with a second latency, the second latency being different from the first latency.
  • these participating clients may all be participants to one and the same communication service of the above-described type.

Landscapes

  • Engineering & Computer Science (AREA)
  • Signal Processing (AREA)
  • Multimedia (AREA)
  • General Engineering & Computer Science (AREA)
  • Databases & Information Systems (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Social Psychology (AREA)
  • General Health & Medical Sciences (AREA)
  • Health & Medical Sciences (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • Business, Economics & Management (AREA)
  • Marketing (AREA)
  • Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
  • User Interface Of Digital Computer (AREA)
  • Studio Devices (AREA)
  • Television Systems (AREA)
PCT/SE2023/050090 2022-02-04 2023-02-03 System and method for producing a video stream Ceased WO2023149835A1 (en)

Priority Applications (9)

Application Number Priority Date Filing Date Title
JP2024545896A JP2025506378A (ja) 2022-02-04 2023-02-03 ビデオストリームを生成するシステムおよび方法
CN202380020400.6A CN118743225A (zh) 2022-02-04 2023-02-03 用于产生视频流的系统和方法
AU2023216645A AU2023216645A1 (en) 2022-02-04 2023-02-03 System and method for producing a video stream
CA3249860A CA3249860A1 (en) 2022-02-04 2023-02-03 SYSTEM AND METHOD FOR PRODUCING A VIDEO STREAM
KR1020257041491A KR20260006042A (ko) 2022-02-04 2023-02-03 비디오 스트림의 생성을 위한 시스템 및 방법
EP23750040.0A EP4473736A4 (en) 2022-02-04 2023-02-03 System and method for generating a video stream
KR1020247027325A KR20240144941A (ko) 2022-02-04 2023-02-03 비디오 스트림의 생성을 위한 시스템 및 방법
US18/434,868 US12262069B2 (en) 2022-02-04 2024-02-07 System and method for producing a video stream
JP2026016587A JP2026082966A (ja) 2022-02-04 2026-02-04 ビデオストリームを生成するシステムおよび方法

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SE2250113-4 2022-02-04
SE2250113A SE545897C2 (en) 2022-02-04 2022-02-04 System and method for producing a shared video stream

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/434,868 Continuation US12262069B2 (en) 2022-02-04 2024-02-07 System and method for producing a video stream

Publications (1)

Publication Number Publication Date
WO2023149835A1 true WO2023149835A1 (en) 2023-08-10

Family

ID=87552725

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/SE2023/050090 Ceased WO2023149835A1 (en) 2022-02-04 2023-02-03 System and method for producing a video stream
PCT/SE2023/050091 Ceased WO2023149836A1 (en) 2022-02-04 2023-02-03 System and method for producing a video stream

Family Applications After (1)

Application Number Title Priority Date Filing Date
PCT/SE2023/050091 Ceased WO2023149836A1 (en) 2022-02-04 2023-02-03 System and method for producing a video stream

Country Status (9)

Country Link
US (3) US12262069B2 (cg-RX-API-DMAC7.html)
EP (2) EP4473737A4 (cg-RX-API-DMAC7.html)
JP (3) JP2025505569A (cg-RX-API-DMAC7.html)
KR (3) KR20240145474A (cg-RX-API-DMAC7.html)
CN (2) CN118743225A (cg-RX-API-DMAC7.html)
AU (2) AU2023215134A1 (cg-RX-API-DMAC7.html)
CA (2) CA3249860A1 (cg-RX-API-DMAC7.html)
SE (1) SE545897C2 (cg-RX-API-DMAC7.html)
WO (2) WO2023149835A1 (cg-RX-API-DMAC7.html)

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
SE545897C2 (en) 2022-02-04 2024-03-05 Livearena Tech Ab System and method for producing a shared video stream
US12363418B2 (en) * 2022-03-22 2025-07-15 Lenovo (Singapore) Pte. Ltd. Video activation based upon detection of user action
LU501985B1 (en) * 2022-05-02 2023-11-06 Barco Nv 3D virtual director
EP4383729B1 (en) * 2022-12-07 2025-11-05 Axis AB Video failover recording
SE548244C2 (en) 2024-01-31 2026-04-30 Livearena Tech Ab Method and device for producing a video stream with user authentication
EP4761232A1 (en) * 2024-12-12 2026-06-17 GN Hearing A/S System and device for creating spatial models of multi-device environments
US20260172642A1 (en) 2024-12-17 2026-06-18 Livearena Technologies Inc. System, computer program product and method for transferring a video stream

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060251382A1 (en) * 2005-05-09 2006-11-09 Microsoft Corporation System and method for automatic video editing using object recognition
US20080297587A1 (en) * 2007-05-31 2008-12-04 Kurtz Andrew F Multi-camera residential communication system
US20100238262A1 (en) * 2009-03-23 2010-09-23 Kurtz Andrew F Automated videography systems
US20150281586A1 (en) * 2012-12-18 2015-10-01 Nokia Corporation Method and apparatus for forming a video sequence
US20160148649A1 (en) * 2014-11-20 2016-05-26 Axis Ab Method and apparatus for visualizing information of a digital video stream
US9438937B1 (en) * 2011-05-31 2016-09-06 Brian K. Buchheit Video server that provides a user tailored video stream consistent with user input using content of a primary stream and an enhanced stream
US20190215464A1 (en) * 2018-01-11 2019-07-11 Blue Jeans Network, Inc. Systems and methods for decomposing a video stream into face streams

Family Cites Families (53)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2003239385A1 (en) 2002-05-10 2003-11-11 Richard R. Reisman Method and apparatus for browsing using multiple coordinated device
US20090102919A1 (en) * 2007-12-31 2009-04-23 Zamierowski David S Audio-video system and method for telecommunications
US8233530B2 (en) 2008-10-28 2012-07-31 International Business Machines Corporation Video scene matching on return from virtual rendering in a consumer digital video recording device
US8477174B2 (en) 2009-02-27 2013-07-02 Avaya Inc. Automatic video switching for multimedia conferencing
US8818175B2 (en) * 2010-03-08 2014-08-26 Vumanity Media, Inc. Generation of composited video programming
US8379077B2 (en) 2010-11-24 2013-02-19 Cisco Technology, Inc. Automatic layout and speaker selection in a continuous presence video conference
US20120236160A1 (en) 2011-03-15 2012-09-20 Tomas Rezek Remote studio management and control
US8760490B2 (en) 2011-11-30 2014-06-24 Cisco Technology, Inc. Techniques for a rate-adaptive video conference bridge
US9571879B2 (en) 2012-01-10 2017-02-14 Microsoft Technology Licensing, Llc Consumption of content with reactions of an individual
US9001178B1 (en) 2012-01-27 2015-04-07 Google Inc. Multimedia conference broadcast system
US8908005B1 (en) 2012-01-27 2014-12-09 Google Inc. Multiway video broadcast system
US8830295B2 (en) 2012-05-23 2014-09-09 Google Inc. Multimedia conference endpoint transfer system
US8830296B1 (en) 2012-06-26 2014-09-09 Google Inc. Endpoint device-specific stream control for multimedia conferencing
US9369670B2 (en) 2012-12-19 2016-06-14 Rabbit, Inc. Audio video streaming system and method
EP2963929B1 (en) 2014-07-03 2016-12-21 Axis AB A method for configuration of video stream output from a digital video camera
US20160066064A1 (en) 2014-08-28 2016-03-03 Dozo LLP Clip Start Point
US10536758B2 (en) 2014-10-09 2020-01-14 Thuuz, Inc. Customized generation of highlight show with narrative component
US10721499B2 (en) 2015-03-27 2020-07-21 Twitter, Inc. Live video streaming services
US9641585B2 (en) 2015-06-08 2017-05-02 Cisco Technology, Inc. Automated video editing based on activity in video conference
CA2992495A1 (en) 2015-07-17 2017-01-26 Tribune Broadcasting Company, Llc Video-production system with social-media features
US9813673B2 (en) 2016-01-20 2017-11-07 Gerard Dirk Smits Holographic video capture and telepresence system
US10887628B1 (en) * 2016-04-27 2021-01-05 United Services Automobile Services (USAA) Systems and methods for adaptive livestreaming
US10136102B2 (en) 2016-04-29 2018-11-20 Microsoft Technology Licensing, Llc Online conference broadcast using broadcast component
US9998769B1 (en) 2016-06-15 2018-06-12 Gopro, Inc. Systems and methods for transcoding media files
US20180077437A1 (en) 2016-09-09 2018-03-15 Barrie Hansen Parallel Video Streaming
US20180098100A1 (en) 2016-10-03 2018-04-05 Intercept Technologies, Inc. Streaming video system
US20180213288A1 (en) 2017-01-26 2018-07-26 Gopro, Inc. Systems and methods for creating video compositions
US10135601B1 (en) 2017-05-16 2018-11-20 Disney Enterprises, Inc. Providing common point of control and configuration in IP-based timing systems
CN109413437A (zh) 2017-08-15 2019-03-01 深圳富泰宏精密工业有限公司 电子设备及传送视频流的方法
US10812700B2 (en) 2017-11-02 2020-10-20 Arris Enterprises Llc Approach to live multi-camera streaming of events with hand-held cameras
US11736406B2 (en) 2017-11-30 2023-08-22 Comcast Cable Communications, Llc Assured related packet transmission, delivery and processing
US10327046B1 (en) 2017-12-19 2019-06-18 Oath Inc. Method and system for generating a time-lapse video
KR102449670B1 (ko) 2018-01-16 2022-09-30 삼성전자주식회사 복수의 카메라를 이용하여 영상 데이터를 생성하는 방법 및 서버
US10872240B2 (en) 2018-09-28 2020-12-22 Opentv, Inc. Systems and methods for generating media content
US10915776B2 (en) * 2018-10-05 2021-02-09 Facebook, Inc. Modifying capture of video data by an image capture device based on identifying an object of interest within capturted video data to the image capture device
CA3122344A1 (en) 2018-12-19 2020-06-25 RxPrism Health Systems Pvt. Ltd A system and a method for creating and sharing interactive content rapidly anywhere and anytime
US11784839B2 (en) 2019-12-31 2023-10-10 Dish Network Technologies India Private Limited Dynamic low latency mode for a digital video production system
US10904485B1 (en) * 2020-01-27 2021-01-26 Plantronics, Inc. Context based target framing in a teleconferencing environment
JP7532811B2 (ja) 2020-03-02 2024-08-14 富士フイルムビジネスイノベーション株式会社 情報処理システム、情報処理装置、および、プログラム
EP3908006A3 (en) * 2020-05-01 2022-01-26 TVU Networks Corporation Systems and methods for real time control of a remote video production with multiple streams
US11924580B2 (en) * 2020-05-07 2024-03-05 Intel Corporation Generating real-time director's cuts of live-streamed events using roles
US11206443B2 (en) 2020-05-08 2021-12-21 Facebook, Inc. Synchronizing streams of co-watching digital video content while providing live digital video chat streams across multiple client devices
US12212807B2 (en) * 2020-06-16 2025-01-28 Nippon Telegraph And Telephone Corporation Transmission apparatus, communication method and program
US11082467B1 (en) * 2020-09-03 2021-08-03 Facebook, Inc. Live group video streaming
US11277620B1 (en) 2020-10-30 2022-03-15 Hulu, LLC Adaptive transcoding of profile ladder for videos
US11621861B2 (en) 2020-11-06 2023-04-04 Microsoft Technology Licensing, Llc Dynamic grouping of live video streams and content views
US20220174357A1 (en) * 2020-11-30 2022-06-02 At&T Intellectual Property I, L.P. Simulating audience feedback in remote broadcast events
US11374988B1 (en) 2021-01-29 2022-06-28 Microsoft Technology Licensing, Llc Controlled user interface transitions for private breakout communication sessions
CN115047966B (zh) * 2021-02-26 2025-08-01 华为技术有限公司 交互方法、电子设备与交互系统
CA3162906A1 (en) 2021-06-15 2022-12-15 Evertz Microsystems Ltd. Method and system for content aware monitoring of media channel output by a media system
US11622147B2 (en) 2021-07-22 2023-04-04 Popio Mobile Video Cloud, Llc Blurring digital video streams upon initiating digital video communications
US11729354B2 (en) 2021-10-29 2023-08-15 Zoom Video Communications, Inc. Remotely adjusting audio capture during video conferences
SE545897C2 (en) 2022-02-04 2024-03-05 Livearena Tech Ab System and method for producing a shared video stream

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20060251382A1 (en) * 2005-05-09 2006-11-09 Microsoft Corporation System and method for automatic video editing using object recognition
US20080297587A1 (en) * 2007-05-31 2008-12-04 Kurtz Andrew F Multi-camera residential communication system
US20100238262A1 (en) * 2009-03-23 2010-09-23 Kurtz Andrew F Automated videography systems
US9438937B1 (en) * 2011-05-31 2016-09-06 Brian K. Buchheit Video server that provides a user tailored video stream consistent with user input using content of a primary stream and an enhanced stream
US20150281586A1 (en) * 2012-12-18 2015-10-01 Nokia Corporation Method and apparatus for forming a video sequence
US20160148649A1 (en) * 2014-11-20 2016-05-26 Axis Ab Method and apparatus for visualizing information of a digital video stream
US20190215464A1 (en) * 2018-01-11 2019-07-11 Blue Jeans Network, Inc. Systems and methods for decomposing a video stream into face streams

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4473736A4 *

Also Published As

Publication number Publication date
US20250193463A1 (en) 2025-06-12
EP4473737A4 (en) 2025-03-26
US12184908B2 (en) 2024-12-31
EP4473736A1 (en) 2024-12-11
KR20240144941A (ko) 2024-10-04
KR20260006042A (ko) 2026-01-12
AU2023215134A1 (en) 2024-08-08
CN118786674A (zh) 2024-10-15
JP2025506378A (ja) 2025-03-11
CA3249860A1 (en) 2023-08-10
JP2025505569A (ja) 2025-02-28
SE545897C2 (en) 2024-03-05
CA3249861A1 (en) 2023-08-10
EP4473737A1 (en) 2024-12-11
US12262069B2 (en) 2025-03-25
AU2023216645A1 (en) 2024-08-08
SE2250113A1 (en) 2023-08-05
KR20240145474A (ko) 2024-10-07
WO2023149836A1 (en) 2023-08-10
EP4473736A4 (en) 2025-04-16
US20240267488A1 (en) 2024-08-08
US20240179357A1 (en) 2024-05-30
JP2026082966A (ja) 2026-05-19
CN118743225A (zh) 2024-10-01

Similar Documents

Publication Publication Date Title
US12184908B2 (en) System and method for producing a video stream
US20250259654A1 (en) System and method for producing a video stream
US12316997B2 (en) System and method for producing a video stream
AU2022368286A1 (en) System and method for producing video stream
Padovani Pandemics, Delays, and Pure Data: on'afterlives'(2020), for Flute and Live Electronics and Visuals

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23750040

Country of ref document: EP

Kind code of ref document: A1

DPE1 Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101)
WWE Wipo information: entry into national phase

Ref document number: AU23216645

Country of ref document: AU

WWE Wipo information: entry into national phase

Ref document number: 2024545896

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 202380020400.6

Country of ref document: CN

ENP Entry into the national phase

Ref document number: 2023216645

Country of ref document: AU

Date of ref document: 20230203

Kind code of ref document: A

REG Reference to national code

Ref country code: BR

Ref legal event code: B01A

Ref document number: 112024015713

Country of ref document: BR

ENP Entry into the national phase

Ref document number: 20247027325

Country of ref document: KR

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 202417064950

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 2023750040

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2023750040

Country of ref document: EP

Effective date: 20240904

ENP Entry into the national phase

Ref document number: 112024015713

Country of ref document: BR

Kind code of ref document: A2

Effective date: 20240731