JP2018521593A - Composition and scaling of angle-separated subscenes - Google Patents

Abstract

A densely synthesized single camera signal can be formed from a panoramic video signal captured from a wide camera and having an aspect ratio substantially greater than 2.4: 1. Two or more sub-scene video signals may be sub-sampled in their respective orientations and combined side by side to form a stage scene video signal having an aspect ratio of substantially 2: 1 or less. More than 80% of the area of the stage scene video signal can be subsampled from the panoramic video signal.

Description

CROSS REFERENCE TO RELATED APPLICATIONS This application claims the benefit based on US Provisional Patent Application Serial No. 62 / 141,822 filed on April 1, 2015 in accordance with US Pat. The entire disclosure of the patent application is incorporated herein by reference.

FIELD Aspects relate to apparatus and methods for image capture and enhancement.

Background Multi-party teleconferencing, video chat, and video conferencing often take place with multiple participants in the same conference room connected to at least one remote party.

  In the person-to-person mode of the video conferencing software, only one local camera, often with a limited horizontal field of view (eg, 70 degrees), is available. Regardless of whether this single camera is positioned in front of one participant or is directed to all participants and positioned at the head of the table, it is far from or relative to that camera. It is difficult for a remote party to understand the voice, body language, and non-verbal cues given by those participants in a meeting room that is sharp and sharp (for example, looking at a contour rather than a human face) ).

  In the multiplayer mode of video conferencing software, several different problems are added because the cameras of two or more mobile devices (laptops, tablets, or cell phones) in the same conference room are available. The more conference room participants logging into the conference, the greater the voice feedback and crosstalk. The camera perspective may be as far away from the participant as the single camera or distorted. Local participants tend to interact with other participants via their mobile devices (thus inheriting the same body language and non-verbal cues as remote parties), even though they are in the same room There can be.

  Angle-separated within a wide scene (eg, wide scenes of two or more conference participants) to make setup very easy for participants in the room or to make the experience automatically and seamless from the remote participant's perspective There is no known commercial or experimental technique for compositing, tracking, and / or displaying a sub-scene and / or subject sub-scene.

Overview In one aspect of this embodiment, the process of outputting a densely synthesized single camera signal is substantially 2.4 captured from a wide camera having a horizontal field of view substantially greater than 90 degrees. A panoramic video signal having an aspect ratio of 1 or more can be recorded. At least two sub-scene video signals can be sub-sampled from the wide camera in their target orientations. Two or more sub-scene video signals may be combined side by side to form a stage scene video signal having an aspect ratio substantially equal to or less than 2: 1. Optionally, an area greater than 80% of the area of the stage scene video signal is subsampled from the panoramic video signal. The stage scene video signal may be formatted as a single camera video signal. Optionally, the panoramic video signal is captured from a wide camera having an aspect ratio of substantially greater than 8: 1 and a horizontal field of view of substantially 360 degrees.

  In a related aspect of this embodiment, the conference camera is configured to output a densely synthesized single camera signal. The conference camera imager or wide camera may be configured to capture and / or record a panoramic video signal having an aspect ratio of substantially 2.4: 1 or greater, the wide camera being substantially 90 degrees or greater. Horizontal angle of view. A processor operably connected to the image sensor or the wide camera may be configured to subsample two or more sub-scene video signals from the wide camera in their target orientations. The processor combines two or more sub-scene video signals into a memory (eg, a buffer and / or video memory) as a side-by-side video signal to produce a stage scene video signal having an aspect ratio substantially equal to or less than 2: 1. Can be configured to form. The processor is configured to synthesize the sub-scene video signal into memory (eg, a buffer and / or video memory) such that more than 80% of the area of the stage scene video signal is sub-sampled from the panoramic video signal. obtain. The processor may be further configured to format the stage scene video signal as a single camera video signal transported over, for example, USB.

  In any one of the above aspects, the processor sub-samples additional sub-scene video signals in their respective orientations from the panoramic video signal and converts the two or more sub-scene video signals into one or more Combining with the additional sub-scene video signal to form a stage scene video signal having an aspect ratio of substantially less than 2: 1 comprising a plurality of side-by-side sub-scene video signals obtain. Optionally, combining two or more subscene video signals with one or more additional subscene video signals to form a stage scene video signal replaces at least one of the two or more subscene video signals Transitioning one or more additional sub-scene video signals to the stage scene video signal by forming a stage scene video signal having an aspect ratio of substantially 2: 1 or less.

  Further optionally, each sub-scene video signal can be assigned a minimum width, and once each transition to the stage scene video signal is complete, each sub-scene video signal is substantially combined side by side with the minimum width and combined to the stage. A scene video signal may be formed. Alternatively or additionally, the composite width of each sub-scene video signal in transition may increase throughout the transition until the composite width is substantially greater than or equal to its corresponding minimum width. Additionally or alternatively, the sub-scene video signal may be combined side by side substantially above its minimum width, each of which is the sum of all combined sub-scene video signals substantially equal to the width of the stage scene video signal. Can be combined with their own width equal to.

  In some cases, the width of the sub-scene video signal in the stage scene video signal may be synthesized to change according to the activity criteria detected in one or more target orientations corresponding to the sub-scene video signal. The width of the stage scene video signal is kept constant. In other cases, combining two or more sub-scene video signals with one or more additional sub-scene video signals to form a stage scene video signal is at least one width of the two or more sub-scene video signals Transitioning the one or more additional sub-scene video signals to the stage scene video signal by reducing by an amount corresponding to the width of the one or more additional sub-scene video signals.

  Further optionally, each sub-scene video signal may be assigned its own minimum width, and each sub-scene video signal is combined side by side with substantially its corresponding minimum width to form a stage scene video signal. obtain. When the sum of the respective minimum widths of the two or more subscene video signals exceeds the width of the stage scene video signal, along with the one or more additional subscene video signals, at least one of the two or more subscene video signals is A transition may be made to be removed from the stage scene video signal. Optionally, the sub-scene video signal that transitions to be removed from the stage scene video signal corresponds to its own target orientation for which the activity criteria were most recently met.

  In any one of the above aspects, the order from left to right for a wide camera between their respective orientations of two or more sub-scene video signals and one or more additional sub-scene video signals is two or more The sub-scene video signal may be combined with one or more additional sub-scene video signals and stored when forming a stage scene video signal.

  Further, in any one of the above aspects, the respective target orientation from the panoramic video signal can be selected depending on the selection criteria detected in the respective target orientation relative to the wide camera. After the selection criteria are no longer true, the corresponding sub-scene video signal may transition to be removed from the stage scene video signal. Alternatively or additionally, the selection criteria may include the presence of activity criteria that are met in each subject orientation. In this case, the processor may count the time since the activity criterion is met in its target orientation. Each sub-scene signal may transition to be removed from the stage scene video signal for a predetermined period of time after the activity criteria are met in each target orientation.

  In a further variation of the above aspect, the processor subsamples a reduced panoramic video signal having an aspect ratio of substantially 8: 1 or greater from the panoramic video signal and reduces two or more subscene video signals. Combining with the panoramic video signal may be performed to form a stage scene video signal having an aspect ratio of substantially 2: 1 or less, including a plurality of side-by-side sub-scene video signals and panoramic video signals. Optionally, two or more sub-scene video signals are combined with the reduced panoramic video signal to include a plurality of side-by-side sub-scene video signals and a panoramic video signal higher than the plurality of side-by-side sub-scene video signals. A stage scene video signal having an aspect ratio of substantially 2: 1 or less, wherein the panoramic video signal is no more than 1/5 of the area of the stage scene video signal and substantially reduces the width of the stage scene video signal. It extends across.

  In a further variation of the above aspect, the processor or associated processor subtexts the text video signal from the text document and replaces at least one of the two or more subscene video signals with the text video signal. The video signal may be transitioned to a stage scene video signal.

  Optionally, the processor may set at least one of the two or more sub-scene video signals as a protected sub-scene video signal that is protected from transition based on retention criteria. In this case, the processor may replace one or more additional sub-scene video signals by replacing at least one of the two or more sub-scene video signals and / or by transitioning sub-scene video signals other than the protected sub-scene. The scene video signal may be transitioned to a stage scene video signal.

  In some cases, the processor may alternatively or additionally set a sub-scene enhancement operation based on an enhancement criterion, and at least one of the two or more sub-scene video signals may be based on a corresponding enhancement criterion. Emphasized according to the sub-scene enhancement operation. Optionally, the processor may set a sub-scene participant notification action based on a criterion detected from the sensor, and a local reminder indicator (eg, light, blinking, or sound) is based on the corresponding detected criterion Is started according to the notification operation.

  In one aspect of this embodiment, a process for tracking a sub-scene in a target orientation within a wide video signal is a range of angles using an acoustic sensor array and a wide camera that observes a field of view substantially greater than 90 degrees. Monitoring. A first target orientation may be identified along with localization of at least one of acoustic recognition and visual recognition detected within the angular range. A first sub-scene video signal may be sub-sampled from the wide camera along the first target orientation. The width of the first sub-scene video signal may be set according to signal characteristics of at least one of acoustic recognition and visual recognition.

  In a related aspect of this embodiment, the conference camera is configured to output a video signal that includes a subsampled and scaled subscene from a wide angle scene and to track the subscene and / or subject orientation within the wide video signal. Can be done. The conference camera and / or its processor may be configured to monitor a range of angles using an acoustic sensor array and a wide camera that observes a field of view substantially greater than 90 degrees. The processor may be configured to identify a first target orientation along localization of at least one of acoustic recognition and visual recognition detected within the angular range. The processor may be further configured to subsample a first sub-scene video signal from a wide camera along a first target orientation into a memory (buffer or video). The processor may be further configured to set the width of the first sub-scene video signal according to signal characteristics of at least one of acoustic recognition and visual recognition.

  In any of the above aspects, the signal characteristic may represent a confidence level of one or both of acoustic recognition or visual recognition. Optionally, the signal characteristic may represent a feature width recognized within one or both of acoustic recognition or visual recognition. Further optionally, the signal characteristic may correspond to an approximate width of a human face recognized along the first target orientation.

  Alternatively or additionally, if the width is not set according to the visual recognition signal characteristics, a predetermined width may be set along with the localization of acoustic recognition detected within the angular range. Further optionally, the first target orientation may be determined by visual recognition, and the width of the first sub-scene video signal is then set according to the visual recognition signal characteristics. Further optionally, the first target orientation can be directed and identified towards acoustic recognition detected within the angular range. In this case, the processor may identify visual recognition proximate to acoustic recognition, and the width of the first sub-scene video signal may then be set according to the signal characteristics of visual recognition proximate to acoustic recognition.

  In another aspect of this embodiment, the processor may be configured to perform a process of tracking a sub-scene in a target orientation within a wide video signal, the process substantially over a wide camera field of view of 90 degrees or more. Scanning a sub-sampling window through a corresponding animated video signal. The processor may be configured to identify candidate orientations within the sub-sampling window, each target orientation corresponding to a localized visual recognition detected within the sub-sampling window. The processor may then record the candidate orientation in a spatial map and monitor the angular range corresponding to the wide camera field of view using an acoustic sensor array for acoustic recognition.

  Optionally, when acoustic recognition is detected proximate to one candidate orientation recorded in the spatial map, the processor further snaps the first target orientation to substantially correspond to one candidate orientation. Thus, the first sub-scene video signal may be sub-sampled from the wide camera along the first target orientation. Optionally, the processor may be further configured to set the width of the first sub-scene video signal according to the signal characteristics of sound recognition. Further optionally, the signal characteristic may represent a confidence level of acoustic recognition or may represent a width of a feature recognized within one or both of acoustic recognition or visual recognition. The signal characteristic may alternatively or additionally correspond to the approximate width of the human face recognized along the first object orientation. Optionally, if the width is not set according to the signal characteristics of visual recognition, a predetermined width can be set along with the localization of acoustic recognition detected within the angular range.

  In another aspect of this embodiment, the processor may be configured to track the sub-scene in the target orientation, which records a video video signal substantially corresponding to a wide camera field of view of 90 degrees or more. Including. The processor monitors an angular range corresponding to the wide camera field of view using an acoustic sensor array for acoustic recognition and is directed toward the acoustic recognition detected within the angular range. May be configured to identify. A subsampling window may be positioned in the video video signal according to the first target orientation and visual recognition may be detected in the subsampling window. Optionally, the processor subsamples a first sub-scene video signal captured from a wide camera substantially centered on visual recognition and determines the width of the first sub-scene video signal according to the signal characteristics of visual recognition. Can be configured to set.

  In a further aspect of this embodiment, the processor may be configured to track a sub-scene in a target orientation within a wide video signal, which is a wide camera that observes an acoustic sensor array and a field of view substantially greater than 90 degrees. And monitoring an angular range. Multiple object orientations can be identified, each oriented towards localization within an angular range. The processor may be configured to maintain a spatial map of recorded characteristics corresponding to the subject orientation and subsample the sub-scene video signal from the wide camera substantially along one or more subject orientations. The width of the sub-scene video signal can be set according to recorded characteristics corresponding to at least one target orientation.

  In a further aspect of this embodiment, the processor may be configured to perform a process of tracking sub-scenes in a target orientation within the wide video signal, the process substantially including a field of view greater than 90 degrees with the acoustic sensor array. Monitoring a range of angles using a wide camera that observes and identifying a plurality of target orientations each directed towards localization within the range of angles. A sub-scene video signal can be sampled from a wide camera substantially along at least one object orientation, and by expanding the sub-scene video signal until a threshold based on at least one recognition criterion is met, The width can be set. Optionally, a change vector for each target orientation can be predicted based on one of the changes in velocity and direction of the recorded characteristics corresponding to localization, and the location of the target orientation can be updated based on the prediction. Optionally, a search region for localization can be predicted based on the most recent location of the recorded characteristic corresponding to localization, and the localization location can be updated based on the prediction.

2 is a schematic block diagram of an embodiment of a device suitable for compositing, tracking, and / or displaying angle separated sub-scenes and / or subject sub-scenes in a wide scene collected by device 100. FIG. 2 is a schematic block diagram of an embodiment of a device suitable for compositing, tracking, and / or displaying angle separated sub-scenes and / or subject sub-scenes in a wide scene collected by device 100. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. 1B is a schematic diagram of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device 100 of FIGS. 1A and 1B. FIG. It is the figure which looked down at the use example of a conference camera which shows three participants from the top. It is the figure which looked down at the conference camera panorama image signal which shows three participants from the top. FIG. 6 is a top down view of a conference camera use case showing a conference table, showing a depiction of three participants and including a face width setting or sub-scene identification depiction. FIG. 6 is a top down view of a conference camera panoramic image signal showing three participants and including a depiction of face width setting or sub-scene identification. FIG. 6 is a top down view of a conference camera use case showing a conference table, showing three participants and including a depiction of shoulder width settings or sub-scene identification. FIG. 5 is a top down view of a conference camera panoramic image signal showing three participants and including a depiction of shoulder width setting or sub-scene identification. FIG. 6 is a top down view of a conference camera use case showing a conference table, showing a depiction of a wider sub-scene showing three participants and a whiteboard. FIG. 3 is a top down view of a conference camera panoramic image signal showing three participants and a whiteboard, including a depiction of wider sub-scene identification. FIG. 6 is a top down view of a conference camera use case showing a 10-person conference table showing five participants and including a depiction of visual minimum width and orientation and acoustic minimum width and orientation identification. FIG. 5 is a top down view of a conference camera panoramic image signal showing five participants and including a depiction of visual minimum width and orientation and acoustic minimum width and orientation identification. FIG. 6 is a schematic diagram of a conference camera video signal, a minimum width, and extraction of a sub-scene video signal and a panoramic video signal to be combined with a stage scene video signal. It is the schematic of the sub scene video signal and panoramic video signal which should be synthesize | combined with a stage scene video signal. FIG. 6 illustrates a possible composite output or stage scene video signal. FIG. 6 illustrates a possible composite output or stage scene video signal. FIG. 6 illustrates a possible composite output or stage scene video signal. FIG. 5 is a schematic diagram of a conference camera video signal, a minimum width, and extraction of an alternative sub-scene video signal and an alternative panoramic video signal to be combined with the stage scene video signal. FIG. 4 is a schematic diagram of an alternative sub-scene video signal and an alternative panoramic video signal to be combined with a stage scene video signal. FIG. 6 illustrates a possible alternative composite output or stage scene video signal. FIG. 6 illustrates a possible alternative composite output or stage scene video signal. FIG. 6 illustrates a possible alternative composite output or stage scene video signal. FIG. 6 is a schematic view of a panoramic video signal adjusted so that the conference table images are arranged in a more natural and comfortable view. FIG. 6 is a schematic diagram of possible composite output or stage scene video signals. FIG. 6 is a schematic diagram of possible composite output or stage scene video signals. FIG. 6 is a schematic diagram of an alternative method in which video conferencing software may display a composite output or stage scene video signal. FIG. 6 is a schematic diagram of an alternative method in which video conferencing software may display a composite output or stage scene video signal. FIG. 5 is a flowchart of steps including steps for synthesizing a stage scene (video signal) video signal. FIG. 6 is a detailed flowchart diagram including steps for synthesizing and generating a sub-scene (sub-scene video signal) based on a target orientation. FIG. 4 is a detailed flowchart diagram including steps for synthesizing a sub-scene with a stage scene video signal. FIG. 6 is a detailed flowchart diagram including steps for outputting a combined stage scene video signal as a single camera signal. FIG. 4 is a detailed flowchart diagram including a first mode for performing localization and / or steps for setting a subject orientation and / or sub-scene width. FIG. 6 is a detailed flowchart diagram including a second mode of performing steps for localization and / or setting the orientation and / or sub-scene width. FIG. 6 is a detailed flowchart diagram including a third mode for performing steps for localization and / or setting the orientation and / or sub-scene width. FIG. 6 illustrates the operation of an embodiment including a conference camera attached to a local PC having a video conferencing client that receives a single camera signal substantially corresponding to FIGS. 3A-5B, the PC then connecting to the Internet Two remote PCs etc. also receive a single camera signal within the video conference display. FIG. 6 illustrates the operation of an embodiment including a conference camera attached to a local PC having a video conferencing client that receives a single camera signal substantially corresponding to FIGS. 3A-5B, the PC then connecting to the Internet Two remote PCs etc. also receive a single camera signal within the video conference display. FIG. 6 illustrates the operation of an embodiment including a conference camera attached to a local PC having a video conferencing client that receives a single camera signal substantially corresponding to FIGS. 3A-5B, the PC then connecting to the Internet Two remote PCs etc. also receive a single camera signal within the video conference display. FIG. 22 illustrates a variation of the system of FIGS. 19-21 where video conferencing clients use overlapping video views instead of individual adjacent views. FIG. 22 shows a variation of the system of FIGS. 19-21 substantially corresponding to FIGS. 6A-6B and including a high resolution camera view for a whiteboard. FIG. 22 illustrates a variation of the system of FIGS. 19-21 including a high resolution text document view (eg, text editor, word processing, presentation, or spreadsheet). FIG. 2 is a schematic diagram of an arrangement in which video conferencing clients are instantiated for each sub-scene using a configuration similar to that of FIG. 1B. FIG. 27 is a schematic illustration of some exemplary icons and symbols used throughout FIGS.

Detailed Description Conference Camera FIGS. 1A and 1B synthesize, track, and / or display angle separated sub-scenes and / or subject sub-scenes in a wide scene collected by a device that is a conference camera 100 FIG. 2 is a schematic block diagram of an embodiment of a device suitable for:

  FIG. 1A communicates as a conference camera 100 or conference “webcam”, eg, as a connected laptop, tablet, or USB peripheral device connected to a USB host or hub of the mobile device 40 and “Google Aspect ratio, pixel count, and ratio commonly used by off-the-shelf video chat or video conferencing software such as Hangout, Skype, or Facetime Fig. 2 illustrates a device constructed to provide a single video image. The device 100 may be a “wide camera” 2, 3 or 5 such as, for example, a camera oriented to overlook a meeting of attendees or participants M1, M2. including. The camera 2, 3 or 5 may include one digital imaging device or lens, or two or more digital imaging devices or lenses (eg, stitched to software or others). It should be noted that the field of view of the wide camera 2, 3 or 5 can be 70 degrees or less depending on the location of the device 100 in the conference. However, in one or more embodiments, wide cameras 2, 3, and 5 are useful in the middle of a conference, where the wide camera is substantially 90 degrees or greater than 140 degrees (not necessarily continuous). Not), or have a horizontal field of view of up to 360 degrees.

  In a large conference room (eg a conference room designed to accommodate more than 8 people), record a wide field of view (eg, substantially 90 degrees or more) and stitch together each other with a very wide scene. It may be useful to have multiple wide-angle camera devices that capture a pleasant angle. For example, a wide-angle camera at the far end of a long (10'-20 ') table may provide an unsatisfactory distant view of the speaker SPKR, but multiple cameras (eg, one for every five seats) distributed throughout the table. ) May provide at least one satisfactory or pleasant view. Cameras 2, 3, and 5 capture or record panoramic scenes (eg, H: V horizontal-vertical ratio, eg 2.4: 1 to 10: 1 aspect ratio) and / or connect this signal via USB Can be made available via

  As described with respect to FIGS. 2A-2L, since the height of the wide cameras 2, 3, 5 from the base of the conference camera 100 is preferably greater than 8 inches, the cameras 2, 3, 5 are typical wraps in a conference. It may be higher than the top screen, thereby having an unobstructed and / or approximately eye-level view to conference attendees M1, M2,. The microphone array 4 includes at least two microphones to obtain a target orientation to a nearby sound or utterance by beamforming, relative time of flight, localization, or received signal strength differences as is known in the art. Can do. The microphone array 4 may include a plurality of microphone pairs that are oriented to cover at least substantially the same angular range as the field of view of the wide camera 2.

  Since the microphone array 4 is arbitrarily arranged together with the wide cameras 2, 3, 5 at a height higher than 8 inches, during the speech of the attendees M 1, M 2, Mn, between the array 4 and the attendee A direct “line of sight” still exists and is not obstructed by a typical laptop screen. A CPU and / or GPU (and associated circuitry such as camera circuitry) 6 for processing computations and graphical events is connected to each of the wide cameras 2, 3, 5 and the microphone array 4. ROM and RAM 8 are connected to the CPU and GPU 6 for holding and receiving executable code. A network interface and stack 10 is provided for USB, Ethernet, and / or WiFi connected to the CPU 6. One or more serial buses interconnect these electronic components and are powered by DC, AC, or battery power.

  The camera circuits of cameras 2, 3 and 5 process or render the processed or rendered image or video stream from 1.25: 1 to 2.4: 1 or 2.5: 1 “H: V” horizontal-vertical. A suitable lens as a single camera image signal, video signal or stream in landscape direction with a ratio or aspect ratio (eg including ratios of 4: 3, 16:10, 16: 9) and / or as described above And / or the stitching circuit may be used to output a panoramic image or video stream as a substantially single camera image signal of 2.4: 1 or higher. The conference camera 100 of FIG. 1A is typically a USB peripheral device such as a laptop, tablet, or mobile device 40 (with a display, network interface, computing processor, memory, camera and microphone unit interconnected by at least one bus. Multi-party video conferencing, video conferencing, or video chat software is hosted and connectable for video conferencing with the remote client 50 over the Internet 60.

  FIG. 1B is a variation of FIG. 1A in which both device 100 and video conferencing device 40 of FIG. 1A are integrated. The camera circuit output as a single camera image signal, video signal or video stream is directly available to the CPU, GPU, associated circuitry and memory 5,6, and video conferencing software is instead associated with the CPU, GPU and associated Hosted by a separate circuit and memory 5,6. The device 100 can be connected directly (e.g., via WiFi or Ethernet) for video conferencing with the remote client 50 via the Internet 60 or INET. Display 12 provides a user interface for operating the video conference software and for viewing video conference views and graphics described herein to conference attendees M1, M2,... M3. The device or conference camera 100 of FIG. 1A is instead connected directly to the Internet 60 so that video can be recorded directly to a remote server by the remote client 50 or live from such a server to the video by the remote client 50. It may be possible to access.

  FIGS. 2A-2L are schematic views of an embodiment of a conference camera 14 or camera tower 14 arrangement suitable for collecting wide and / or panoramic scenes for the device or conference camera 100 of FIGS. 1A and 1B. It is. Although “camera tower” 14 and “conference camera” 14 may be used substantially interchangeably herein, the conference camera may not be a camera tower. The height of the wide cameras 2, 3, 5 from the base of the device 100 in FIGS. 2A-2L is preferably greater than 8 inches and less than 15 inches.

  In the camera tower 14 arrangement of FIG. 2A, a plurality of cameras are arranged around the camera level of the camera tower 14 (8 to 15 inches) and are spaced equiangularly. The number of cameras is determined by the field of view of the camera and the angle to be spanned, and when forming a panoramic stitched view, the spanning cumulative angle should have an overlap between the individual cameras. For example, in the case of FIG. 2A, four cameras 2a, 2b, 2c, and 2d (labeled 2a to 2d) each having a field of view of 100 to 110 degrees (shown by broken lines) are arranged at 90 degrees with respect to each other. Thus providing a 360 degree cumulative view around the camera tower 14 or a stitchable or stitched view.

  For example, in the case of FIG. 2B, three cameras 2a, 2b, 2c (labeled 2a-2c) each having a field of view of 130 degrees or more (shown by a broken line) are arranged at 120 degrees with respect to each other. , Providing a 360 degree cumulative view or stitchable view around the tower 14. The vertical field of view of the cameras 2a to 2d is smaller than the horizontal field of view, for example, less than 80 degrees. Images, videos or sub-scenes from each camera 2a-2d can be processed before and after known optical corrections such as stitching, dewarping, or distortion compensation to identify the target orientation or sub-scene, but typically output That will be corrected before.

  In the camera tower 14 arrangement of FIG. 2C, a single fisheye camera or an approximately fisheye camera 3a oriented upwards is arranged on top of the camera level (8 to 15 inches) of the camera tower 14. In this case, the fisheye camera lens is arranged with a continuous horizontal view of 360 degrees and a vertical field of view (shown in broken lines) of about 215 (eg 190-230) degrees. Instead, a single catadioptric “cylindrical image” camera or lens 3b with a cylindrical transmissive shell, top paraboloid mirror, black center post, telecentric lens configuration, for example as shown in FIG. It has a continuous horizontal view and is arranged with a vertical field of view of about 40-80 degrees and is approximately centered on the horizontal line. For each fisheye camera and cylindrical imaging camera, the vertical field of view, positioned 8-15 inches above the conference table, extends below the horizontal line to allow attendees M1, M2,. It is possible to take an image up to below. Images, videos or sub-scenes from each camera 3a or 3b may be processed before and after known optical corrections for fisheye or catadioptric lenses such as dewarping or distortion compensation to identify the target orientation or sub-scene, Will be so corrected before output.

  In the camera tower 14 arrangement of FIG. 2L, a plurality of cameras are arranged around the camera level (8 to 15 inches) of the camera tower 14 and spaced equiangularly. The number of cameras is not intended in this case to form a completely continuous panoramic stitched view, and the spanning cumulative angle has no overlap between the individual cameras. For example, in the case of FIG. 2L, two cameras 2a, 2b, each having a field of view of 130 degrees or more (shown by a broken line), are arranged at 90 degrees from each other, and include separate 260 degrees or more on both sides of the camera tower 14. Provide a view. This arrangement is useful for long conference tables CT. For example, in the case of FIG. 2E, the two cameras 2a-2b are panning and / or rotatable about the vertical axis and cover the target orientations B1, B2,... Bn described herein. . Images, videos or sub-scenes from each camera 2a-2b may be scanned or analyzed as described herein before and after optical correction.

  In FIG. 2F and FIG. 2G, the table head and end arrangement is shown, ie, each of the camera towers 14 shown in FIG. 2F and FIG. 2G is intended to be advantageously placed on the head of the conference table CT. Yes. As shown in FIGS. 3A-6A, a large flat panel display FP for presentation and video conferencing is often placed at the head or end of the conference table CT, and the arrangement of FIGS. It is placed close to the front. In the camera tower 14 arrangement of FIG. 2F, two cameras with a field of view of about 130 degrees are placed at 120 degrees with each other to cover two sides of a long conference table CT. A display and touch interface 12 is oriented on the table (especially useful when there is no flat panel FP on the wall) and displays the client for video conferencing software. The display 12 can be a connected, connectable or removable tablet or mobile device. In the camera tower arrangement of FIG. 2G, one high-resolution, arbitrarily tilted camera 7 (optionally connected to its own independent videoconferencing client software or instance) is the target object (whiteboard WB or table CT). Orientation such that two independently panning / tilting cameras 5a, 5b with a field of view of 100-110 degrees, for example, cover the target orientation. Can be attached or oriented.

  Images, videos or sub-scenes from each camera 2a, 2b, 5a, 5b, 7 can be scanned or analyzed as described herein before and after optical correction. FIG. 2H shows that two identical units, each having two cameras 2a-2b or 2c-2d at 100-130 degrees arranged 90 degrees apart, view> 180 degrees at the head or end of the table CT. It can be used independently as a unit, but more optionally substantially similar to the unit of FIG. 2A with four cameras 2a-2d suitably combined in the center of a conference table CT combined back to back and spanning the entire room. The deformation | transformation which can form the same unit is shown. Each of the tower units 14, 14 of FIG. 2H will be provided with a network interface and / or a physical interface to form a combined unit. The two units may alternatively or additionally be freely arranged, or may be arranged in concert as described with respect to FIGS. 2K, 6A, 6B and 14 below.

  In FIG. 2J, a fisheye camera or lens 3a similar to the camera of FIG. 2c (which can be physically and / or conceptually interchangeable with the catadioptric lens 3b) is the camera level of the camera tower 14 (8 to 15 inches). It is arranged on the top. One rotatable, high-resolution, optionally tilting camera 7 (optionally connected to its own independent videoconferencing client software or instance) is the target object (page on whiteboard WB or table CT plane) Or paper). As shown in FIG. 6A, FIG. 6B and FIG. 14, this arrangement is advantageous because the first videoconference client (in FIG. 14 “when on or connected to the conference room (local) display)” For example, via a first physical or virtual network interface or channel 10a, the synthesized sub-scene is received from the scene SC cameras 3a, 3b as a single camera image or a synthesized output CO and is sent to a second videoconference client (FIG. 14 is present in the device 100 and is connected to the Internet via a second physical or virtual network interface or channel 10b) when receiving an independent high resolution image from the camera 7.

  FIG. 2K shows a similar arrangement in which separate video conferencing channels for images from cameras 3a, 3b and 7 may be advantageous, but in the arrangement of FIG. 2K, each camera 3a, 3b pair 7 Each has its own tower 14 and is optionally connected to the rest of the tower 14 via an interface 15 (which may be wired or wireless). In the arrangement of FIG. 2K, the panoramic tower 14 with the scene SC cameras 3a, 3b may be placed in the center of the conference table CT, and the oriented high resolution tower 14 is directed to or at the head of the table CT. Alternatively, a high resolution, separate client image or video stream may be placed anywhere. Images, videos or sub-scenes from each camera 3a, 7 can be scanned or analyzed as described herein before and after optical correction.

Using Conference Cameras Referring to FIGS. 3A, 3B and 12, according to an embodiment of the present method for combining and outputting a photography scene, a device or conference camera 100 (or 200) is, for example, circular or rectangular It is placed on the conference table CT. The device 100 may be arranged according to the convenience or intention of the conference participants M1, M2, M3.

  In any typical conference, participants M1, M2,... Mn will be angularly distributed with respect to device 100. If the device 100 is placed in the middle of the participants M1, M2,... Mn, the participants can be captured with a panoramic camera as described herein. Conversely, if the device 100 is placed on one side of the participant (eg, placed at one end of the table or attached to the flat panel FP), a wide camera (eg, to span the participants M1, M2,... Mn) 90 degrees or more) may be sufficient.

  As shown in FIG. 3A, each of the participants M1, M2,... Mn will have their respective orientations B1, B2,... Bn from the device 100, measured for example from the origin OR for illustration. Each orientation B1, B2,... Bn can be a range of angles or nominal angles. As shown in FIG. 3B, an “unrolled”, projected or dewarped fisheye, panorama or wide scene SC is each participant arranged in their expected orientation B1, B2... Bn. M1, M2... Mn images are included. In particular, in the case of a rectangular table CT and / or an array of devices 100 on one side of the table CT, the image of each participant M1, M2,... Mn may be shortened or include perspective distortion according to the participant's facing angle ( 3B and schematically throughout the drawing with the expected shortening direction). Perspective and / or visual geometric correction, as is well known to those skilled in the art, can be applied to images, sub-scenes, or scenes SC that have shortened or perspective distortion, but may not be necessary.

Face Detection and Widening As an example, modern face detection libraries and APIs that use a common algorithm (for example, Android's FaceDetector. Face class, Objective C's CIDetector class and CIFaceFeature, among the more than 50 available APIs and SDKs) Object, OpenCV's CascadeClassifier class using the Haar cascade) usually returns the interpupillary distance and the spatial location of facial features and poses. If participant Mn's ears should be included in the range, a rough lower limit for face width estimation can be about twice the interpupillary distance / angle, and a rough upper limit is three times the interpupillary distance / angle. obtain. The rough lower limit for portrait width estimation (ie, the head plus some shoulder width) can be twice the face width / angle, and the rough upper limit can be four times the face width / angle. Alternatively, a fixed angle of the sub-scene width or other more direct setting may be used.

  4A-4B and 5A-5B show both face width and shoulder width, either of which can be the minimum width as described herein to set the initial sub-scene width. One exemplary two-stage and / or separate identification is shown. Faces set according to inter-pupil distance or other dimensional analysis of features (features, classes, colors, segments, patches, textures, trained classifiers, or other features), as shown in FIGS. 4A and 4B The widths FW1, FW2,... FWn are obtained from the panoramic scene SC. In contrast, in FIGS. 5A, 5B, 6A and 6B, shoulder widths SW1, SW2,... SWn are set according to the same analysis and scaled by about 3 or 4 times, or by default speech decomposition or width.

7A and 7B show five participants M1, M2, M3, M4, and M5, with visual minimum width Min.2 and corresponding angular range object orientation B5, and acoustic maximum. A top-down view of the use case of the conference camera 100 showing a conference table CT for about 10 people and a conference camera panoramic image signal SC, each including a depiction of the small Min.5 and the corresponding vector object orientation B2. FIG.

  In FIG. 7A, the conference camera 100 is arranged at the center of a long conference table CT for ten people. Accordingly, the participants M1, M2, M3 on the center side of the table CT have the smallest shortening and occupy the maximum image area and angle view of the camera 100, whereas the participants M5 and M4 on the end side of the table CT. Is the shortest and occupies the smallest image area.

  In FIG. 7B, the entire scene video signal SC is a 360-degree video signal, for example, and includes all participants M1... M5. The conference table CT appears in the scene SC with the distorted “W” shape characteristics of the panoramic view, whereas the participants M1... M5 have different sizes depending on their position and distance from the conference camera 100. Appear with different abbreviated aspects (simply represented simply by a rectangular body and an elliptical head). As shown in FIGS. 7A and 7B, each participant M1... M5 is represented in memory 8 by their orientation B1... B5 and can be determined by acoustic or visual or sensor localization of sounds, actions or features. . As depicted in FIGS. 7A and 7B, participant M2 has been localized by face detection (and the corresponding vector-like orientation B2 and minimum width Min.2 are recorded in memory, Participant M5 is localized by the time of flight of an audio signal such as beamforming, relative signal strength, and / or utterance (which may be determined in proportion to the face width obtained from the face detection heuristic). (And the corresponding sector-like orientation B5 and minimum width Min.5 are recorded in the memory and are obtained in proportion to the approximate resolution of the acoustic array 4).

  FIG. 8A shows a conference camera 100 video signal and a minimum width Min. n, sub-scene video signals SS2 and SS5 to be combined with stage scene video signals STG and CO, and panoramic video signal SC. Schematic diagram with R extraction. The upper part of FIG. 8A essentially reproduces FIG. 7B. As shown in FIG. 8A, the entire scene video signal SC from FIG. 7B is limited to the target orientation (in this example limited to orientations B2 and B5) and width (in this example limited to widths Min.2 and Min.5). Subsampled). The sub-scene video signal SS2 includes a face width limit Min. 2 may be wider than the width, height, and / or available area of the stage STG, or the aspect ratio and available area of the composite output CO, or more It may be widely scaled. The sub-scene video signal SS5 includes an acoustic approximate Min. It may be at least as wide as 5, but it may be wider as well, or may be more widely scaled and limited. A reduced panorama scene SC. R is a version in which the upper and lower sides of the entire scene SC are cropped, and in this case, it is cropped to an aspect ratio of 10: 1. Instead, a reduced panoramic scene SC. R may be obtained from the entire panoramic scene video signal SC by proportional scaling or anamorphic scaling (eg, the upper and lower portions remain but are compressed more than the central portion). In any case, in the example of FIGS. 8A and 8B, three different video signal sources SS2, SS5 and SC. R can be used to be combined with the stage STG or the combined output CO.

  FIG. 8B essentially reproduces the lower part of FIG. 8A, and sub-scene video signals SS2 and SS5 and panoramic video signal SC. A schematic diagram of R is shown. 8C-8E show three possible composite outputs or stage scene video signals STG or CO.

  In the synthesized output CO or stage scene video signal STG shown in FIG. R is synthesized across the entire top of the stage STG, which in this case occupies less than 1/5 or 20% of the stage area. The sub-scene SS5 is synthesized so as to occupy at least the minimum area thereof, and is not scaled as a whole, but is widened to satisfy about ½ of the stage width. The sub-scene SS2 is also synthesized so as to occupy at least the (substantially small) minimum area, is not scaled as a whole, and is also widened to satisfy about ½ of the stage width. In this composite output CO, the two sub-scenes are given approximately the same area, but the participants have different apparent sizes corresponding to their distance from the camera 100. Also, the left and right or clockwise order of the two synthesized sub-scenes is the order of the target orientation from the room participants or the camera 100 (and appearing in the reduced panoramic view SC.R). It should be noted that they are the same. Further, any of the transitions described herein can be used in combining the sub-scene video signals SS2, SS5 into the stage video signal STG. For example, both sub-scenes may simply fill stage STG instantaneously, or after one slides in from its corresponding left and right stage direction to fill the entire stage, the other fills its corresponding left and right It may be progressively narrower by sliding in from the stage direction, etc. In any case, the sub-scene window, frame, outline, etc. display the video stream throughout the transition.

  In the synthesized output CO or the stage scene video signal STG shown in FIG. R is similarly synthesized into the scene STG, but each of the signals SS5 and SS2 is scaled or zoomed proportionally so that the participants M5 and M2 occupy a larger area of the stage STG. The minimum width of each signal SS5 and SS2 is also zoomed in, and signals SS5 and SS2 still occupy more than their minimum widths, but each is widened to fill approximately half of the stage. (In the case of SS5, the minimum width occupies 1/2 of the stage). Participants M5 and M3 are substantially the same size on stage STG or in composite output signal CO.

  In the combined output CO or stage scene video signal STG shown in FIG. 8E, the reduced panoramic video signal SC. R is similarly synthesized into the scene STG, but each of the signals SS5 and SS2 is scaled or zoomed depending on the situation. Sub-scene signals SS5 and SS2 still occupy more than their minimum widths, but each has been widened to fill a different amount of stage. In this case, the sub-scene signal SS5 is not scaled up or zoomed, but has a wider minimum width and occupies more than 2/3 of the stage SG. On the other hand, the minimum width of the signal SS2 is zoomed and occupies about three times the minimum width. One situation in which the relative proportions and conditions of FIG. 8E occur is that there is no visual localization for participant M5 and a wide uncertainty (low confidence level) object orientation and a wide minimum width can be given. , And moreover, it may be a case where the participant M5 keeps speaking for a long time and arbitrarily increases the occupation ratio of the sub-scene SS5 of the stage STG. At the same time, participant M2 may have reliable face width detection, allowing sub-scene SS2 to be scaled and / or widened to consume an area larger than its minimum width.

  9A also shows the conference camera 100 video signal and the minimum width Min. n, an alternative sub-scene video signal SSn to be combined with the stage scene video signal, and an alternative panoramic video signal SC. Schematic diagram with R extraction. In the upper part of FIG. 9A, the participant M1 is the latest speaker, and the corresponding sub-scene SS1 corresponds to the minimum width Min. Except for having 1, essentially reproduces FIG. 7B. As shown in FIG. 9A, the entire scene video signal SC from FIG. 7B follows the target orientation (here, orientations B1, B2, and B5) and width (here, widths Min.1, Min.2, and Min.5). Can be subsampled. Each of the sub-scene video signals SS1, SS2 and SS5 has its minimum width Min. (Determined visually, acoustically or with a sensor). 1, Min. 2 and Min. 5 may be wider than the width, height, and / or available area of the stage STG or the aspect ratio and available area of the composite output CO, or at least as wide as 5 It may be scaled. A reduced panorama scene SC. R is a cropped version of the entire scene SC, in this case spanning only the most relevant / most recent speakers M1, M2 and M5 with an aspect ratio of about 7.5: 1. To be cropped. In the example of FIGS. 9A and 9B, four different video signal sources SS1, SS2, SS5 and SC. R can be used to be combined with the stage STG or the combined output CO.

  FIG. 9B essentially reproduces the lower part of FIG. 9A and shows a schematic diagram of a sub-scene video signal and a panoramic video signal to be combined with the stage scene video signal. 9C-9E show three possible composite outputs or stage scene video signals.

  In the combined output CO or stage scene video signal STG shown in FIG. 9C, the reduced panoramic video signal SC. R is synthesized almost completely across the top of the stage STG, and in this case occupies less than ¼ of the stage area. The sub-scene SS5 is again synthesized so as to occupy at least the minimum area thereof, and is not scaled as a whole, but is widened to satisfy about 1/3 of the stage width. Sub-scenes SS2 and SS1 are also synthesized to occupy at least their smaller minimum area, are not entirely scaled, and are each widened to meet about 1/3 of the stage width. Yes. In this composite output CO, the three sub-scenes are given approximately the same area, but the participants have different apparent sizes corresponding to their distance from the camera 100. The left or right or clockwise order of the two sub-scenes that have been synthesized or transitioned is the order of the subject orientation from the room participants or the camera 100 (and as appearing in the reduced panoramic view SC.R). Remains the same. Further, any of the transitions described herein can be used when combining the sub-scene video signals SS1, SS2, SS5 into the stage video signal STG. In particular, the transition is a reduced panoramic view SC. It is more comfortable in the same left-right order as R or as a sliding transition approaching from that order (for example, if M1 and M2 are already on the stage, M5 slides in from the right of the stage and M1 and M5 are If it is already on the stage, M2 slides in between them from above or below, and if M2 and M5 are already on the stage, M1 slides in from the left of the stage and panoramic view SC. R's M1, M2, M5 order should be preserved).

  In the combined output CO or stage scene video signal STG shown in FIG. 9D, the reduced panoramic video signal SC. R is similarly synthesized into the scene STG, but each of the signals SS1, SS2, and SS5 is scaled or zoomed proportionally so that the participants M1, M2, M5 occupy a larger area of the stage STG. . The minimum width of each signal SS1, SS2, SS5 is also zoomed in and the signals SS1, SS2, SS5 still occupy more than their zoomed minimum width, but the sub-scene SS5 is zoomed on the stage. It has been widened to fill an area slightly larger than the minimum width, SS5 occupies 60 percent of the stage width, SS2 only occupies 15 percent, and SS3 occupies the remaining 25 percent is occupying. Participants M1, M2, and M5 are substantially the same height or face size on stage STG or in composite output signal CO, but participant M2 and sub-scene SS2 are from head and / or body width. May be substantially cropped to show only a slightly larger area.

  In the combined output CO or stage scene video signal STG shown in FIG. 9E, the reduced panoramic video signal SC. R is similarly synthesized into the scene STG, but each of the signals SS1, SS2, SS5 is scaled or zoomed according to the situation. The sub-scene signals SS1, SS2, SS5 still occupy more than their minimum widths, but each has been widened to satisfy a different amount of stage. In this case, none of the sub-scene signals SS1, SS2, SS5 has been scaled up or zoomed, but the sub-scene SS1 with the most recent or most relevant speaker M1 has an area larger than ½ of the stage SG. is occupying. On the other hand, each of the sub-scenes SS2 and SS5 occupies a smaller or reduced occupancy ratio of the stage STG, but since the sub-scene SS5 has a minimum width, a further decrease in the occupancy ratio of the stage STG is sub-scene. Taken from SS2 or SS1. One situation in which the relative proportions and conditions of FIG. 9E occur is that visual localization may be performed for participant M1, but participant M1 continues to speak for a long time, in sub-scene SS1 of stage STG. It may be a case where the occupation ratio is arbitrarily increased with respect to the other two sub-scenes.

  9F or the reduced panorama scene SC. In R, the conference camera 1000 is not at the center of the table CT, but instead is placed at one end of the table CT (eg, as indicated by the dashed line position on the right side of FIG. 7A), and the flat panel FP identifies the remote conference participants. Show. In this case, the conference table CT also appears in a “W” shape that is greatly distorted. As shown at the top of FIG. 9F, if the index direction or origin OR of the conference camera 100 or panoramic scene SC is oriented so that the limit of the high aspect ratio panoramic scene SC “divides” the conference table CT, the table CT It is very difficult to refer to the position of the person around. However, if the conference camera 100 or panoramic scene index direction or origin OR is arranged so that the table CT is continuous and / or positioned so that everyone is facing one side, the scene is more natural Become. According to this embodiment, the processor 6 can perform image analysis to change the index position or the starting position of the panoramic image. In one example, the index position or origin position of the panoramic image is “rotated” so that the area of a single continuous segmentation of the image patch corresponding to the table area is maximized (eg, the table does not break). obtain. In another example, the index position or origin position of the panoramic image may be “rotated” so that the two closest or largest face recognitions are furthest away from each other (eg, the table does not break). In the third example, in another example, the index position or origin position of the panoramic image is located at the panoramic edge where the lowest height segmentation of the image patch corresponding to the table area is located (eg, the “W” shape is rotated). , Place the table edge closest to the conference camera 100 on the panorama edge).

  FIG. 10A shows a schematic diagram of a possible composite output CO or stage scene video signal STG, which substantially reproduces the composite output signal CO or stage video signal STG of FIG. It is synthesized to occupy less than a quarter of the top of the STG, and three different sub-scene video signals are synthesized to occupy the remaining different amounts of the stage STG. FIG. 10B shows an alternative schematic of possible combined output or stage scene video signals, where three different sub-scene video signals adjacent to each other are combined to occupy different amounts of stage STG or combined output signal CO. ing.

  11A and 11B show schematic diagrams of two alternative ways in which video conferencing software may display a composite output or stage scene video signal. In FIGS. 11A and 11B, the combined output signal CO is as a single camera signal (eg, via a USB port) with accompanying audio (optionally mixed and / or beamformed to enhance the current speaker's voice). ) Received and integrated into a video conferencing application as a single camera signal. As shown in FIG. 11A, each single camera signal is provided with a separate window, and a selection signal such as a composite output signal CO or an active signal or foreground signal is reproduced as a thumbnail. In contrast, in the example shown in FIG. 11B, the selected single camera signal is given the largest possible area on the display as practical, and the selection signal such as the composite output signal CO or the active or foreground signal. Are displayed as shaded thumbnails or grayed-out thumbnails.

Sub-Scene Identification and Compositing As shown in FIG. 12, when new sub-scenes SS1, SS2,... SSn are recognized in, for example, the panoramic video signal SC in step S10, they can be generated and tracked according to the scene. Thereafter, in step S30, the sub-scenes SS1, SS2,... SSn can be combined according to the target orientation, conditions, and recognition described herein. The composite output or stage scene STG, CO can then be output at step 50.

  In the additional details shown in FIG. 13 and as shown in FIGS. 3A to 7B (including FIGS. 3A and 7B), in step S12, the device 100 includes one or more at least partially panoramic cameras 2. Alternatively, the scene SC is captured at a wide angle (for example, an angle of 90 to 360 degrees) with an angle of view of at least 90 degrees from 2a.

  Subsequent processing for tracking and sub-scene identification may be performed on a native, undistorted or unstitched scene SC, or unrolled, distorted corrected, or It may be performed on the stitched scene SC.

  In step S14, new object orientations B1, B2,... Bn are obtained from the wide angle view SC using one or more of beamforming, recognition, identification, vectorization, or homing techniques.

  In step S16, one or more new orientations can be obtained from the initial angular range (eg, 0-5 degrees), typical human head, and / or typical human shoulder, or other default width ( Widened to an angular range sufficient to span (e.g., measured in pixels or angular ranges). It should be noted that the analysis order may be reversed. For example, a face may be detected first and then the orientation to the face may be obtained. The widening may be performed in one, two, or more steps, and the two steps described herein are merely examples. Also, “widening” does not require a gradual widening process, for example, “widening” can mean setting the angular range directly based on detection, recognition, threshold, or value. Different methods may be used to set the sub-scene angle range. In some cases, such as when two or more faces are close to each other, the “broadening” may be selected to include all of these faces even if there is only one face in the correct target orientation B1.

  In step S16 (and as shown in FIGS. 5A and 5B), the shoulder width sub-scene SS1, SS2,... SSn is the interpupillary distance or other facial, head, torso or other visible feature (feature, class , Colors, segments, patches, textures, trained classifiers, or other features) may be set or adjusted as in step S18 according to measurements taken from the scene SC. The width of the sub-scenes SS1, SS2,... SSn can be set according to the shoulder width (alternatively according to the face width FW), or alternatively as a predetermined width related to the angular resolution of the audio microphone array 4.

  Instead, in step S16, the upper and / or lower limits for the sub-scene width for each or all subject orientations are set in step S18, eg, as a peak, average, or representative shoulder width SW and face width FW, respectively. Or it can be adjusted. The notation of FW and SW is herein a “face width” FW or “shoulder width” SW (ie, a face or shoulder span to be captured angularly as a sub-scene), and a result representing the face width FW or shoulder width SW. To be used interchangeably as a typical face width or shoulder width sub-scene SS (ie, a block of pixels or a corresponding width sub-scene identified, obtained, adjusted, selected or captured from a wide scene SC) Should be noted.

  In step S16, or alternatively or in addition to steps S16 to S18, the first individual sub-scene with an angle of view of at least 20 degrees (eg FW1 and / or SW1) is the first target orientation B1, B2,. Obtained from the wide-angle scene SC. Instead of or in addition to the setting of an angle of view (eg FW1 and / or SW1) of at least 20 degrees, the first individual sub-scene FW1 and / or SW1 is (eg unique to M1 or M1, M2 As an angle of view spanning at least two to twelve times the interpupillary distance (representing Mn), or alternatively or additionally, the interpupillary distance (eg, characteristic of M1 or M1, M2 ... Mn) It can be obtained from the wide-angle scene SC as an angle of view scaled to capture the width between the shoulder width (eg, M1 specific or representing M1, M2... Mn). Sub-scene capture with a wider or shoulder width SWn may record a narrower face width FWn for later reference.

  If the second target orientation B1, B2,... Bn is available, the second individual sub-scene (eg FW2 and / or SS2) is replaced in step S16 or alternatively or in addition in steps S16 to S18. For example, it is similarly obtained from the wide-angle view SC in the second target orientation, which is B2. If successive object orientations B3... Bn are available, successive individual sub-scenes (eg FW3... N and / or SS3... N) are similarly obtained from the wide angle view SC in the successive object orientations B3. .

  The first and second target orientations B1, B2 (and subsequent target orientations B3... Bn) are the same device 100 regardless of whether they are obtained by stitching different camera images or from a single panoramic camera. So that it can have a substantially common angular origin with respect to the first target orientation. Optionally, one or more additional target orientations Bn from different angular origins can be obtained from another camera 5 or 7 of device 100 or from a connected device (eg, connected laptop, tablet, or FIG. 1A). It can be obtained from a camera on the mobile device 40 or a connected satellite camera 7) on the satellite tower 14b of FIG. 2K.

  As described above, the set, obtained, or widened sub-scene SS representing the width FW or SW is, for example, (i) such that it is of a size equivalent to or consistent with other sub-scenes. (Ii) can be uniformly divided or divided (eg, 2, 3 or 4) with respect to the aspect ratio of the output image or stream signal, optionally not less than the lower limit of the above width or not exceeding the upper limit. (Iii) to avoid overlap with other sub-scenes in near object orientations and / or (iv) brightness, contrast, or other video with other sub-scenes It can be adjusted in step S18 so that the characteristics match.

  In step S20 (which may include modes 1, 2, or 3 from FIGS. 16-18 in a reasonable and operable combination), the identified object orientations B1, B2... Bn and sub-scenes FW1, FW2. And / or data and / or metadata regarding SS1, SS2... SSn may be recorded for tracking purposes. For example, the relative position from the origin OR (determined by sensor or calculation, for example), width, height, and / or any of the adjusted parameters described above can be recorded.

  Alternatively, characteristic data, prediction data or tracking data associated with the sub-scene may be recorded at step S20 and added to, for example, a sub-scene, orientation, or other feature tracking database at step S20. For example, the sub-scenes FW1, FW2... FWn and / or SS1, SS2... SSn can be instantaneous images, image blocks or video blocks identified in the image or video scene SC. In the case of video, depending on the video compression / decompression approach, prediction data can be associated with a scene or sub-scene and recorded as data or metadata associated with the sub-scene, but additional new sub-scenes to track Tend to be a part of.

Following the recording of tracking data or other target data, processing returns to the main routine.
Sub-scene synthesis for each situation In step S30 of FIG. 12, the processor 6 performs for each situation (for example, for each data, flag, indicator, setting, or other action parameter recorded as tracking data or scene data in step S20). To) sub-scene SSn, ie first, optionally second, and optionally subsequent individual sub-scene SSn corresponding to different widths FW1, FW2... FWn and / or SW1, SW2. Are combined into a composite scene or single camera image or video signal STG or CO. As used herein, a single camera image or video signal STG, CO is a USB (or other peripheral bus or network) peripheral image or video signal corresponding to a single USB (or other peripheral bus or network) camera or It may refer to a single video frame representing a stream or a single composite video frame.

  In step S32, the device 100, its circuitry, and / or its executable code may identify the related sub-scene SSn to be arranged as a combined and combined image or video stream STG or CO. “Relevant” may be determined according to the criteria set forth with respect to identification at step S14 and / or updating and tracking at step S20. For example, one related sub-scene may be the most recent speaker sub-scene and the second related sub-scene may be the second most recent speaker sub-scene. These two most recent speakers may continue to be the most relevant until the third speaker becomes more relevant by speaking. Embodiments herein include three speakers in a sub-scene in a composite scene, each of which is an equal width segment, or a segment that is wide enough to hold its own head and / or shoulder have. However, two speakers or four speakers or more speakers can also be easily accommodated with a wider or narrower occupancy of the combined screen width, respectively.

  By selecting a sub-scene SSn that encapsulates the face only in height and width, a maximum of eight speakers can be reasonably accommodated (eg, four at the top of the composite scene and four at the bottom), 4 An array of 8 to 8 speakers can be used to buffer and compose appropriate screens and / or windows (sub-scenes corresponding to windows) (eg, sub-scenes as a deck of overlapping cards or more The related speakers are large in the foreground and the less related speakers present as a shortened ring of small, deep views. With reference to FIGS. 6A and 6B, the system determines that the most relevant scene to be displayed is WB (eg, when captured by the second camera 7 as depicted in FIG. 6A). If so, the scene SSn may further include whiteboard content WB. The whiteboard or whiteboard scene WB is prominently presented and may occupy most or most of the scene, whereas the speakers M1, M2 ... Mn or SPKR can optionally be presented in picture-in-picture along with the whiteboard WB content .

  In step S34, the associated sub-scene set SS1, SS2,... SSn is compared with the previously associated sub-scene SSn. Steps S34 and S32 may be performed in the reverse order. Depending on this comparison, the previously associated sub-scene SSn is available, should remain on the stage STG or CO, should be removed from the stage STG or CO, smaller or larger It is determined whether it should be reconstructed to size or perspective, or else it needs to be modified from a previously synthesized scene or stage STG or CO. When a new sub-scene SSn is to be displayed, there may be too many candidate scenes SSn for scene change. For example, in step S36, a scene change threshold may be ascertained (this step may be performed before or during steps S32 and S34). For example, when the number of individual sub-scenes SSn is larger than a threshold number (for example, 3), the entire wide-angle scene SC or the reduced panoramic scene SC. It may be preferable to output R (eg, as it is or segmented and stacked to fit within the aspect ratio of the USB peripheral device camera). Instead, it may be best to present a single camera scene instead of a composite scene of multiple sub-scenes SSn or as a composite output CO.

  In step S38, the device 100, its circuitry, and / or its executable code may set the sub-scene members SS1, SS2,... SSn and the order in which they transition to the composite output CO and / or are composited. In other words, once it is determined whether the candidate members of the sub-scene complements SS1, SS2,... SSn to be output as stage STG or CO and any rules or thresholds for scene changes are met or exceeded, The order of SSn and the transition in which they are added, removed, switched or rearranged may be determined in step S38. It should be noted that step S38 is more or less important depending on the previous steps and the history of the speakers SPKR or M1, M2. If two or three speakers M1, M2... Mn or SPKR are identified and should be displayed at the same time as the device 100 begins to operate, step S38 starts with a blank sheet and follows the default associated rules. (For example, start with 3 or fewer speakers in composite output CO, presenting speaker SPKR clockwise). If the same three speakers M1, M2,... Mn continue to be related, the sub-scene members, order, and composition may not be changed in step S38.

  As described above, the composite output CO can be changed in steps S32 to S40 by the identification described with respect to step S18 and the prediction / update described with respect to step S20. In step S40, the transition and synthesis to be performed are determined.

  For example, device 100 may obtain subsequent (eg, third, fourth, or subsequent) individual sub-scene SSn from a wide-angle or panoramic scene SC in a subsequent subject orientation. In steps S32 to S38, the subsequent sub-scene SSn can be set to be combined or combined with the combined scene or combined output CO. Further, in steps S32-S38, another sub-scene SSn other than the subsequent sub-scene (eg, the previous or less relevant sub-scene) is set to be removed from the composite scene (by the composite transition). Obtained (and synthesized and output as a synthesized scene CO or a synthesized output CO in step S50).

  As an additional example or alternative, the device 100 in steps S32-S38 may include additional criteria as described with reference to steps S18 and / or S20 (eg, speech time, speech frequency, audible frequency cough / sneeze / The sub-scene SSn to be synthesized or combined with the synthesized scene or synthesized output CO, or to be removed from the synthesized scene or synthesized output CO, according to the settings of the doorbell, sound amplitude, speech angle and face recognition) Can do. In steps S32 to S38, only subsequent sub-scenes SSn that meet the additional criteria may be set to be combined with the composite scene CO. In step S40, the transition and synthesis steps to be performed are determined. The stage scene is then combined and output as a combined output CO formatted as a single camera scene in step S50.

  As an additional example or alternative, the device 100 in steps S32-S38, the retention criteria as described with reference to steps S18 and / or S20 (eg, voice / speech time, voice / speech frequency, last speech, etc.) Sub-scene SSn may be set as a protected sub-scene protected from removal. In steps S32 to S38, removing the sub-scene SSn other than the subsequent sub-scene does not set the protection sub-scene to be removed from the composite scene. In step S40, the transition and synthesis to be performed are determined. The combined scene is then combined and output as a combined output CO formatted as a single camera scene in step S50.

  As an additional or alternative, the device 100 rotates in steps S32-S38 in an emphasis criterion (eg, repeat speaker, designated presenter, most recent speaker, loudest speaker, in hand / scene change). Sub-scene SSn enhancement operations (eg, scaling, blinking, genie, bouncing, card sorting) as described with reference to steps S18 and / or S20 based on object, high frequency scene activity in frequency domain, raised hand) , Ordering, cornering). In steps S32 to S38, at least one of the individual sub-scenes SSn may be set to be enhanced according to the sub-scene enhancement operation based on their own or corresponding enhancement criteria. In step S40, the transition and synthesis to be performed are determined. The combined scene is then combined and output as a combined output CO formatted as a single camera scene in step S50.

  As an additional example or alternative, the device 100 may perform sub-steps as described with reference to steps S18 and / or S20 in steps S32-S38 based on sensors or sensed criteria (eg, too quiet, remote pork). A scene participant notification or reminder action (eg, blinking light to a person on the side of a sub-scene) may be set. In steps S32-S38, local reminder indicators may be set to be activated according to notifications or reminder actions based on their own or corresponding sensed criteria. In step S40, the transition and synthesis to be performed are determined. The combined scene is then combined and output as a combined output CO formatted as a single camera scene in step S50.

  In step S40, the device 100, its circuitry, and / or its executable code generates a transition and composition to smoothly change the sub-scene complement of the composite image. Following the synthesis of the tracking data or other subject data synthesis output CO, processing returns to the main routine.

Composite Output In steps S52-S56 of FIG. 15, (optionally in reverse order), the composite scene STG or CO is formatted, i.e., combined and / or transitioned, to be transmitted or received as a single camera scene. Is rendered or composited into a buffer, screen or frame (where “buffer”, “screen” or “frame” corresponds to a single camera view output). Device 100, its circuitry, and / or its executable code uses a synthesis window or screen manager, optionally with GPU acceleration, to provide an off-screen buffer for each sub-scene, and to use the buffer as peripheral and transition graphics. Together, it can be combined into a single camera image representing a single camera view, and the result written to output or display memory. The composite window or subscreen manager circuit performs blending, fading, scaling, rotation, replication, bending, twisting, shuffling, blurring, or other processing on the buffered window, or flip switching, Drop shadows and animations such as stack switching, cover switching, ring switching, grouping, tiling can be rendered. The composite window manager may provide a visual transition that can be composited so that sub-scenes entering the composite scene are added, removed, or switched with transition effects. The sub-scene fades in or out, visually shrinks in or out, and smoothly spreads radially inward or outward. All scenes being synthesized or transitioned can be video scenes, for example, each containing an ongoing video stream subsampled from a panoramic scene SC.

  In step S52, the transition or composition is rendered into a frame, buffer, or video memory (repeatedly, incrementally, or continuously as needed) (note that transition and composition are individual frames or video streams). And can be an ongoing process through many frames of video of the entire scene STG, CO and the individual constituent sub-scenes SS1, SS2... SSn.

  In step S54, the device 100, its circuitry, and / or its executable code may select and transition an audio stream. Similar to the window, scene, video, or sub-scene composition manager, the audio stream may or may not be enhanced to emphasize the sub-scene being synthesized, especially in the case of the beams forming the array 4. Also good. Similarly, the audio may be synchronized to the synthesized video scene.

  In step S56, the device 100, its circuitry, and / or its executable code outputs a single camera video and audio simulation as a composite output CO. As described above, this output simulates a single, eg, webcam view, of a peripheral USB device, eg, an aspect ratio less than 2: 1 and an aspect ratio typically less than 1.78: 1. Aspect ratio and number of pixels, which can be used as external webcam input by group video conferencing software. When rendering the webcam input as a display view, the video conferencing software treats the composite output CO as any other USB camera and interacts with the host device 40 (or the directly connected device 100 version of FIG. 1B). Presents the composite output CO in all main and thumbnail views corresponding to the host device (or a version of the directly connected device 100 of FIG. 1B).

Example of Sub-Scene Synthesis As described with reference to FIGS. 12 to 16, the conference camera 100 and the processor 6 synthesize (in step S30) the single camera video signals STG, CO and output (in step S50). Can do. A processor 6 operably connected to the ROM / RAM 8 has an aspect ratio substantially greater than 2.4: 1 captured from wide cameras 2, 3 and 5 having a horizontal field of view substantially greater than 90 degrees. Can be recorded (in step S12). In one optional version, the panoramic video signal is captured from a wide camera having an aspect ratio substantially greater than 8: 1 and having a horizontal field of view of substantially 360 degrees.

  The processor 6 (for example in step S14) receives at least two sub-scene video signals SS1, SS2... SSn (for example in FIGS. 8C to 8E and FIGS. 9C to 9E) in the respective target orientations B1, B2. SS2 and SS5) may be subsampled (eg, in steps S32-S40). The processor 6 arranges two or more sub-scene video signals SS1, SS2... SSn (for example, SS2 and SS5 in FIGS. 8C to 8E and 9C to 9E) (buffers, frames, or video memories in steps S32 to S40). To) to form (in steps S52-S56) a stage scene video signal CO, STG having an aspect ratio of substantially less than 2: 1. Optionally, to fill as much of the single camera video signal as possible (leading to a larger view of the participants), substantially 80% or more of the area of the stage scene video signal CO, STG is a panoramic video signal. Can be subsampled from the SC. A processor 6 operably connected to the USB / LAN interface 10 may output stage scene video signals CO, STG that are formatted as single camera video signals (as in steps S52-S56).

  Optimally, the processor 6 is from the panoramic video signal SC (and / or optionally, for example in the GPU 6 and / or ROM / RAM 8, from a buffer, frame or video memory, and / or the wide camera 2, 3, 5 (For example, SS1 in FIGS. 9C-9E) additional sub-scene video signals SS1, SS2... SS3 (for example, third, fourth, or subsequent) in their respective orientations B1, B2,. Sub-sampling can be performed. The processor then converts two or more sub-scene video signals SS1, SS2... SS3 (eg SS2 and SS5 in FIGS. 9C-9E) originally synthesized on stages STG, CO into one or more additional sub-scenes. Combined with video signals SS1, SS2... SSn (eg, SS1 in FIGS. 9C-9E), and having a plurality of side-by-side sub-scene video signals (eg, in a row) having an aspect ratio of substantially 2: 1 or less. Alternatively, stage scene video signals STG, CO may be formed that include two, three, four or more sub-scene video signals SS1, SS2,. The processor 6 may set or store in a memory one or more additional criteria for one or more target orientations or sub-scene video signals SS1, SS2,. In this case, for example, only those additional sub-scene video signals SS1, SS2,... SSn that meet additional criteria (eg, sufficient quality, sufficient illumination, etc.) may transition to the stage scene video signals STG, CO.

  Alternatively or additionally, the additional sub-scene video signals SS1, SS2... SSn replace one or more of the sub-scene video signals SS1, SS2... SSn that may have already been combined in the stages STG, CO. It can be synthesized by the processor 6 to the stage scene video signals STG, CO by forming a stage scene video signal STG, CO that still has an aspect ratio of substantially less than 2: 1. Each sub-scene video signal SS1, SS2,... SSn to be synthesized has a minimum width Min. 1, Min. 2 ... Min. n and each sub-scene video signal SS1, SS2,... SSn is substantially its minimum width Min. 1, Min. 2 ... Min. Stage scene video signals STG and CO can be formed by being combined side by side with n or more.

  In some cases, for example, in steps S16-S18, the processor 6 determines the combined width of each sub-scene video signal SS1, SS2,... SSn in transition to a minimum width Min. 1, Min. 2 ... Min. It can be increased to increase throughout the transition until n or more. Alternatively or additionally, each sub-scene video signal SS1, SS2,... SSn is substantially at its minimum width Min. 1, Min. 2 ... Min. Each SS1, SS2,... SSn is greater than or equal to n and the sum of all synthesized sub-scene video signals SS1, SS2... SSn is substantially equal to the width of the stage scene video signal or synthesized output STG, CO. Can be combined side by side by the processor 6 in width.

  Alternatively or additionally, the width of the sub-scene video signals SS1, SS2,... SSn in the stage scene video signals STG, CO is one or more target orientations B1, B2,. Synthesized by processor 6 to vary according to one or more activity criteria detected in Bn (eg, visual motion, sensed motion, acoustic detection of speech, etc.) (eg, as in steps S16-S18). On the other hand, the width of the stage scene video signal or the synthesized output STG, CO is kept constant.

  Optionally, the processor 6 converts one or more sub-scene video signals SS1, SS2 ... SSn (eg SS2 and SS5 in FIGS. 9C-9E) to one or more additional sub-scene video signals SS1, SS2 ... SSn (eg Combined with SS1) in FIGS. 9C-9E, the width of one or more sub-scene video signals SS1, SS2,... SSn (eg, SS2 and SS5 in FIGS. 9C-9E) is one or more. One or more additional sub-scene video signals SS1, SS2 by reducing by an amount corresponding to the width of the added or subsequent sub-scene video signals SS1, SS2... SSn (eg SS1 in FIGS. 9C-9E). ... SSn (for example, SS1 in FIGS. 9C to 9E) is transferred to the stage scene video signals STG and CO. Therefore, to form a stage scene video signal.

  In some cases, the processor 6 uses its minimum width Min. For each sub-scene video signal SS1, SS2,. 1, Min. 2 ... Min. n and assign each sub-scene video signal SS1, SS2... SSn substantially to its corresponding minimum width Min. 1, Min. 2 ... Min. A stage scene video signal or composite output STG, CO can be formed by combining side by side with n or more. The minimum width Min. Of each of the two or more sub-scene video signals SS1, SS2... SSn, together with one or more additional sub-scene video signals SS1, SS2. 1, Min. 2 ... Min. When the sum of n exceeds the width of the stage scene video signal STG, CO, one or more of the two sub-scene video signals SS1, SS2,... SSn are removed from the stage scene video signal or the synthesized output STG, CO. It can be migrated by the processor 6.

  In another alternative, the processor 9 may have its own target that most recently met one or more activity criteria (eg, visual motion, sensed motion, acoustic detection of speech, time since last speech, etc.). At least one of the two or more sub-scene video signals SS1, SS2,... SSn to be transferred so as to be removed from the stage scene video signals STG, CO may be selected so as to correspond to the orientations B1, B2,.

  In many cases, as shown in FIGS. 8B-8E and FIGS. 9B-9E, processor 6 includes two or more sub-scene video signals SS1, SS2,... SSn (eg, SS2 and SS5 in FIGS. 9C-9E) and From left to right for wide cameras 2, 3, 5 between their respective orientations B1, B2,... Bn of one or more additional sub-scene video signals SS1, SS2... SSn (eg SS1 in FIGS. 9C-9E). Two or more subscene video signals SS1, SS2,... SSn are combined with at least one subsequent subscene video signal SS1, SS2. It can be preserved when forming the output STG, CO.

  Alternatively or additionally, the processor 6 may use one or more selection criteria (eg, visual motion, sensed motion, utterances) detected in the respective target orientations B1, B2,. Depending on the sound detection, the time since the last utterance, etc., the respective target orientations B1, B2,... Bn from the panoramic video signal SC can be selected. After one or more selection criteria are no longer true, the processor 6 may transition to remove its corresponding sub-scene video signal SS1, SS2,... SSn from the stage scene video signal or composite output STG, CO. The selection criteria may include the presence of activity criteria that are met in their target orientations B1, B2,. The processor 9 may count the time since one or more activity criteria are met in each of the target orientations B1, B2,. The processor 6 removes its sub-scene signal SS1, SS2,... SSn from the stage scene video signal STG for a predetermined period after one or more activity criteria are met in its target orientation B1, B2,. Can be transitioned to.

  8A to 8C, 9A to 9C, 10A, 1B, 11A, 11B, and 22, the reduced panoramic video signal SC. With respect to R, the processor 6 determines from the panorama video signal SC that the panorama video signal SC. R can be subsampled. The processor 6 then reduces the panoramic video signal SC.2 from a reduced version of two or more sub-scene video signals (eg SS2 and SS5 in FIGS. 8C-8E and 9C-9E). R and a plurality of side-by-side sub-scene video signals (for example, SS2 and SS5 in FIGS. 8C to 8E, SS1, SS2 and SS5 in FIGS. 9C to 9E) and the panoramic video signal SC. Stage scene video signals STG and CO having an aspect ratio of substantially 2: 1 or less, including R, can be formed.

  In this case, the processor 6 reduces the panoramic video signal SC.2 obtained by reducing two or more sub-scene video signals (for example, SS2 and SS5 in FIGS. 8C to 8E and SS1, SS2 and SS5 in FIGS. 9C to 9E). Combined with R, a plurality of side-by-side sub-scene video signals (for example, SS2 and SS5 in FIGS. 8C to 8E, SS1, SS2 and SS5 in FIGS. 9C to 9E) and a plurality of side-by-side sub-scene video signals High panoramic video signal SC. And a stage scene video signal having an aspect ratio of substantially 2: 1 or less, including R, wherein the panoramic video signal is 1/5 or less of the area of the stage scene video signal or the composite output STG or CO. , Extending substantially across the width of the stage scene video signal or composite output STG or CO.

  In the alternative, as shown in FIG. 24, processor 6 sub-samples subsamples from text video signal TD1 from a text document (eg, from a text editor, word processor, spreadsheet, presentation, or other document that renders text). It can be sampled or the subsample can be provided to the processor 6. The processor 6 then converts at least one of the two or more sub-scene video signals into a text video signal TD1 or equivalent TD1. R by replacing the text video signal TD1 or a rendered or reduced version TD1. R can be shifted to stage scene video signals STG, CO.

  Optionally, the processor 6 determines one of the two sub-scene video signals based on one or more retention criteria (eg, visual motion, sensed motion, acoustic detection of speech, time since last speech, etc.). The above can be set as protected sub-scene video signals SS1, SS2,. In this case, the processor 6 replaces at least one of the two or more sub-scene video signals SS1, SS2... SSn, but in particular sub-scene video signals SS1, SS2. By transitioning, one or more additional sub-scene video signals SS1, SS2,... SSn may be transitioned to stage scene video signals.

  Instead, the processor 6 uses a sub-scene enhancement operation (eg, blinking, high) based on one or more enhancement criteria (eg, visual action, detected action, acoustic detection of the utterance, time since last utterance, etc.). Light display, outline display, icon overlay, etc.) can be set. In this case, one or more sub-scene video signals are enhanced based on the corresponding enhancement criteria according to the sub-scene enhancement operation.

  In an additional variation, the processor 6 may detect a reference sensed from the sensor (e.g., sound waves, vibrations, electromagnetic radiation, heat, UV radiation, wireless, microscopic, detected by sensors such as RF elements, passive infrared elements or distance recognition elements). Sub-scene participant notification behavior may be set based on wave, electrical characteristics, or depth / range detection). The processor 6 may activate one or more local reminder indicators according to the notification action based on the corresponding detected criteria.

Examples of target orientations For example, target orientations are, for example, speaking participants M1, M2... Mn, eg, beamforming, localization, or comparative received signal strength, or comparatively using at least two microphones. One or more audio signals or their orientations corresponding to detection, such as participants M1, M2,... Mn, angle-recognized, vectorized, or identified by microphone array 4 according to time of flight Also good. Thresholding or frequency domain analysis may be used to determine whether the speech signal is strong enough or clear enough, at least to throw away mismatched pairs, multipaths, and / or redundancy Filtering may be performed using three microphones. The three microphones have the advantage of forming three pairs for comparison.

  As another example, alternatively or in addition, the orientation of the object may be acted upon by features, images, patterns, classes, and / or motion detection circuitry or executable code capable of scanning an image or video or RGBD from the camera 2. There may be those orientations that are detected, angle-recognized, vectorized, or identified in the scene.

  As another example, alternatively or additionally, the target orientation is the angle at which the face structure is detected in the scene by a face detection circuit or executable code that can scan an image or video or RGBD signal from the camera 2. These orientations may be recognized, vectorized, or identified. Skeletal structures can also be detected in this way.

  As another example, alternatively or additionally, the target orientation may be edge detection, corner detection, blob detection or segmentation, extreme value detection, and / or features that can scan an image or video or RGBD signal from camera 2 With those orientations in which a substantially continuous structure of color, texture, and / or pattern is detected, angle-recognized, vectorized, or identified by a detection circuit or executable code There may be. Recognition may refer to previously recorded, learned, or trained image patches, colors, textures, or patterns.

  As another example, alternatively or additionally, the orientation of the subject may differ from a known environment by a difference that can scan an image or video or RGBD signal from the camera 2 and / or a change detection circuit or executable code. There may be those orientations that are detected, angle-recognized, vectorized, or identified in the scene. For example, the device 100 maintains one or more visual maps of an empty conference room in which the device is located, and sufficiently obstructing entities such as people are blocking known features or areas in the map. Can be detected.

  As another example, alternatively or additionally, the target orientation may be “white” by a feature, image, pattern, class, and / or motion detection circuit or executable code capable of scanning an image or animated video or RGBD from camera 2. Regular shapes such as rectangles including “board” shapes, door shapes, or chair back shapes may be identified, angle recognized, vectorized, or their orientation identified.

  As another example, alternatively or additionally, the orientation of the object is detected by an active or passive acoustic emitter or transducer, and / or a passive or active optical or visual fiducial marker, and / or RFID or other electromagnetically detected Reference objects or features recognizable as artificial landmarks, including possible ones, may be those orientations placed by a person using device 100, which are angle-recognized by one or more of the techniques described above, and vectors Or identified.

  If the original or new target orientation is not obtained in this way (eg because no participant M1, M2 ... Mn has yet spoken), the default view is output as a single camera scene instead of the composite scene. Can be set as follows. For example, as one default view, the entire panoramic scene (eg, from 2: 1 to 10: 1 H: V horizontal-vertical ratio) can be fragmented and arranged into a single camera ratio that is output (eg, generally Landscape landscape orientations range from 1.25: 1 to 2.4: 1 or 2.5: 1 H: V aspect ratio or horizontal-vertical ratio, although corresponding "reverse" portrait orientation ratios are possible ). As another example, a default view in front of the target orientation is initially obtained and the “window” corresponding to the output scene ratio is tracked at a fixed rate, eg, across the scene SC, eg as a simulation of a slowly panning camera. Can be done. As another example, the default view may consist of a “face photo” (including an additional width of 5-20% within the margin) of each meeting attendee M1, M2,. Adjusted to optimize display area.

Aspect Ratio Examples Although embodiments and aspects of the invention may be useful in any angular range or aspect ratio, the benefits are arbitrarily increased because the sub-scene has an aspect ratio of substantially 2.4: 1 or higher. Formed from a camera that provides a panoramic video signal having an aspect ratio (representing either frame or pixel dimensions) and as seen on most laptops or television displays (usually 1.78: 1 or less) Synthesized into a multi-participant stage video signal having an overall aspect ratio of substantially 2: 1 or less (eg 16: 9, 16:10 or 4: 3, etc.), and optionally a stage video signal sub-scene Fills more than 80% of the entire recorded frame and / or stage video signal subscene and performance This is a case where any of the additionally synthesized thumbnails formed with the Norama video signal fills an area exceeding 90% of the total synthesized frame. Thus, each participant shown fills the screen as much as practically possible.

  The corresponding ratio between the vertical and horizontal angles of the view can be determined as the ratio from α = 2 arctangent (d / 2f), where d is the vertical or horizontal dimension of the sensor and f is the effective lens. The focal length. Different wide-angle cameras for meetings can have a 90, 120, or 180 degree field of view from a single lens, but each camera has a 1080p image (eg, 1920x1080 image) or aspect ratio with an aspect ratio of 1.78: 1 A much wider image with 3.5: 1 or other aspect ratio can be output. When observing a conference scene, a smaller aspect ratio (eg 2: 1 or less) combined with a 120 or 180 degree wide camera may indicate more ceilings, walls, or tables than may be desired. Accordingly, the aspect ratio of the scene or panoramic video signal SC and the angle of view FOV of the camera 100 may be independent, but a wider camera 100 (90 degrees or more) can have a wider aspect ratio (for example, 2.4: 1). In this embodiment, it is arbitrarily advantageous that the maximum wide camera (for example, 360 ° panoramic view) matches the widest aspect ratio (for example, 8: 1 or more). is there.

Example of Sub-Scene or Orientation Tracking The process performed by the device of FIGS. 1A and 1B, as shown in FIGS. 12-18, in particular FIGS. 16-18, is subject to orientation B1, B2 in the wide video signal SC. ... may include tracking sub-scenes FW, SS in Bn. As shown in FIG. 16, the processor 6 operatively connected to the acoustic sensor or microphone array 4 (with any beam forming circuit) and the wide cameras 2, 3, 5 is optionally or preferably in step S202. A substantially common angular range that is substantially greater than 90 degrees is monitored.

  In step S204 and step S206, the processor 6 performs acoustic recognition (eg, frequency, pattern, or other speech recognition) or visual recognition (eg, motion detection, face detection, skeletal detection, Identify first object orientations B1, B2,... Bn along one or both localizations (eg, color blob segmentation or detection) (eg, a measurement in Cartesian or polar coordinates or representing a position in a direction) Code may be executed or may include or be operatively connected to the identifying circuit. As in step S10 and in steps S12 and S14, the sub-scene video signal SS is subsampled from the wide cameras 2, 3, 5 along the target orientations B1, B2,... Bn identified in step S14. (For example, it is newly sampled from the image sensors of the wide cameras 2, 3, 5 or sub-sampled from the panoramic scene SC captured in step S12). The width of the sub-scene video signal SS (for example, the minimum width Min.1, Min.2... Min.n, or the sub-scene display width DWid.1, DWid.2... / Can be set by the processor 6 according to one or both signal characteristics of visual recognition. The signal characteristics can represent various acoustic or visual recognition quality or confidence levels. As used herein, “acoustic recognition” may include any recognition based on sound waves or vibrations (eg, meet measurement thresholds, match descriptors, etc.), including frequency analysis of waveforms such as Doppler analysis. In contrast, "visual recognition" corresponds to electromagnetic radiation, such as heat or UV radiation, radio or microwave, electrical property recognition or depth / range detected by sensors such as RF elements, passive infrared elements or distance recognition elements. Any recognition (eg, meeting a measurement threshold, matching a descriptor, etc.) may be included.

  For example, the target orientations B1, B2,... Bn identified in step S14 can be determined by a combination of such acoustic recognition and visual recognition in different orders, some of which are modes in FIGS. 1, 2 or 3 (which can be rationally and logically combined with each other). For example, as in step S220 of FIG. 18, in one order, the orientation of acoustic recognition is first recorded (but this order can be repeated and / or changed). Optionally, such orientations B1, B2,... Bn can be an angle, an angle with tolerance, or an approximate or angular range orientation (such as orientation B5 in FIG. 7A). As shown in steps S228 to S232 of FIG. 18, the recorded sound recognition orientation is determined by visual recognition (if the sufficiently reliable visual recognition is substantially within the recorded sound recognition threshold angle range). It can be refined (narrowed or reevaluated) based on eg face recognition. Any acoustic recognition that is not associated with visual recognition, in the same mode, or combined with another mode, eg, as in step S218 of FIG. 17, may remain candidate object orientations B1, B2,.

  Optionally, as in step S210 of FIG. 16, the signal characteristic represents a confidence level of one or both of acoustic recognition and visual recognition. A “confidence level” need not meet the formal probabilistic definition, but any comparison that establishes some degree of confidence (eg, signal quality, signal / noise ratio or equivalent, or success criteria above threshold amplitude) Can mean measurement. Alternatively or additionally, as in step S210 of FIG. 16, the signal characteristics may be one or both of acoustic recognition (eg, the angular range in which sound can occur) or visual recognition (eg, interpupillary distance, face width, body width). It can represent the width of the feature recognized within. For example, the signal characteristics may correspond to the approximate width of a human face recognized along the target orientations B1, B2,... Bn (eg, determined by visual recognition). The widths of the first sub-scene video signals SS1, SS2,... SSn can be set according to the visual recognition signal characteristics.

  For example, as in step S228 of FIG. 18, in some cases, when the width is not set according to the signal characteristics of visual recognition (for example, cannot be reliably set when the width defining feature cannot be recognized), step S230 of FIG. As such, a predetermined width may be set along with the localization of acoustic recognition detected within the angular range. For example, as in steps S228 and S232 of FIG. 18, when a face cannot be recognized by image analysis along the target orientations B1, B2,... Bn evaluated as having an acoustic signal indicating human speech, Along with the acoustic direction for defining the sub-scene SS, the default width (for example, a sub-scene having a width equivalent to 1/10 to 1/4 of the entire width of the scene SC) is maintained or is maintained, for example, as in step S230. Can be set. For example, FIG. 7A shows an attendee and speaker scenario where attendee M5's face is facing attendee M4 and M5 is speaking. In this case, the acoustic microphone array 4 of the conference camera 100 may be able to localize the speaker M5 along the target direction B5 (where the target direction B5 is drawn as a range of directions instead of a vector), but the wide camera Image analysis of the panoramic scene SC of 2, 3, 5 video signals may not be able to resolve the face or other visual perception. In such a case, the default width Min. 5 may be set as the minimum width to initially define, limit, or render the sub-scene SS5 along the subject orientation B5.

  In another embodiment, the target orientations B1, B2,... Bn may be directed and identified towards acoustic recognition detected within the conference camera 100 angular range. In this case, the processor 6 is arbitrarily close to the acoustic recognition as in step S209 of FIG. 16 (for example, in the target azimuth B1, B2,... Bn, overlapping with the azimuth or next to the azimuth. For example, visual recognition (within 5-20 degrees of the arc of the object orientation B1, B2,... Bn) may be identified. In this case, the width of the first sub-scene video signal SS1, SS2,... SSn was close (or is) to acoustic recognition or otherwise matched (or is) acoustic recognition. It can be set according to the signal characteristics of visual recognition. This can happen, for example, when the target orientations B1, B2,... Bn are first identified by the acoustic microphone array 4 and then close enough or otherwise matched using the video image from the wide camera 100. It is when the validity is verified or confirmed by recognition.

  In one variation, as described with reference to FIGS. 17 and 16, a system that includes a conference or wide camera 100 uses spatial or visual recognition to generate a spatial map as in step S218 of FIG. Create and then rely on this spatial map, as in step S209 of FIG. 16, to match the validity of later associated, matched, close, or “snapped” recognition Or by different or other recognition approaches. For example, in some cases, the entire panoramic scene SC may be too large to be effectively scanned frame by frame for face recognition and the like. In this case, people do not move significantly in a meeting situation using the camera 100, especially after sitting in their seat for a meeting, so that only a part of the entire panoramic scene SC is, for example, every video frame. Can be scanned.

  For example, as shown in step S212 in FIG. 17, in order to track the sub-scenes SS1, SS2,... SSn in the target orientations B1, B2,. The sub-sampling window can be scanned through the animated video signal SC corresponding to 100 views. The processor 6 or the circuit associated therewith performs sub-sampling by substantially satisfying a threshold value for defining a suitable signal quality for the candidate target orientations B1, B2,... Bn, for example as in step S214 of FIG. Candidate target orientations B1, B2,... Bn in the window can be identified. Each target orientation B1, B2,... Bn may correspond to localization of visual recognition detected in the sub-sampling window, for example, as in step S216 of FIG. As in step S218 of FIG. 17, the candidate orientations B1, B2,... Bn can be recorded in a spatial map (eg, a memory or database structure that keeps track of the position, location, and / or orientation of the candidate orientations). For example, in this manner, face recognition or other visual recognition (eg, motion) can be stored in the spatial map even if acoustic detection has not yet occurred in that orientation. Thereafter, the angular range of the wide camera 100 can be monitored by the processor 6 using an acoustic sensor or microphone array 4 for acoustic recognition (this is used to verify the validity of the candidate target orientations B1, B2,... Bn. Can be).

  For example, referring to FIG. 7A, the processor 6 of the conference camera 100 may scan different subsampled windows of the entire panoramic scene SC for visual recognition (eg, face, color, motion, etc.). Depending on the lighting, motion, face orientation, etc., in FIG. 7, the potential object orientations corresponding to the faces, motions or similar detections of attendees M1... M5 may be stored in the spatial map. However, in the scenario shown in FIG. A potential target orientation on one side may not be validated later by the acoustic signal if it corresponds to an attendee who is not speaking (and this attendee is not captured at all in the sub-scene, Can only be captured within a panoramic scene). When attendees M1 ... M5 speak or begin to speak, the validity of potential subject orientations including or attending these attendees can be verified and recorded as subject orientations B1, B2 ... B5 .

  Optionally, as in step S209 in FIG. 16, close to one candidate orientation recorded in the spatial map (substantially adjacent, next to, or within a +/− 5 to 20 degree arc). When acoustic recognition is detected, the processor 6 may snap the target orientations B1, B2,... Bn to substantially correspond to the one candidate orientation. Step S209 of FIG. 16 indicates that the target orientation is consistent with the corresponding spatial map, and “match” may include associating, replacing, or changing the target orientation value. For example, face recognition or motion recognition in the window and / or panoramic scene SC may have better resolution than the acoustic array or microphone array 4, but detection due to acoustic recognition due to less frequent or reliable detection. The target orientations B1, B2,... Bn made can be changed according to visual recognition, recorded, or otherwise corrected or adjusted. In this case, instead of sub-sampling the sub-scene video signals SS1, SS2... SSn along the obvious object orientations B1, B2. Sub-sample the sub-scene video signal from the wide camera 100 and / or the panoramic scene SC along the target direction B1, B2,... Bn following the snap operation after the acoustic target directions B1, B2,. obtain. In this case, as in step S210 of FIG. 16, the width of the sub-scene video signal SS is set according to the detected face width or motion width, or alternatively, the signal characteristics of acoustic recognition (eg, default width, resolution of the array 4). , Confidence level, width of features recognized within one or both of acoustic recognition and / or visual recognition, approximate width of human face recognized along the subject orientation). As in step S210 of FIG. 16 or step S230 of FIG. 18, when the sub-scene SS width is not set according to the visual recognition signal characteristics such as the face width or the motion range, a predetermined width (for example, as shown in FIG. 7A). Default width Min.5 etc.) can be set according to the acoustic recognition.

  In the example of FIG. 18, the conference camera 100 and the processor 6 record a sub-scene in the target orientations B1, B2,... Bn by recording a moving image video signal substantially corresponding to the field of view FOV of the wide camera 100 of 90 degrees or more. Can be tracked. In step S220, the processor can monitor the angular range corresponding to the field of view FOV of the wide camera 100 using the acoustic sensor array 4 for acoustic recognition. If the range of acoustic recognition is detected in step S222, the processor In S224, the target orientations B1, B2,... Bn directed toward the acoustic recognition detected within the angular range can be identified. The processor 6 and associated circuitry then in step S226, in the video video signal of the panoramic scene SC, next according to the corresponding range of the target orientations B1, B2,... Bn (eg similar to the range of the target orientation B5 in FIG. 7A). A sub-sampling window can be positioned. The processor may then localize the detected visual recognition within the sub-sampling window when visual recognition is detected within the range as in step S228. The processor 6 may then subsample the captured sub-scene video signal SS from the wide camera 100 (directly from the camera 100 or from the panoramic scene recording SC), optionally substantially centered on visual recognition. As in step S232, the processor 6 may then set the width of the sub-scene video signal SS according to the visual recognition signal characteristics. If visual recognition is not possible, not preferred, not detected, or not selected, as in step S228 of FIG. 18, the processor 6 may maintain or select the acoustic minimum width, as in step S230 of FIG.

  Instead, as shown in FIGS. 16 to 18, the conference camera 100 and the processor 6, for example, in step S <b> 212 of FIG. 17, the wide camera 2 that observes the visual field substantially 90 degrees or more with the acoustic sensor array 4. 3 and 5 can be used to monitor a sub-scene in a target orientation B1, B2,... Bn in a wide video signal such as a panoramic scene SC. The processor 6 may identify a plurality of target orientations B1, B2,... Bn, each directed towards localization within the angular range (acoustic or visual or sensor-based as in step S216) As the orientations B1, B2,... Bn, corresponding recognition, corresponding localization, or data representing them are sequentially stored as in step S218 of FIG. 17, the recorded characteristics corresponding to the target orientations B1, B2,. Maintain a spatial map of Thereafter, as in step S210 of FIG. 16, for example, the processor 6 subsamples the sub-scene video signals SS1, SS2,... SSn from the wide camera 100 substantially along at least one target orientation B1, B2,. , The width of the sub-scene video signals SS1, SS2,... SSn may be set according to the recorded characteristics corresponding to at least one target orientation B1, B2,.

Predictive tracking examples In the above description of structures, devices, methods and techniques for identifying new target orientations, various detection, recognition, triggering, or other causes to identify such new target orientations Is explained. The following discussion describes updating, tracking, or predicting changes in object orientation and sub-scene orientation, direction, location, pose, width, or other characteristics, but this update, tracking, and prediction are described above. Can also be true. The description of the method for identifying new object orientations and updating or predicting orientation or subscene changes is relevant in that re-acquisition of the object orientation or subscene is facilitated by tracking or prediction. The methods and techniques described herein are used to scan, identify, update, track, record, or reacquire orientation and / or subscenes in steps S20, S32, S54, or S56 And vice versa.

  For example, prediction HEVC, H.I. H.264, MPEG-4, other MPEG I slices, P slices, and B slices (or frames or macroblocks); other intra-frame and interframe, photos, macroblocks or slices; H.264 or other SI frame / slice, SP frame / slice (switching P), and / or multi-frame motion prediction; VP9 or VP10 superblock, block, macroblock or superframe, intraframe and interframe prediction, component prediction, Predictive video data may be recorded for each sub-scene, such as data that is encoded according to or related to motion compensation, motion vector prediction, and / or segmentation.

  For example, motion vectors obtained from speech motion on a microphone array, or direct or pixel based methods (eg block matching, phase correlation, frequency domain correlation, pixel recursion, optical flow) and / or indirect or feature based Other methods such as those described above independent of the video standard or motion compensation SPI, such as motion vectors obtained from this method (feature detection such as corner detection using statistical functions such as RANSAC applied on sub-scenes or scene regions) Prediction or tracking data can be recorded.

  Additionally or alternatively, per-scene updates or tracking can be performed, for example, amplitude, utterance frequency, utterance length, associated attendees M1, M2... Mn (two subscenes with mutual traffic), moderator Participants in the role or coordinator Lead (sub-scenes with regular short speech), recognized signal phase (eg applause, “keep me pointing at the camera” and other speech and speech recognition obtained speech parameters, etc.) Related indicators or data or information representing them may be recorded, identified or scored, these parameters or indicators may be recorded independently of the tracking step or at different times during the tracking step. Also, sub-scene tracking can be performed, for example, with audio representing coughing or sneezing; periodic or periodic motion or video representing machine, wind, or blinking; at high enough frequency to be transient or transient Errors or irrelevant indicators, such as actions, may be recorded, identified, or scored.

  Additionally or alternatively, sub-scene updates or tracking may be, for example, retention criteria (eg, voice / speech time, speech / speech frequency, time since last speech, tagged for retention) Based on the above, an indicator or data or information representing it may be recorded, identified, or scored for setting up a sub-scene and / or protecting the sub-scene from removal. In subsequent processing for compositing, removing a sub-scene other than a new or subsequent sub-scene does not remove the protected sub-scene from the composite scene. In other words, the protection sub-scene has a low priority to be removed from the composite scene.

  In addition, or alternatively, sub-scene updates or tracking may be an indicator for setting additional criteria or data or information representing it (eg, speech time, speech frequency, audible frequency cough / sneeze / doorway Bell, sound amplitude, utterance angle and face recognition match) may be recorded, identified, or scored. In the process for compilation, only subsequent sub-scenes that meet the additional criteria are combined into the composite scene.

  Additionally or alternatively, sub-scene updates or tracking can be done with emphasis criteria (eg, repeated speakers, designated presenters, most recent speakers, loudest speakers, objects that rotate in hand / scene changes) Based on motion detection, high-frequency scene activity in the frequency domain, raised hand motion or skeleton recognition), for example, an index for setting sub-scene enhancement motion such as speech, CGI, image, video, or composite effect, or Data or information representing it can be recorded, identified, or scored (e.g., scaling one sub-scene to a larger scale, blinking or pulsing one sub-scene boundary, new with genie effects Insert a sub-scene (increase from small to large), Enhancing or inserting sub-scenes with ungs effects, arranging one or more sub-scenes with card sorting or shuffle effects, ordering sub-scenes with overlap effects, sub-scenes with the appearance of a “folded” graphic corner Cornering). In the compilation process, at least one of the individual sub-scenes is enhanced according to the sub-scene enhancement operation based on their own or corresponding enhancement criteria.

  Additionally or alternatively, sub-scene updates or tracking may be used to set sub-scene participant notifications or reminder actions based on sensors or sensed criteria (eg, too quiet, remote poke from social media) Can be recorded, identified, or scored (eg, lights on device 100 for attendees M1, M2,... Mn, optionally on the same side as the sub-scene) To blink). In a compilation process or other process, a local reminder indicator is activated according to a notification or reminder action based on its own or corresponding detected criteria.

  Additionally or alternatively, sub-scene-by-sub-scene updates or tracking can be performed, for example, for each recognized or localized recorded characteristic (eg, color blob, face, voice as described herein with respect to steps S14 or S20). For predicting or setting a change vector for each angular sector FW1, FW2... FWn or SW1, SW2... SWn based on a change in speed or direction and / or for each angular sector based on the prediction or setting Indicators for updating the direction of FW1, FW2... FWn or SW1, SW2... SWn or data or information representing it may be recorded, identified or scored.

  Additionally or alternatively, sub-scene updates or tracking can be based on, for example, lost recognition or locality based on the most recent location of each recognition or localization recorded characteristic (eg, color blob, face, speech). Index or data or information for predicting or setting a search area for re-acquisition or re-acquisition and / or for updating the direction of each angular sector based on the prediction or setting May be recorded, identified, or scored. The recorded property may be at least one color blob, segmentation, or blob object representing skin and / or clothing.

  Additionally or alternatively, sub-scene-by-sub-scene updating or tracking may be performed on recorded characteristics (eg, orientations B1, B2,... Bn or angle from origin OR in scene SC, and angular sectors FW, SW in scene SC. A Cartesian map (based on an angular range such as sub-scene SS1, SS2... SSn, etc.) or in particular or optionally a polar map, each of the recorded characteristics can be recorded in the recorded characteristics orientation B1, B2. At least one parameter representing.

  Thus, alternatively or additionally, embodiments of executable code stored and executed in device 100, its circuitry, and / or ROM / RAM 8 and / or CPU / GPU 6 may be configured to target angle ranges (eg, scene SC). The horizontal range of the camera 2n, 3n, 5 or 7 to be formed, or a subset thereof) is monitored using the acoustic sensor array 4 and the light sensor arrays 2, 3, 5 and / or 7 in the wide-angle scene SC. The target sub-scenes SS1, SS2,... SSn corresponding to the width FW and / or SW may be tracked. Device 100, its circuitry, and / or its executable code are described herein with respect to, for example, step S14 (identification of a new target orientation) and / or step S20 (tracking and characteristic information for orientation / subscene) of FIG. As described in the document, the target angular range SC may be scanned for recognition criteria (eg, sound, face). Device 100, its circuitry, and / or its executable code may be subject to a first recognition (eg, detection, identification, triggering, or) by at least one of acoustic sensor array 4 and photosensor arrays 2, 3, 5, and / or 7. The first target orientation B1 may be identified based on other causes) and localization (eg, angle, vector, pose, or location). The device 100, its circuitry, and / or its executable code is subject to a second recognition and localization (and optionally a third) by at least one of the acoustic sensor array 4 and the photosensor arrays 2, 3, 5, and / or 7. And subsequent recognition and localization) may identify the second target orientation B2 (and optionally the third and subsequent target orientations B3 ... Bn).

  Device 100, its circuitry, and / or its executable code recognizes at least one angular sub-scene (eg, an initial small angular range or a face-based sub-scene FW) that includes its target orientation B1, B2,... Bn. Reference (for example, the set or reset angle span is greater than, twice, or more than the interpupillary distance; the set or reset angle span is the head-to-wall contrast, distance, edge By enlarging, widening, setting or resetting until a threshold based on (or wider than the difference or motion transition) (e.g., a width threshold as described with reference to steps S16 to S18 in FIG. 13) is satisfied, Each angle sector (eg FW, SW or others) for each target orientation B1, B2,... Bn may be set.

  The device 100, its circuitry, and / or its executable code may be the direction of recorded characteristics (eg, color blob, face, voice) within each recognition and / or localization or representing each recognition and / or localization, or Update or track their respective angular sectors FW1, FW2 ... FWn and / or SW1, SW2 ... SWn directions or orientations B1, B2 ... Bn based on changes in orientations B1, B2 ... Bn (these terms are described herein) Used interchangeably). Optionally, as described herein, device 100, its circuitry, and / or its executable code may update or track its respective angular sectors FW1, FW2 ... FWn and / or SW1, SW2 ... SWn. , First, second, and / or third, and / or subsequent angular changes in the target orientations B1, B2,... Bn.

Composite output example (for video conferencing)
In FIGS. 8A-8D, 10A-10B, and 19-24, the “composite output CO”, ie, the combined or synthesized sub as a combined rendered / composed camera view. The scene is shown with a lead to both the main view of the remote display RD1 (representing the scene received from the conference room local display LD) and the network interface 10 or 10a, and the TV of the conference room (local) display LD. The conference client treats the video signal received from the USB peripheral device 100 “transparently” as a single camera view and represents the composite output CO to the remote client or remote display RD1 and RD2. It should be noted that all thumbnail views can also show the composite output CO. In general, FIGS. 19, 20, and 22 correspond to the attendee arrangement shown in FIGS. 3A-5B, and another attendee sits in the empty seat shown in FIGS. 3A-5B in FIG. I'm joining.

  Between example transitions, the reduced panoramic video signal SC. R (accounting for about 25% of the vertical screen) may indicate a “zoomed in” portion of the panoramic scene video signal SC (eg, as shown in FIGS. 9A-9E). The zoom level can be determined by the number of pixels contained in this approximately 25%. When the persons / objects M1, M2,... Mn become related, the corresponding sub-scene SS1, SS2,... SSn transitions to the stage scene STG or the composite output CO (for example, by synthesizing a sliding video panel) and participates. Its clockwise or left-to-right position between M1, M2 ... Mn is maintained. At the same time, the processor uses the GPU 6 memory or the ROM / RAM 8 to reduce the panorama video signal SC.1 in order to display the current target orientations B1, B2,. R can slowly scroll left or right. The current target orientation can be highlighted. Once a new associated sub-scene SS1, SS2,... SSn is identified, the reduced panoramic video signal SC. R is a panorama video signal SC.2 in which the latest sub-scenes SS1, SS2,. It can be rotated or panned so that it is centered in R. With this configuration, a reduced panoramic video signal SC. R is continuously re-rendered and substantially panned to indicate the relevant part of the room.

  As shown in FIG. 19, in a typical videoconference display, each attendee's display shows a master view and multiple thumbnail views, each substantially determined by the output signal of the webcam. The master view is typically one of the remote attendees, and the thumbnail view represents the other attendees. Depending on whether it is a video conferencing or a chat system, the master view can be selected to show active speakers among the attendees, or often select a local scene in some cases by selecting a thumbnail. Can be switched to another attendee. In some systems, the local scene thumbnails always remain in the overall display so that each attendee can position themselves relative to the camera to present a useful scene (an example of this is shown in FIG. 19). ).

  As shown in FIG. 19, an embodiment according to the present invention provides a composite stage view of multiple attendees instead of a single camera scene. For example, in FIG. 19, the conference camera 100 can utilize potential target orientations B1, B2, and B3 to attendees M1, M2, and M2 (represented by icon diagrams M1, M2, and M3). As described herein, there are three possible attendees M1, M2, M3 that are localized or otherwise identified, and one SPKR is speaking, so stage STG ( The composite output CO) can initially be populated with a default number (in this case two) of related sub-scenes, including the active speaker SPKR sub-scene, which is attendee M2 in FIG.

  FIG. 19 shows the display of three participants, that is, a local display LD such as a personal computer connected to the conference camera 100 and to the Internet INET, and a first remote attendant A.E. hex's first personal computer ("PC") or tablet display remote display RD1 and a second remote attendant A.H. A diamond second PC or tablet display RD2 is shown. As expected in the context of video conferencing, the local display LD shows the most prominent remote speakers selected by the operator of the local display PC or video conferencing software (A. hex in FIG. 19), whereas 2 One remote display RD1, RD2 shows a view (eg, active speaker view, composite view CO of conference camera 100) selected by the remote operator or software.

  The arrangement of attendees in the master view and thumbnail view depends to some extent on user selection and even automatic selection in the video conference or video chat system, but in the example of FIG. 19, the local display LD seems to be typical. With a master view showing the last selected remote attendee (eg A. hex who is working on a PC or laptop with remote display RD1) and essentially all attendance A thumbnail row (including a synthesized stage view from the local conference camera 100) is shown. Each of the remote displays RD1 and RD2, in contrast, shows a master view including the synthesized stage view CO, STG (since speaker SPKR is currently speaking), and the thumbnail column is again the remaining attendees Includes views.

  FIG. 19 shows that attendee M3 has already spoken or has been previously selected as the default occupant of stage STG and already occupies the most relevant sub-scene (eg, the most recently associated sub-scene It was assumed). As shown in FIG. 19, sub-scene SS1 corresponding to speaker M2 (icon diagram M2 and silhouette M2 with an open mouth on remote display 2) is in a single camera view with a slide transition (represented by a block arrow). Synthesized. The preferred slide transition starts with zero or negligible width, and the middle, ie, the target orientations B1, B2 ... Bn of the corresponding sub-scene SS1, SS2 ... SSn slide on the stage and then synthesized The width of the corresponding sub-scene SS1, SS2,... SSn can be grown at least until it reaches a minimum width, and the width of the corresponding sub-scene SS1, SS2. Since the composite (intermediate transition) and composite scene are provided to the video conference client of the conference room (local) display LD as a camera view, the composite and composite scene is the main of the local client display LD and the two remote client displays RD1, RD2. It can be presented substantially simultaneously in the view and thumbnail view (ie, presented as the current view).

  In FIG. 20, after FIG. 19, attendee M1 becomes the nearest and / or most relevant speaker (eg, the previous situation is that speaker M2 is the nearest and / or most relevant speaker. This is the situation in FIG. Sub-scenes SS3 and SS2 for attendees M3 and M2 continue to be related according to the tracking and identification criteria (by scaling or cropping, the 2-12 times width limit and as described herein) (Optionally limited by other methods) can be reconfigured to a smaller width if desired. Sub-scene SS2 is similarly composed on a compatible size and is then configured on stage STG with a slide transition (again represented by a block arrow). As described herein with respect to FIGS. 9, 10A-10B, and 11A-11B, the new speaker SPKR is in the orientation of attendee M2 already displayed (clockwise in the down view). A) Since the attendee M1 is on the right side, arbitrarily place the sub-scene SS1 on the stage so as to preserve the left-right image or left-to-right order (M3, M2, M1), in this case the transition from the right You may move to.

  In FIG. 21, after FIG. 20, the new attendee M4 arriving in the room becomes the most recently associated speaker. Sub-scenes SS2 and SS1 for speakers M2 and M1 continue to be related according to the tracking and identification criteria and remain synthesized in a “3 to 1” width. The sub-scene corresponding to speaker M3 “ages out” and is no longer as relevant as the most recent speaker (but many other priorities and relevance are described herein). Sub-scene SS4 corresponding to speaker M4 is composited to a compatible size and then composited to the camera output with flip transitions (again represented by block arrows), and sub-scene SS3 is flipped out as a removal. This may be a slide transition or an alternative transition. Although not shown, as an alternative, the new speaker SPKR is an attendee M4 on the left (clockwise in the view down) of the attendees M2 and M1 already displayed, so any sub-scene SS4 may be shifted onto the stage so as to preserve the left-right image or left-to-right order (M4, M2, M1), in this case the shift from the left. In this case, each of the sub-scenes SS2 and SS1 may move one place to the right, and the sub-scene M3 may go to the right of the stage (so as to be separated by slide transition).

  As described herein, FIGS. 19-21 are illustrated on a mobile device as an example, where a composite, tracked, and / or displayed composite scene is received and displayed as a single camera scene. Fig. 6 illustrates exemplary local and remote video conferencing modes. These are referenced and described in the context of the previous paragraph.

  Although the overall information is the same, FIG. 22 presents a form for displaying a video conference that is a variation of the form of FIG. In particular, in FIG. 19, the thumbnail view does not overlap the master view, and the thumbnail view that matches the master view is held in the thumbnail column, but in the form of FIG. 22, the thumbnail overlaps the master view. (Eg, synthesized to be superimposed on the master view) and the current master view is not highlighted in the thumbnail column (eg, by dimming).

  FIG. 23 shows that a fourth client corresponding to a high resolution, close-up or similar separate camera 7 connects his client to the video conference group via the network interface 10b. FIG. 23 shows a variant of FIGS. 19 to 22 in which the composite output CO and its transition are presented to the conference room (local) display LD via the network interface 10a.

  FIG. 24 shows a variation of FIGS. 19-22 in which a client reviewing code or a document with a text review window connects to the conference camera 100 via a local wireless connection (although in some variations the code Or the client reviewing the document may connect via the Internet from a remote station). In one example, a first device or client (PC or tablet) runs a video conferencing or chat client that shows attendees in a panoramic view, and a second client or device (PC or tablet) runs a code or document review client Then, it is provided to the conference camera 100 as a video signal in the same form as the web camera. The conference camera 100 synthesizes the code or document review client document window / video signal as a full frame sub-scene SSn to the stage STG or CO and optionally further includes a local including conference attendees, eg, higher than the stage STG or CO. Combining panoramic scenes. Thus, instead of individual attendee sub-scenes, the text shown in the video signal is available to all attendees, but attendees may still be confirmed by referring to the panoramic view SC. . Although not shown, the conference camera 100 device may instead create, instantiate, or execute a second video conference client to host the document view. Instead, the high-resolution, close-up, or simply separate camera 7 connects its client to the video conference group via the network interface 10b, whereas the composite output CO and its transition is a network interface. It is presented on the conference room (local) display via 10a.

  In at least one embodiment, meeting attendees M1, M2,... Mn may always be indicated in the stage scene video signal or composite output STG, CO. For example, as shown in FIG. 25, based on at least face width detection, the processor 6 crops the faces as face-only sub-scenes SS1, SS2,... SSn, and places them on top of the stage scene video signal or composite output STG, CO or Can be aligned along the bottom. In this case, a participant using a device such as the remote device RD1 communicates with the local display LD by clicking or touching (in the case of a touch screen) the sub-scene SS1, SS2,. It may be desirable to be able to create a stage scene video signal STG concentrated on that person. In one exemplary solution, using a configuration directly connected to the Internet INET, similar to FIG. 1B, the conference camera 100 creates or instantiates an appropriate number of virtual video conferencing clients and / or virtual to each Cameras can be assigned.

  FIG. 26 shows some icons and symbols used throughout FIGS. In particular, an arrow extending from the center of the camera lens can correspond to the target orientations B1, B2,... Bn regardless of whether the arrow is labeled as such in the various figures. A dashed line extending from the camera lens at a “V” shaped angle may correspond to the field of view of the lens, regardless of whether the dashed line is labeled as such in the various figures. A depiction of a schematic “bar drawing” of a person with an oval head and a rectangular or trapezoidal body, regardless of whether this schematic person is so labeled in various figures Can respond to participants. This schematic person's open mouth depiction may depict the current speaker SPKR, regardless of whether this schematic person whose mouth is open is so labeled in the various figures. A thick arrow that extends from left to right, from right to left, from top to bottom, or in a spiral, indicates whether a transition in progress or whether the arrow is labeled as such in the various figures. A transitional synthesis may be shown.

  In this disclosure, “wide angle camera” and “wide scene” depend on the field of view and distance from the subject, and any camera with a field of view that is wide enough to capture two different people who are not shoulder-to-side in a meeting including.

  “Field of view” is the horizontal field of view of the camera unless a vertical field of view is specified. As used herein, a “scene” means an image (still image or moving image) of a scene captured by a camera. Generally, with exceptions, a panoramic “scene” SC is one of the largest images or video streams or signals handled by the system, regardless of whether the signal is captured by a single camera or stitched from multiple cameras. One. The most frequently mentioned scene “SC” referred to herein is a panorama captured by a conformal distribution of a camera coupled to a fisheye lens, a camera coupled to a panoramic optic, or an overlapping camera. The scene SC which is the scene SC is included. The panoramic optics may provide the camera with a panoramic scene substantially directly, and in the case of a fisheye lens, the panoramic scene SC is a long, high aspect ratio rectangle with the fisheye view surrounding or horizontal bands separated. It can be a horizontal band that is dewarped to the image, and in the case of overlapping cameras, panoramic scenes can be stitched and cropped (and possibly dewarped) from individual overlapping views. Good. “Sub-scene” means a sub-portion of a scene, for example, a continuous, usually rectangular pixel block that is smaller than the entire scene. Even if the panoramic scene is cropped to less than 360 degrees, it may be referred to as the entire scene SC in which sub-scenes are handled.

  As used herein, “aspect ratio” is described as an H: V horizontal: vertical ratio, with a “greater” aspect ratio increasing the horizontal ratio (widely short) relative to vertical. Aspect ratios greater than 1: 1 (eg, 1.1: 1, 2: 1, 10: 1) are considered “scenery formats”, and for purposes of this disclosure, an aspect ratio of 1: 1 or less “Format” (eg, 1: 1.1, 1: 2, 1: 3). Each “single camera” video signal is incorporated herein by reference in its entirety (ie http://www.usb.org/developers/docs/devclass_docs/USB_Video_Class_1_5.zip USB_Video_Class_1_1_090711 at the same URL). .zip), also known as “USB Device Class Definition for Video Devices” 1.1 or 1.5 by USB Implementers Forum, formatted as a video signal corresponding to a single camera, eg UVC. Any of the signals described in the UVC is a “single camera video signal” regardless of whether the signal is transported, transported, transmitted or tunneled via USB. possible.

  “Display” means any direct display screen or projection display. “Camera” means a digital imaging device, which may be a CCD or CMOS camera, a thermal imaging camera, or an RGBD depth or time of flight camera. The camera may be formed by two or more stitched camera views and / or a virtual camera with a wide aspect, panorama, wide angle, fisheye, or catadioptric perspective.

  A “participant” is a person, device, or location that is connected to a group videoconference session and displaying a view from a webcam, and in most cases the “participant” is not only a participant but also a conference You are in the same room as the camera 100. A “speaker” is an attendee who is speaking or an attendee who has spoken recently enough to identify the speaker, although the conference camera 100 or related remote server is speaking. Or a participant who has spoken recently enough to identify the speaker.

  “Combining” generally means digital composition as is known in the art, ie, digitally assembling multiple video signals (and / or images or other media objects) to create the final video signal. This includes alpha compositing and blending, anti-aliasing, node-based compositing, key framing, layer-based compositing, nesting compositing or compositing, deep image compositing (whether function based or sample based) , Opacity, and depth using deep data). Compositing is an ongoing process that includes motion and / or animation of sub-scenes, each containing a video stream, for example, each of the various frames, windows, and sub-scenes in the entire stage scene. As different stage scenes may display different ongoing video streams as they are moved, transitioned, blended, or otherwise synthesized. Composition as used herein may use a composition window manager having one or more off-screen buffers for one or more windows, or a stacking window manager. Any off-screen buffer or display memory content may be double or triple buffered or otherwise buffered. Compositing is further buffered, such as applying 2D and 3D animation effects, blending, fading, scaling, zooming, rotating, duplicating, bending, twisting, shuffling, blurring, drop shadow, glow, preview, and adding animation. Processing for one or both of a specific window and a display memory window. Compositing may further include applying them to vector-oriented graphical elements or pixel or voxel-oriented graphical elements. Compositing renders a pop-up preview when touched, mouse over, hover or click, window switching by rearranging several windows against the background to allow selection by touch, mouse over, hover or click , And flip switching, cover switching, ring switching, exposure switching, and the like. As described herein, various visual transitions such as fading, sliding, growing or shrinking, and combinations thereof can be used on the stage. As used herein, “transition” includes the necessary synthesis steps.

  The method or algorithm steps described in connection with the embodiments disclosed herein may be directly embodied in hardware, in software modules executed by a processor, or in a combination of the two. A software module may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, removable disk, CD-ROM, or other form of storage medium known in the art. . An exemplary storage medium may be coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and storage medium may reside in an ASIC. The ASIC may exist in the user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a user terminal.

  All of the processes described above may be embodied in and fully automated via software code modules executed by one or more general purpose or special purpose computers or processors. The code modules may be stored on any type of computer readable medium or other computer storage device or collection of storage devices. Alternatively, some or all of the methods may be embodied in specialized computer hardware.

  All of the methods and tasks described herein may be performed by a computer system and fully automated. Computer systems often include a plurality of individual computers or computing devices (eg, physical servers, workstations, storage arrays, etc.) that communicate over a network and interoperate to perform the functions described above. Also good. Each such computing device typically resides in a processor (or a plurality of processors or circuits or collections of circuits, eg, modules), or memory or other non-transitory computer-readable storage medium that executes program instructions. Contains stored modules. The various functions disclosed herein may be embodied in such program instructions, but some or all of the disclosed functions instead are application-specific circuits (eg, ASICs or FPGAs) for computer systems. May be realized. If a computer system includes multiple computing devices, these devices may be located at the same location, but need not be so. The results of the disclosed methods and tasks may be stored persistently by converting physical storage devices such as solid state memory chips and / or magnetic disks to different states.

Claims (40)

A method of synthesizing and outputting a video signal,
Recording a panoramic video signal having an aspect ratio of substantially 2.4: 1 or greater, captured from a wide camera having a horizontal field of view of substantially 90 degrees or greater;
Sub-sampling at least two sub-scene video signals from the wide camera in their respective orientations;
Combining the at least two sub-scene video signals side by side to form a stage scene video signal having an aspect ratio substantially equal to or less than 2: 1, wherein more than 80% of the area of the stage scene video signal A large area is subsampled from the panoramic video signal, and
Outputting the stage scene video signal formatted as a single camera video signal. Sub-sampling additional sub-scene video signals in their respective orientations from the panoramic video signal;
A stage scene having an aspect ratio of substantially less than or equal to 2: 1 comprising a plurality of side-by-side sub-scene video signals, wherein the at least two sub-scene video signals are combined with at least one of the additional sub-scene video signals The method of claim 1, further comprising forming a video signal. Combining at least two sub-scene video signals with at least one additional sub-scene video signal to form a stage scene video signal;
At least one of the additional sub-scene video signals is transferred to the stage scene video signal by replacing at least one of the at least two sub-scene video signals and has an aspect ratio substantially equal to or less than 2: 1. The method of claim 2, comprising forming a stage scene video signal.   A minimum width is assigned to each sub-scene video signal, and when each transition to the stage scene video signal is completed, each sub-scene video signal is combined side by side with substantially the minimum width or more and synthesized. The method of claim 3, wherein:   5. The method of claim 4, wherein the composite width of each sub-scene video signal in transition increases throughout the transition until the composite width is substantially greater than or equal to its corresponding minimum width.   Each sub-scene video signal is substantially greater than or equal to its minimum width, and each has its own width in which the sum of all synthesized sub-scene video signals is substantially equal to the width of the stage scene video signal, The method of claim 4, wherein the methods are synthesized side by side.   The width of the sub-scene video signal in the stage scene video signal is synthesized to change according to the activity criteria detected in at least one target orientation corresponding to the sub-scene video signal, whereas the stage scene video The method of claim 6, wherein the width of the signal is kept constant. Combining the at least two sub-scene video signals with at least one additional sub-scene video signal to form a stage scene video signal;
Reducing at least one width of the at least two sub-scene video signals by an amount corresponding to a width of at least one of the additional sub-scene video signals, thereby reducing at least one of the additional sub-scene video signals to the stage The method of claim 2 including transitioning to a scene video signal.   Each sub-scene video signal is assigned its own minimum width, and each sub-scene video signal is combined side by side substantially above its corresponding minimum width to form the stage scene video signal, with at least one Along with the additional sub-scene video signal, when the sum of the respective minimum widths of the at least two sub-scene video signals exceeds the width of the stage scene video signal, at least one of the at least two sub-scene video signals is The method of claim 8, wherein the transition is made to be removed from the stage scene video signal.   10. The at least one of the two sub-scene video signals that transition to be removed from the stage scene video signal corresponds to a respective target orientation for which an activity criterion was met most recently. Method.   The order from left to right for the wide camera between the respective orientations of the at least two sub-scene video signals and at least one of the additional sub-scene video signals is such that the at least two sub-scene video signals are at least one The method of claim 9, wherein the method is combined with the additional sub-scene video signal and stored when forming the stage scene video signal. Each target orientation from the panoramic video signal is selected depending on selection criteria detected in the target orientation relative to the wide camera, and
The method of claim 1, comprising transitioning the corresponding sub-scene video signal to be removed from the stage scene video signal after the selection criteria is no longer true. The selection criteria includes the presence of activity criteria satisfied in the respective target orientation, and
Counting the time since the activity criterion is satisfied in each of the target orientations, and each sub-scene signal is a predetermined period after the activity criterion is satisfied in the respective target orientations. The method of claim 12, transitioning to be removed from the stage scene video signal. Subsampling from the panoramic video signal a reduced panoramic video signal having an aspect ratio of substantially 8: 1 or higher;
A stage having an aspect ratio of substantially less than or equal to 2: 1 including a plurality of side-by-side sub-scene video signals and the panoramic video signal by combining the at least two sub-scene video signals with the reduced panoramic video signal The method of claim 1, further comprising forming a scene video signal.   Combining the at least two sub-scene video signals with the reduced panoramic video signal to include a plurality of side-by-side sub-scene video signals and the panoramic video signal higher than the plurality of side-by-side sub-scene video signals; Further comprising forming a stage scene video signal having an aspect ratio of substantially 2: 1 or less, wherein the panoramic video signal is 1/5 or less of the area of the stage scene video signal, 15. The method of claim 14, wherein the method extends substantially across the width. Subsampling a text video signal from a text document;
The method of claim 14, further comprising transitioning the text video signal to the stage scene video signal by replacing at least one of the at least two sub-scene video signals with the text video signal.   Further comprising setting at least one of the at least two sub-scene video signals as a protected sub-scene video signal that is protected from transition based on a retention criterion, the at least one of the at least two sub-scene video signals 4. The method of claim 3, wherein transitioning at least one of the additional subscene video signals to the stage scene video signal by replacing s transitions subscene video signals other than the protected subscene.   The method further comprises setting a sub-scene enhancement operation based on an enhancement criterion, wherein at least one of the at least two sub-scene video signals is enhanced according to the sub-scene enhancement operation based on a corresponding enhancement criterion. The method according to 1.   The method of claim 1, further comprising setting a sub-scene participant notification action based on a criterion detected from a sensor, wherein a local reminder indicator is activated according to the notification action based on a corresponding detected criterion. the method of.   The method of claim 1, wherein the panoramic video signal is captured from a wide camera having an aspect ratio substantially equal to or greater than 8: 1 and having a horizontal angle of view of substantially 360 degrees. A method of tracking a sub-scene in a target orientation within a wide video signal,
Monitoring a range of angles using an acoustic sensor array and a wide camera that observes a field of view substantially greater than 90 degrees;
Identifying a first target orientation along localization of at least one of acoustic recognition and visual recognition detected within the angular range;
Sub-sampling a first sub-scene video signal from the wide camera along the first target orientation;
Setting a width of the first sub-scene video signal according to signal characteristics of at least one of the acoustic recognition and the visual recognition.   The method of claim 21, wherein the signal characteristic represents a confidence level of at least one of the acoustic recognition and the visual recognition.   The method of claim 21, wherein the signal characteristic represents a width of a feature recognized within at least one of the acoustic recognition and the visual recognition.   24. The method of claim 23, wherein the signal characteristic corresponds to an approximate width of a human face recognized along the first target orientation.   24. The method of claim 23, wherein if a width is not set according to the visual recognition signal characteristic, a predetermined width is set along with localization of acoustic recognition detected within the angular range.   The method of claim 21, wherein the first target orientation is determined by visual recognition and the width of the first sub-scene video signal is set according to the signal characteristics of the visual recognition. The first target orientation is directed and identified toward acoustic recognition detected within the angular range; and
23. The method of claim 21, comprising identifying visual recognition proximate to the acoustic recognition, wherein the width of the first sub-scene video signal is set according to signal characteristics of the visual recognition proximate to the acoustic recognition. Method. A method of tracking a sub-scene in a target orientation within a wide video signal,
Scanning the subsampling window through a video signal that substantially corresponds to a wide camera field of view greater than 90 degrees;
Identifying candidate orientations in the sub-sampling window, each target orientation corresponding to localization of visual recognition detected in the sub-sampling window;
Recording the candidate orientation in a spatial map;
Monitoring an angular range corresponding to the wide camera field of view using an acoustic sensor array for acoustic recognition. When acoustic recognition is detected in proximity to one candidate orientation recorded in the spatial map,
Snapping a first target orientation to substantially correspond to the one candidate orientation;
29. The method of claim 28, comprising sub-sampling a first sub-scene video signal from the wide camera along the first target orientation.   30. The method of claim 29, further comprising setting a width of the first sub-scene video signal according to the signal characteristics of the acoustic recognition.   The method of claim 30, wherein the signal characteristic represents a confidence level of the acoustic recognition.   The method of claim 30, wherein the signal characteristic represents a width of a feature recognized within at least one of the acoustic recognition and the visual recognition.   31. The method of claim 30, wherein the signal characteristic corresponds to an approximate width of a human face recognized along the first target orientation.   31. The method of claim 30, wherein if a width is not set according to the visual recognition signal characteristic, a predetermined width is set along with localization of acoustic recognition detected within the angular range. A method for tracking a sub-scene in a target orientation,
Recording a video signal corresponding to a wide camera field of view substantially greater than 90 degrees;
Monitoring an angular range corresponding to the wide camera field of view using an acoustic sensor array for acoustic recognition;
Identifying a first target orientation that is directed toward acoustic recognition detected within the angular range;
Locating a subsampling window in the video signal according to the first target orientation;
Localizing the visual recognition detected within the sub-sampling window. Sub-sampling a first sub-scene video signal captured from the wide camera substantially centered on the visual recognition;
36. The method of claim 35, further comprising setting a width of the first sub-scene video signal according to the visual recognition signal characteristics. A method of tracking a sub-scene in a target orientation within a wide video signal,
Monitoring a range of angles using an acoustic sensor array and a wide camera that observes a field of view substantially greater than 90 degrees;
Identifying a plurality of object orientations each directed toward localization within the angular range;
Maintaining a spatial map of recorded characteristics corresponding to the target orientation;
Sub-sampling a sub-scene video signal from the wide camera substantially along at least one target orientation;
Setting a width of the sub-scene video signal according to a recorded characteristic corresponding to the at least one target orientation. A method of tracking a sub-scene in a target orientation within a wide video signal,
Monitoring a range of angles using an acoustic sensor array and a wide camera that observes a field of view substantially greater than 90 degrees;
Identifying a plurality of object orientations each directed toward localization within the angular range;
Sub-sampling a sub-scene video signal from the wide camera substantially along at least one target orientation;
Setting the width of the sub-scene video signal by expanding the sub-scene video signal until a threshold based on at least one recognition criterion is met. Predicting a change vector for each target orientation based on one change in velocity and direction of the recorded characteristic corresponding to localization;
39. The method of claim 38, further comprising updating the position of each target orientation based on the prediction. Predicting a search area for localization based on the most recent position of the recorded characteristic corresponding to localization;
39. The method of claim 38, further comprising updating the localization location based on the prediction.