JP2007264633A - Surround visual field system, method for synthesizing surround visual field relating to input stream, and surround visual field controller - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To synthesize a three-dimensional surround visual field. <P>SOLUTION: A surround visual field that has a characteristic or characteristics which relate to an audio/visual presentation is displayed in an area partially surrounding or surrounding the video content being displayed. This surround visual field may comprise a plurality of elements that further enhance the effect of the content being displayed. For example, elements within the surround visual field move in relation to motion within the video content being displayed. Other characteristics of the video content may also be supplemented by the surround visual field or, surround visual field can be partially made at least so that it may correspond to contents itself. The surround visual field may be a rendering of a three-dimensional environment. Further, one or more otherwise idle display areas may be used to display a surround visual field. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates generally to visual enhancement of audio / video presentations, and more specifically to the synthesis and display of surround visual fields associated with audio / visual presentations.

  Various technological advances in the audio / visual entertainment industry have greatly improved the personal experience of viewing media content. Some of these technological advances have improved the quality of video displayed on devices such as televisions, movie screening systems, computers, portable video devices, and other similar electronic devices. Other advances have improved the quality of audio provided to individuals during the display of media content. These advancements in audio / visual presentation technology were aimed at improving the enjoyment of individuals viewing this media content.

  An important element in media content presentations is to encourage individuals to immerse in the presentations that are viewed. Media presentations are often more attractive by feeling that an individual is part of a scene or that content is being viewed “live”. Such dynamic presentations temporarily stop viewers' distrust and create a more satisfying experience.

  This immersive principle has already been dealt with quite a bit with respect to the audio component of the media experience. Audio systems such as surround sound imitate real experience by providing audio content to individuals from several sources in the room. For example, multiple loudspeakers can be placed in a room and connected to an audio controller. The audio controller can generate sound from a particular speaker in relation to the corresponding video display and speaker location in the room. Such an audio system is intended to simulate a sound field in which a video scene is displayed.

US Patent Application Publication No. 2004/0119725

Current video display technology has not been so effective in creating immersive experiences for individuals. Some techniques use an external light source or projector in conjunction with a conventional display to enhance the immersive feeling. For example, Philips' Ambilight TV projects one of a certain number of color backlights behind the TV. Such a technique is insufficient because it does not address the problem of using the entire display area of the device when displaying content. In addition, current video display devices often do not fully adapt to the viewing range of the individual viewing the device or utilize a significant portion of the display. As a result, the immersive effect is reduced, thus reducing the personal viewing experience.
Accordingly, systems, apparatus, and methods that address the above limitations are desired.

  One embodiment of the present invention provides a surround visual field associated with displayed audio or visual content. In one embodiment of the invention, the surround visual field is synthesized and projected onto a surface that partially or completely surrounds the device displaying the content. This surround visual field is intended to further improve the viewing experience of the displayed content. Accordingly, the surround visual field can augment, expand, or otherwise supplement one or several of the characteristics of the displayed content. One skilled in the art will appreciate that the surround visual field may be associated with one or more cues or control signals. The cue or control signal associated with the input stream includes, but is not limited to, movement, color, brightness, audio, genre, and behavior within the display content. This includes, but is not limited to, device inputs, or other inputs, and includes queues associated with user-provided inputs. In one embodiment, one or more elements in the surround visual field can be associated with the queue or queues by responding to the queue or queues.

  In one embodiment of the invention, the surround visual field is projected or displayed during the presentation of audio / video content. The size, location, and shape of this surround visual field can be defined by the visual field producer, can be related to the displayed content, or can be defined separately. Furthermore, the characteristics of the surround visual field can include various shapes, textures, patterns, waves or any other visual effect that enhances viewing of the content of the display device. Those skilled in the art will appreciate that various visual / visual or projection systems can be used to generate and control the surround visual field. All these systems are intended to fall within the scope of the present invention.

  In an exemplary embodiment of the invention, the surround visual field can relate to movement within the displayed content. For example, the movement in the displayed content can be modeled and extrapolated. The surround visual field, or its components, can move according to extrapolated movement in the content. Shapes, patterns or any other element within the surround visual field can also have characteristics that are further related to the movement of the content or any other characteristics.

  In an embodiment of the invention, a three-dimensional surround visual field can be synthesized or generated, where one or more elements of the surround field are affected by one or more cues associated with the input stream. For example, motion cues associated with display content can be provided and modeled within a 3D surround visual field environment. The surround visual field or elements therein can move according to extrapolated movement in the content. Light source, graphics, camera movement, and dynamics of composite elements in a 3D surround visual field environment can also have properties that are further related to the input stream.

  In embodiments, the surround visual field can be displayed in one or more portions of the empty display area. As with other embodiments, the surround visual field or portion thereof can be modified or changed based on one or more control signals extracted from the input stream. Alternatively, or additionally, the surround visual field displayed in the empty display area can be based on produced or partially produced content or cues.

  Reference will now be made to embodiments of the invention, examples of which are illustrated in the accompanying drawings.

  While the features and advantages of the invention will be generally described in the form of embodiments in this summary and the following detailed description, it will be understood that the scope of the invention should not be limited to these specific embodiments. . Many more features and advantages will be apparent to those skilled in the art in view of the drawings, specification, and claims of this application.

  Systems, apparatus, and methods that can be used in conjunction with audio / visual content are described. In one embodiment of the invention, the surround visual field is synthesized and displayed during the presentation of the audio / visual content. The surround visual field can consist of various visual effects, including but not limited to images, various patterns, colors, shapes, textures, graphics, text, and the like. In one embodiment, the surround visual field has one or more characteristics related to audio / visual content and supplements the content viewing experience. In one embodiment, the elements in the surround visual field, or the surround visual field itself, change visually with respect to the audio / visual content or the environment in which the audio / visual content is displayed. For example, elements in the surround visual field can move in relation to movement and / or color in the displayed video content.

  In another embodiment of the invention, the surround visual field may be created to relate to audio / visual content and not automatically generated when viewed. For example, the surround visual field can be synchronized to the content to enhance the viewing experience of both the content and the surround visual field. Those skilled in the art will appreciate that the surround visual field and audio / visual content can be associated and presented visually to the individual in a number of ways, all of which are within the scope of the present invention.

  In the following description, for the purposes of explanation, specific details are set forth in order to provide a thorough understanding of the invention. However, one skilled in the art will understand that the invention may be practiced without these details. Those skilled in the art will appreciate that several embodiments of the present invention, described below, may include several different systems, including projection systems, theater systems, televisions, home entertainment systems, and other types of audio / visual entertainment systems. Understand that it can be incorporated into the device. Embodiments of the invention may also reside in software, hardware, firmware, or a combination thereof. The structures and devices shown below in the block diagram illustrate exemplary embodiments of the invention and are intended to avoid obscuring the invention. Further, the connections between the components and / or modules in the figures are not intended to be limited to direct connections. Rather, the data between these components and modules can be modified, reformatted, or otherwise changed by intermediate components and modules.

  References herein to “one embodiment” or “an embodiment” include that a particular property, structure, feature, or function described in connection with that embodiment is included in at least one embodiment of the invention Means. The appearances of the phrases “in one embodiment” or “in an embodiment” in various places in the specification are not necessarily all referring to the same embodiment.

[Overview]
FIG. 1 illustrates a surround visual field display system that can be incorporated into a theater or home video environment, according to one embodiment of the invention. System 100 includes a projector 120 that projects video content in a first area 110 and a surround visual field in a second area 130 that surrounds the first area 110. The surround visual field does not necessarily have to be projected around the first area 110. Rather, this second region 130 partially surrounds the first region 110, can be adjacent to the first region 110, or otherwise projected within the individual's field of view.

  Projectors can be controlled by radiating and controlling a single ordinary projector, a single panoramic projector, multiple multitude projectors, a mirror projector, a new projector with a panoramic projection field, any hybrid of these projectors, or a surround visual field Can be any other projector. By using wide-angle optics, a wide field of view can be projected by one or more projectors. Methods to achieve this include, but are not limited to, catadioptric systems involving the use of fisheye lenses and the use of curved mirrors, cone mirrors or specular pyramids. The surround visual field projected in the second region 130 may include various images, patterns, shapes, colors, and textures that may include discontinuous elements having various sizes and attributes. It can relate to the characteristics of the displayed audio / video content. These patterns and textures can include, without limitation, starry patterns, fireworks, waves, or any other pattern or texture.

  In one embodiment of the invention, the surround visual field is projected in the second area 130, but not in the first area 110 where the actual video is displayed. In another embodiment of the invention, the surround visual field may be projected in the first area 110 or may be projected in both the first area 110 and the second area 130. In one embodiment, when a surround visual field is projected in the first area 110, certain aspects of the displayed video content are highlighted, highlighted, or otherwise supplemented by the surround visual field. Can do. For example, by projecting a visual field onto a subject that has a specific movement in the video content, that specific movement displayed in the first region 110 can be highlighted.

  In yet another embodiment, a texture synthesis pattern can be generated that effectively extends the content of the video out of its frame. If there is a regular or quasi-regular pattern in the video frame, the projector 120 can spread the same or similar pattern out of the first area 110 and project it in the second area 130. For example, a corn field in a video frame can be extended outside the first region 110 by generating a pattern that looks like an extension of the corn field.

  FIG. 2 illustrates a surround visual field for a television receiver, according to one embodiment of the invention. A television receiver with a clear viewing screen 210 is supplemented with a surround visual field projected onto the wall surface 230 behind the television receiver. For example, video content may be displayed using a large television receiver or a video wall consisting of walls for displaying a projected image or set of displays. This surface 230 may vary in size and shape and is not limited to a single wall, but can be spread over as much of the area as is desirable in the room. Further, the surface 230 need not necessarily surround the television receiver as shown, but may partially surround the television receiver or be located elsewhere on the wall or walls. As described above, the surround visual field can have various characteristics associated with content displayed on the television receiver screen 210. Various embodiments of the invention can be used to project a surround visual field onto a wall surface, two examples of which are described below.

  FIG. 3 illustrates one embodiment in which a surround visual field is projected directly into an area 330 to supplement content displayed on the television receiver 310 or other surface. Although region 330 is shown as shown on only one wall, it can extend over multiple walls depending on the projector 320 or room configuration used. Projector 320 is integrated or connected to a device (not shown) that controls the surround visual field to be projected. In one embodiment, the device can be provided with a video stream that is displayed on the television screen 310. In another embodiment, the device can include data produced to synchronize the surround visual field with the projection in the content displayed on the television screen 310. In various embodiments of the invention, the audio / video stream is analyzed for one or more characteristics of the input stream, and the surround visual field is correctly represented and animated to synchronize with the content displayed on the television screen 310. The

  In yet another embodiment of the invention, the video display and surround visual field can be displayed within the boundaries of the display device, such as a television receiver, computer monitor, laptop computer, portable device, and the like. In this particular embodiment, there may or may not be a projection device that extends the surround visual field beyond the boundaries of the display device. The surround visual field displayed within the boundaries of the display device can have a variety of shapes and can include a variety of content including images, patterns, textures, text, changing colors, or other content.

  FIG. 4 illustrates a reflection system that provides a surround visual field according to another embodiment of the invention. System 400 may include a single projector or multiple projectors 440 that generate a surround visual field. In one embodiment of the invention, a plurality of projectors 440 generate a visual field that is reflected by the specular pyramid 420 to effectively create a virtual projector. Multiple projectors 440 can be integrated into the same projector housing or housed in separate housings. The specular pyramid 420 has a plurality of reflective surfaces that can reflect light from the projector in a region that is preferred for display of the surround visual field. The design of the specular pyramid 420 can vary depending on the area desired for visual field display and the type and number of projectors used in the system. In addition, other types of reflectors can be used in the system to reflect the visual field from the projector to the desired surface. In another embodiment, a single projector using one reflective surface of the specular pyramid 420 and effectively using a plane mirror can be used. A single projector can also project on multiple surfaces of the specular pyramid 420, in which case multiple virtual optical centers are created.

  In one embodiment of the invention, the projector or projectors 440 project a surround visual field 430 that is reflected and projected onto the surface of the wall 450 behind the television 410. As described above, this surround visual field can consist of various images, shapes, patterns, textures, colors, etc. and can be related to the content displayed on television 410 in various ways.

  Those skilled in the art will appreciate that various reflectors and configurations can be used in the system 400 to achieve various effects in the surround visual field. Further, the projector 440 or the plurality of projectors can be integrated into the television 410 or furniture that houses the television 410. Those skilled in the art will further appreciate that one or more televisions can be used to display input content and surround fields, including but not limited to display sets such as single display or tiled display. .

[Use of surround visual field]
While the above description generally describes the use of the Salangdo visual field in connection with audio / visual environments such as home television and projection systems, theater systems, display devices, and portable display devices, the invention is It can be applied to many environments. Furthermore, the system used to generate and control the surround visual field can have additional functions that further complement the basic implementation described above. Only a few such examples are given below, and those skilled in the art will appreciate that other applications not described below are within the scope of the present invention.

[Use of game]
The surround visual field can be created and controlled in relation to the characteristics of the video game that the individual is playing. For example, when the user is moving in the left direction, the contents of the previously displayed screen can be stitched together and displayed on the right side of the surround area. Other effects such as game controller vibrations can also be associated with the displayed surround visual field to enhance the vibration experience. In one embodiment, the surround visual field is synthesized by processing the video stream of the game being played.

[Interactive Surround Visual Field]
The surround visual field can be interactively controlled by users such as watching videos, listening to music, playing video games, and so on. In one embodiment, the user can control certain aspects of the surround visual field that is displayed. In another embodiment, the surround visual field can sense its environment and respond to events in the environment, such as responding to the viewer's location in the room where the system is operating.

  Perspective correction can also be provided by a surround visual system. Often the viewer is not in the same position as the momentary center of the surround visual system projection. In such cases, the surround visual field may appear distorted by the three-dimensional shape of the room. For example, a uniform pattern may appear more dense on one side and more sparse on the other side due to a mismatch between the instantaneous center of the projector and the viewer's position. However, if the viewer's position can be sensed, the system can correct the discrepancy in the surround visual field projection. This position can be sensed using a variety of techniques including the use of sensors (eg, infrared LEDs) located on the television remote control. Other sensors such as cameras, microphones, and other input devices such as game controllers, keyboards, pointing devices, etc. can be used so that the user can provide input queues.

[Sensor enhancement display]
Ensure that correct alignment and calibration between components is maintained using sensors placed on components in the surround visual field system, ensure that the system is adapted to the specific environment, and An input queue can be provided. For example, in the system shown in FIG. 3, it is important for the projector 320 to identify where the television is located within the projection area. This identification causes the projector 320 to (1) focus the surround visual field (in area 330) around the television receiver screen 310, and (2) allow the surround visual field to be projected on the television if desired. And (3) assisting the surround visual field pattern to be seamlessly matched with the television receiver display.

  In one embodiment, the sensor can be mounted separately from the projection or display optics. In another embodiment, the sensors can be designed to share the light path of the projector or display using a beam splitter or the like.

  In yet another embodiment, certain media may incorporate one or more surround video tracks that can be displayed in a surround visual field. One possible form of such media can be a built-in sprite or animated visual object that can be deployed as appropriate within the surround visual field to create an optical illusion or enhancement. For example, an explosion in a display video can be extended beyond the boundaries of a television receiver by simulating the explosion effect in a surround visual field. In yet another example, a thrown kite can be extended beyond the television screen to visualize its trajectory in the surround visual field. These extensions within the surround visual field can be produced by an individual or content provider and synchronized with the media content being displayed.

  Other implementations such as telepresence and augmented reality can also be provided by the present invention. Telepresence uses a surround visual field to display images captured in places other than the room, creating the illusion that viewers are transported to another location. For example, a pattern showing a panoramic view from a beach resort or rainforest can be displayed on the wall. In addition, images captured by visual sensors in various surround visual field systems can be used to create images on the wall that are a mixture of real and composite objects.

[Animation of surround visual field]
As mentioned above, the present invention allows the generation and control of surround visual fields related to displayed audio / visual content. In one embodiment, the surround visual field can be colored based on colors sampled from a regular video stream. For example, if a video stream displays a dominant color and the surround visual stream displays a particular simulation, the surround visual field may reflect this dominant color within that field. it can. Elements in the surround visual field can be changed to the dominant color, the surround visual field itself can be changed to the dominant color, or other characteristics of the surround visual field can be used To supplement the colors in the video stream. This colorization of the surround visual field can be used to enhance the mood effect of lighting normally used for normal content, such as a sequence of color filters, lightning, etc.

  In yet another embodiment, the surround visual field system can relate to the audio characteristics of the video stream, such as a “surround sound” audio component. For example, the surround visual field can be responsive to the strength of the audio portion of the video stream, the pitch of the audio portion, or other audio characteristics. Thus, the surround visual field is not limited to the visual content of the video stream, but can be related to audio or other characteristics.

  For illustrative purposes, an embodiment is described that uses movement in video content to define movement of elements in a surround visual field. Those skilled in the art will appreciate that various other characteristics of audio / visual content can be used to generate or control a surround visual field. Furthermore, the surround visual field cues or content can be related to and / or synchronized with the content produced and displayed by the individual.

[Surround visual field related to movement]
FIG. 5 illustrates an exemplary surround visual field controller that uses motion in video content to generate a surround visual field in one embodiment of the invention. The controller 500 may be integrated into the projection device, connected to the projection device, or otherwise enabled to control a surround visual field that is projected and displayed in the viewing area. In one embodiment, the controller 500 is provided with a video signal that is subsequently processed to generate and control at least one surround visual field with respect to the characteristics of the video signal, or cue / control signals. For example, the controller 500 can display and control surround visual fields related to movement within the displayed video content.

  In one embodiment, the controller 500 creates a global motion model 510 between pairs of consecutive video frames, a motion field extrapolation component that extrapolates the global motion model across video frame boundaries. 540, and a surround visual field animation component 550 that displays and controls the surround visual field and the elements therein in relation to the extrapolation motion model. In one embodiment, motion estimation component 510 constructs a global motion model using optical flow estimation component 515 that identifies an optical flow vector between pairs of consecutive video frames and the identified optical flow vector. A global motion modeling component 525 is included. Each component will be described in detail below.

[Motion estimation component]
The motion estimation component 510 analyzes motion between pairs of video frames and creates a model that can estimate motion between pairs of frames. The accuracy of the model depends on the density of the optical flow vector used to generate the model, the type of model used and the number of parameters in the model, and the amount and consistency of motion between the video frame pair. Depending on the factors. The following embodiments are described with respect to successive video frames. However, the present invention can estimate and extrapolate motion between any two or more frames in the video signal, and use the extrapolated motion to control the surround visual field.

  In one example, motion vectors encoded in the video signal can be extracted and used to identify motion trajectories between video frames. Those skilled in the art can encode and extract these motion vectors into video signals in various ways, including those defined in various video coding standards (eg, MPEG, H.264, etc.). Let's understand that. In another example described in more detail below, an optical flow vector that describes motion between video frames can be identified. Various other methods can be used to identify motion within the video signal, all of which are intended to fall within the scope of the present invention.

[Optical flow estimation component]
In one embodiment of the invention, the optical flow estimation component identifies a plurality of optical flow vectors between a pair of frames. Vectors can be defined in various motion granularities, including pixel-to-pixel vectors and block-to-block vectors. These vectors can be used to create an optical flow vector field that describes the motion between frames.

  Vectors are identified using a variety of techniques, including correlation methods, coded motion vector extraction, spatio-temporal motion gradient detection methods, feature-based motion detection methods, and other methods for tracking motion between video frames. can do.

  The correlation method for determining optical flow may include comparing a portion of the first image with a portion of a second image having a similar luminance pattern. Correlation is usually used to match the characteristics of an image or to find the motion of an image once the characteristics have been determined by another method.

  The optical flow can be determined using the motion vectors generated in the coding of the video frame. Typically, a motion estimation procedure is performed during the encoding process to identify similar blocks of pixels and draw the motion of these blocks of pixels across multiple video frames. These blocks include 16 × 16 macroblocks and sub-blocks therein, and can be of various sizes. This motion information can be extracted to generate an optical flow vector field.

  The determination method based on the gradient of optical flow uses a spatiotemporal partial derivative to estimate the image flow at each point of the image. For example, the spatio-temporal derivative of the image luminance function can be used to identify changes in luminance or pixel intensity, which can partially determine the optical flow of the image. Using gradient-based methods to identify optical flow may result in the observed optical flow deviating from actual image flow in regions other than regions with strong image gradients (eg, edges). However, this deviation may be acceptable when developing a global motion model for a pair of video frames.

  The characteristic-based optical flow determination method is focused on computing and analyzing optical flow at a small number of distinct image characteristics, such as edges within a frame. For example, a distinct set of characteristics can be mapped to identify motion between two consecutive video frames. Other methods are also known that can map characteristics through a series of frames and define the course of movement of a characteristic through a larger number of consecutive video frames.

  FIG. 6 shows a representative optical flow vector at the pixel level between successive video frames, according to one embodiment of the invention. A first set of pixel points in frame (k-1) 610 of the first frame is identified. This identification can be based on motion identified in previous frames, motion vector information extracted from the coding of video frame 610, randomly generated, or otherwise selected for multiple points. Can be specified.

A vector describing the two-dimensional motion of the pixel from the position in the first video frame 610 to the position in the second video frame 620 is identified. For example, the movement of the first pixel at the position (x ₁ , y ₁ ) 611 can be specified by the motion vector 641 to the position (u ₁ , v ₁ ) 621 in the second frame. The field of optical flow vectors may include a variable number (N) of vectors that describe the movement of pixels between the first frame 610 and the second frame 620.

  FIG. 7 shows a pair of consecutive video frames in which optical flow vectors between blocks are specified, according to one embodiment of the invention. As described above, the optical flow vector also depicts the motion of the pixel block including the macroblock and the sub-blocks therein between the first frame, frame (k-1) 710 and second frame, frame (k) 720. Can do. These vectors are generated using the various techniques described above, including being extracted from the coded video that provides both motion and distortion between the video blocks so that the video can be played on the display device. Can do. Using the extracted motion vectors, a field of optical flow vectors can then be generated. The field of the optical flow vector can also be generated by performing motion estimation, in which case the blocks in the first frame 710 search for similar blocks having the same or nearly the same pixel values in the second frame. Is specified in the second frame 720. Once a block is identified in each frame, a motion vector that describes the two-dimensional motion of the block can be generated.

[Global motion modeling component]
An optical flow vector can be used to generate a global model of motion that occurs between pairs of consecutive video frames. Using the identified optical flow vector field, motion between pairs of video frames can be modeled. Various models can be used to estimate the optional optical flow between a pair of video frames. Usually the accuracy of a model depends on the number of parameters defined in the model and the characteristics of the motion they represent. For example, a three parameter model can represent movement along two axes and the associated rotation angle. The four parameter model can represent the movement along two axes, the rotation angle, and the magnification at which the motion in the frame is drawn.

  In one embodiment of the invention, a six parameter model called an “Affine Model” is used to model motion within a video frame. This specific model represents the movement vector, the rotation angle, two magnifications along the axis, and the azimuth angle of the magnification. In general, this model is a composition of rotation, translation, expansion, and displacement that represents motion between a pair of video frames.

A global motion modeling component 525 receives the optical flow vector field information and generates a six parameter affine model that estimates the global motion between a pair of video frames. From this model, motion between frame pairs can be estimated by the following two equations:
u = a ₁ + a ₂ x + a ₃ y Expression (1)
v = a ₄ + a ₅ x + a ₆ y (2)
Here a ₁ ... a ₆ is a parameter of the model.

In order to solve the _six parameters, a ₁ to a ₆ , at least three optical flow vectors need to be defined in advance. However, depending on the accuracy desired for the model, the optical flow vector field that creates the model can be of higher density to improve the robustness and accuracy of the model.

Global motion modeling component 525 defines the model by optimizing parameters for the provided optical flow vector field. For example, N optical flow vectors and corresponding N point pairs (x ₁ , y ₁ )... (X _N , y _N ) and (u ₁ , v ₁ )... (U _N , v _N ) If provided, the parameters a ₁ to a ₆ can be solved by optimization calculations or procedures.

Once the six parameters are optimized to identify the minimum error between the model and the optical flow vector field, a global motion model is generated. One way to optimize the parameters is a least square error that fits for each vector in the optical flow vector field. A parameter value is selected that provides a least square error between the optical flow vector field and the corresponding modeled vector.

FIG. 8 shows an example of how motion vectors in the global motion model 810 can be generated according to one embodiment of the invention. In this example, the motion for (x _i , y _i ) 820 is identified by solving equation 850 that computes (u _i , v _i ) 830 for the affine model. A motion vector 825 can be calculated from these two points and used to grow the global motion model 810.

  The use of an affine model as described above to generate a global motion model is not intended to exclude other models. For example, an eight parameter model representing a further three-dimensional rotation can also be used, which may be able to more accurately depict the motion within the video frame. However, as the parameters increase, more computation is required to build and extrapolate the model. Accordingly, those skilled in the art will appreciate that various models can be used depending on the desired accuracy of the global motion model and the computing power available in the system.

  FIG. 9 illustrates an exemplary global motion model 910 between video frames, according to one embodiment of the invention. The figure shows a plurality of motion vectors in the model, including four vectors that estimate the motion associated with the four optical flow vectors shown in the previous figures.

  FIG. 10A shows a representative optical flow vector field 1010 overlaid on a video frame, and FIG. 10B shows a global motion model 1030 generated from the representative vector field and overlaid on the same video frame. . By reviewing this, one skilled in the art will understand that a global motion model can be used to extrapolate modeled motion within a video frame across video frame boundaries.

[Motion field extrapolation components]
The motion field extrapolation component 540 extends the global motion model beyond the boundaries of the video frame and allows elements within the surround visual field that cross these frame boundaries to respond to motion within the frame. In one embodiment of the invention, the estimated motion across frame boundaries is extended using affine model equations that define motion vectors from (x _N , y _N ) to (u _N , v _N ). Where (x _N , y _N ) is located beyond the boundaries of the video frame.

  FIG. 11 illustrates exemplary motion extrapolation that can be performed according to an embodiment of the invention. A first set of motion vectors 1120 is shown having a slightly upward angle and a motion toward the left boundary. This motion can be extrapolated across frame boundaries by using a global motion model. Thereby, a second set of motion vectors 1130 beyond the boundaries of the video frame is generated. In another example, a third set of motion vectors 1140 having a clockwise rotation is shown. This rotational motion can also be extrapolated beyond the video frame by using a global motion model, resulting in a fourth set of motion vectors 1150 outside the frame boundaries.

  These motion vectors (eg, 1130, 1150) can be used to define the motion of the surround visual field and / or elements therein that are projected around the display of the video frame. As the motion in the frame changes, the global motion model responds, resulting in a change in the surround visual field. In one embodiment of the invention, elements in the surround visual field are then responsive to and controlled by extrapolated motion vectors using a global motion model.

  FIG. 12 illustrates an exemplary extrapolated global motion model that can be used to control the motion of the surround visual field around a displayed video frame, according to one embodiment of the invention. The vector defined by the global motion model 1220 in the frame is shown and estimates the motion in the frame itself. This model extends beyond the frame boundaries to provide an extrapolated global motion model 1230. The vectors in the extrapolated global motion model 1230 can control the motion of elements in the surround visual field.

  The surround visual field can also be projected on a device that displays a video frame. In this case, the motion of the element in the surround visual field on the device is controlled by a vector in the global motion model 1220 that estimates the motion in the video frame.

[Surround Visual Field Animation Component]
The surround visual field animation component creates, animates, and maintains the projected surround visual field according to at least one characteristic of the video content. In one embodiment, as described above, elements in the surround visual field move in relation to movement in the video being displayed.

  Surround visual fields can be generated and maintained using various techniques. In one embodiment of the invention, elements in the surround visual field are randomly generated in the field and disappear over time. Another element that randomly replaces the missing element is randomly inserted into the surround visual field. These other elements also decay with time and disappear gradually. Element decay and random replacement of elements within the surround visual field reduce the concentration or grouping of elements within the surround visual field due to long-term movement of the elements.

[Surround visual field element shape]
In addition to motion, other characteristics including elements of the surround visual field can be controlled by the extrapolated global motion model. For example, the shape of each element in the field can be determined by a vector in the global motion model. FIG. 13 illustrates one way in which the shape of an element is associated with a corresponding motion vector.

  In one embodiment of the invention, the shape of element 1310 is affected by a motion vector 1320 that corresponds to the position of element 1310 relative to the global motion model. For example, an element 1310 can be expanded along a weight given to the axis of the corresponding motion vector 1320 and the direction of the motion vector. In the example shown in FIG. 13, the deformed element 1340 is stretched along the axis of motion, resulting in a narrower tail 1360 and a wider head 1350 in the direction of the motion vector.

  Other characteristics of the changed element 1340 can also be changed to reflect the motion vector 1320. For example, the brightness of the head of the element 1340 having a changed shape is bright and can gradually become darker as it approaches the tail 1360 of the element. This reduction in luminance with respect to motion can enhance perceived motion blur as the element moves in the surround visual field.

  In yet another embodiment, the shape of the element can correspond to the motion of a continuous motion vector related to the element itself. FIG. 14 can be defined according to a plurality of motion vectors where the shape and motion of an element occur over time in the global motion model. In this embodiment, the element moves with respect to two consecutive motion vectors 1410, 1420 modeled from two video frame pairs. A trajectory defined by two vectors includes a sharp turn at the end of the first vector 1410 and the beginning of the second vector 1420. This swirling can degrade the quality of the movement of the element following the trajectory and can be tangled when the element moves.

  The trajectory can be smoothed into a curved trajectory 1430 that does not include a sudden change in motion. This smoothing can be done with various mathematical equations and models. For example, an altered element 1450 can reflect a curved trajectory with the element 1450 stretched along the curve 1430. The brightness of the deformed element 1450 can vary, and the brightness near the head of the point is the brightest to further enhance the movement, and gradually decreases as it approaches the tail. it can.

  Those skilled in the art will appreciate that there are other ways to change the shape of an element to emphasize the movement of the surround visual field.

  FIG. 15 is an example illustrating a video presentation including a surround visual field according to one embodiment of the invention. In this example, a video in which the counterclockwise movement is dominant in the frame is displayed on the screen 1510. This dominant motion is modeled, extrapolated, and used to animate the surround visual field 1530. In this specific example, the surround visual field 1530 also rotates counterclockwise, thereby enhancing the movement within the screen 1510. This surround visual field can greatly expand the area where motion is displayed to the individual and increase the immersive effect of the video itself.

[Create a 3D surround environment]
In embodiments, an interactive and immersive three-dimensional (3D) environment can be created that uses various techniques to improve the field of view of normal displays. For example, a three-dimensional environment of natural phenomena such as terrain, sea, etc. can be synthesized and a two-dimensional depiction displayed as a surround video field. As mentioned above, embodiments of the present invention can improve the immersiveness of the entertainment system by creating a surround field presentation using one or more associated cues or control signals of the input stream. In an embodiment, the three-dimensional environment may be interactive, with elements in the environment changing, such as scene lighting, camera movement, audio, etc., in response to changes in the input stream.

  In an embodiment, interactivity can be achieved using physical simulation, where one or more dynamics or elements of the surround scene are controlled by one or more cues or control signals associated with the input stream. In one embodiment, data from the input stream can be used to represent these three-dimensional surround simulations in real time, and one or more image-based representation algorithms can be used. In one embodiment, the surround field can be generated from pre-computed data, including but not limited to image-based representations and produced cues and / or produced content.

[Surround Field Controller]
FIG. 16 illustrates an exemplary surround field controller 1600 according to one embodiment of the invention, where queues or control signals are extracted from the input stream or received from other sources to generate the surround field. Used. The controller 1600 is integrated into a display device (considered to include a projection device), connected to the display device, or otherwise enabled to control the surround field displayed in the viewing area. Controller 1600 may be implemented in software, hardware, firmware, or a combination thereof. In one embodiment, the controller 1600 receives one or more input signals that can be subsequently processed to generate and / or control at least one surround visual field.

  As depicted in FIG. 16, the controller 1600 includes a control signal or cue extraction component 1610 that includes an input audio / video stream, an input device (such as a game controller), one or more sensors (included in the remote control). And a plurality of extraction components 1612-1620 that extract or derive control signals from a variety of sources including embedded or produced control signals or produced content. For example, the controller 1600 can represent and control surround visual fields associated with movement within the displayed content. In one embodiment, the control signal extraction component 1610 can obtain a motion cue or control signal from the video as described above. In one embodiment, the control signal extraction component 1610 can obtain a control signal from an audio signal using audio signals such as phase differences between audio channels, volume levels, audio frequency analysis, and the like. In one embodiment, the content provider can embed control signals in the input video stream or include control signals in the data channel.

  In one embodiment, the control signal extraction component 1610 can be coupled to a surround visual field generation / animation component 1650. The term “coupled” or “communicatively coupled”, whether used with respect to a module, device, or system, is understood to include direct connection, indirect connection through one or more intermediate devices, and wireless connection. . The extracted control signal is provided to a surround field generation component or an animation component 1650, which creates or synthesizes a surround field using the control signal. The surround field generation component 1650 can be comprised of one or more subcomponents or modules as described in connection with FIG. In an embodiment, the surround field generation component 1650 can generate a three-dimensional environment using physics-based modeling and control signals, and make the elements of the surround field react realistically. Embodiments of the surround field generation component 1650 can generate a surround field using pre-computed data such as image-based techniques and / or produced content. In embodiments, the surround field generation component 1650 can generate non-realistic effects, or supplemental effects such as motion extensions or highlights, scene extensions, color extensions or highlights, and the like.

  The surround field generation component or animation component 1650 can use more than one control signal in creating the surround field. In an embodiment, the generation component 1650 can use multiple control signals to animate the elements in the surround field to match the content creator's design. The generation component 1650 can further simplify the determination to resolve conflicting control signals by using or comparing the control signals.

  In an embodiment, the controller 1600 can animate the surround field based on the surround field information provided by the content provider. For example, a video game or movie provider can produce or include surround field information in conjunction with an input stream. In one embodiment, the surround field can be fully produced. In another embodiment, the surround field can be partially created. In an embodiment, one or more control signals may be provided and a surround field associated with the provided control signal may be generated.

  It is noted that the particular configuration of controller 1600 is not critical to the present invention. Those skilled in the art will appreciate that other configurations or functionality can be excluded from or included in the controller and such configurations are within the scope of the invention.

[Derivation of animation control signals from images and videos]
As discussed above, the elements displayed in the surround video can be animated based on control signals or cues extracted from the input stream. In one embodiment, control signals from the input stream can be obtained from one or more sources, which can be obtained from video frames (such as color and motion), audio channels, game controllers, input sensors. But includes, but is not limited to, viewer position, remote control, and input from other sensors. In embodiments, one or more video control signals driven by or associated with one or more cues may be computed.

[light source]
In one embodiment, input control signals can be used to control the position, brightness, and / or color of one or more light sources in a three-dimensional surround environment. For example, if the source video displays a bright object moving in one direction, such as light, moon, sun, car headlights, etc., from left to right, a virtual light source of the same color as the bright object is also 3D It can move in the same direction within the surround environment and can cause changes in the appearance of the scene due to surface shading differences and shadow movement. The virtual light source in the scene can further change its brightness based on the overall brightness of the video frame.

[Wind Field]
In another illustrative example, motion in the source video stream can be further used to create a wind field in a 3D surround environment. For example, if an object moves across the video, a virtual three-dimensional surround field can be created in the wind field and the elements in the scene can be moved in the same direction. That is, elements in the scene, such as trees, can move or sway in relation to the wind field.

[Disturbance]
In one embodiment, events detected from one or more input queues can also be used to introduce disturbances in a 3D surround environment. In an embodiment, if the video transitions from a slight or zero motion period to a scene with a lot of motion, a “disturb” event may be introduced to allow elements in the surround scene to react to the event.

  As an example, consider a surround scene where fish are swimming. If one or more input cues show a disruption event, such as a dramatic increase in audio volume and / or a sudden movement in the video, the fish can move and swim faster and faster when the disturbance is introduced. . In one embodiment, the fish may be allowed to swim away from what is seen as the center point of the disturbance.

[Composition of 3D surround field]
One aspect of the present invention is the synthesis of a three-dimensional environment, which can then be displayed as a surround field. In embodiments, the surround field can be synthesized using physics-based simulation and representation techniques well known to those skilled in the art. In one embodiment, a realistic background of natural phenomena such as mountains, forests, waves, clouds, etc. can be synthesized. In embodiments, other backgrounds or environments are depicted and may react at least in part according to one or more control signals. To generate interactive content and display it in a surround field, two-dimensional and / or three-dimensional simulations can be combined or provided with control signals extracted from the input stream.

  For purposes of illustration, consider the following 3D simulation embodiment in which dynamics are approximated by a Perlin noise function. The Perlin noise function is widely used in computer graphics to model terrain, texture, and water, and is discussed in the following: Ken Perlin's "Image Synthesizer" ("An image synthesizer"), Computer Graphics ( Proceedings of SIGGRAPH 1985), Vol. 19, pp. 287296, July 1985, Claes Johans' "Real-time water rendering", Master of Science Thesis, Lund University, March 2004 , And Ken Perlin and Eric M. Hoffert, "Hypertexture" (Hypertexture), Computer Graphics (Proceedings of SIGGRAPH 1989), Vol. 23, 253262, July 1989. Each of which is incorporated herein by reference in its entirety. It is noted that the techniques described herein include, but are not limited to, physics-based systems and can be extended to other types of 3D simulations.

  A one-dimensional perlin function is obtained by summing several noise generators Noise (x) at different amplitudes and frequencies:

The function Noise (x) is a seeded random number generator that takes an integer as an input parameter and returns a random number based on the input. The number of noise generators can be controlled by the parameter octaves, and the frequency at each level is incremented by a square. The parameter α controls the amplitude of each level, and β controls the overall scaling. Using the two-dimensional version of equation (4), natural terrain can be simulated. Using a three-dimensional version of equation (4), a water simulation can be created.

  Parameters for simulating water in real time are driven by synthesizing a responsive 3D surround field using the input video stream. Camera motion, light source, and three-dimensional water simulation can be combined into sampled motion vectors, colors, and audio signals from the video.

In one embodiment, the motion of the virtual camera can be governed by the main motion of the input video stream. To create a 3D simulation response “fly-through” as described above, an affine motion model as described above can be matched to a motion vector from a video stream. The affine motion field can be decomposed into pan, tilt and zoom components around the image center (c _x , c _y ). Using these three components, the movement of the camera can be controlled in the simulation.

  FIG. 17 represents an input video stream 1710 and a motion vector field 1740, where the pan, tilt and zoom components can be computed from the motion vector field. In one embodiment, the pan, tilt and zoom components can be obtained by computing the motion vector projections at four points 1760A-1760D that are equidistant from a center 1750. The four points 1760A-1760D and the direction of projection are depicted in FIG.

The pan component is obtained by summing the velocity vectors (u _i , v _i ) of the four symmetry points (x _i , y _i ) 1760A-1760D around the image center 1750:

  The tilt component is obtained by summing the vertical components of the velocity vector at the same four points:

In one embodiment, the control signal can be used to control the light source in the 3D synthesis. Three-dimensional simulations have expression parameters that control the final color of the output that is usually represented. Hue in the composition environment may be controlled or influenced by one or more color values extracted from the input stream. In one embodiment, the three-dimensional environment may be controlled or influenced by the three-dimensional light source C _light , the overall brightness C _avg , and the ambient color C _amb . In one embodiment, the average luminance, brightest color, and median color can be computed for each frame of video and these values can be assigned to C _avg and C _amb , respectively. One skilled in the art will appreciate that other color values or frequency of color sampling can be used.

In one embodiment, the simulation dynamics can be controlled by the parameters α and β in equation (4). Illustratively, in a water simulation, the parameter α controls the amount of ripples in the water, and the parameter β controls the overall size of the waves. In one embodiment, these two simulation parameters can be combined into an audio amplitude A _amp and a motion amplitude M _amp as follows:

Here, M _amp = V _pan + V _tilt + V _zoom , f (A _amp ) and g (M _amp ) are linear shapes that change the parameters between the allowable intervals (α _min , α _max ) and (β _min , β _max ). It is a function. As a result of the above equations, the simulation is responsive to both audio and motion events in the input video stream.

  Those skilled in the art of simulation and representation techniques, including but not limited to computer animation, may implement other embodiments to generate the surround field, and such implementation is within the scope of the present invention. Let's understand that it is included.

[Still scene]
Those skilled in the art will appreciate that any static scene, such as nature, countryside, urban, indoor, outdoor, surreal, fantasy, etc., can be used in a surround field. Three-dimensional models of scenes such as forests, sky, and deserts are well suited for surround video of still scenes that can be controlled by light sources from the input stream.

  Consider the three-dimensional surround background 1830 depicted in FIG. In the depicted example, the surround field 1830 is a terrain background scene and lighting is associated with the light source of the input video stream 1810. In order to model the illumination of the surround field 1830 in a three-dimensional environment, cues associated with the input video 1810 such as the location, brightness, and color of the light source (in this case the sun) can be used. As the sun rises through different video frames, the surround field 1830 can change color and lighting, including shades and shadows, in response to a three-dimensional modeling of the sunrise in the input stream 1810. As depicted in FIGS. 18A and 18B, as the lighting conditions of the input stream change, those changes are modeled in a three-dimensional environment and the surround field terrain responds to the changing conditions.

[Dynamic scene]
As described above, generating a surround video that moves in response to the input video can create a strong sense of immersion. To achieve this effect in embodiments, the background portion can be numerically simulated or animated using physical laws. Mathematical equations or models can be used to improve the practical appearance of the surround field. In an embodiment, by setting initial and boundary conditions and using physics-based animation, the control signal associated with the input stream is applied to the model and used to generate the interaction of the elements in the surround field. be able to. Physics-based animation includes numerical simulations, or well-known mathematical relationships such as laws of motion, fluid dynamics, etc. can be applied. Such and other methods are known to those skilled in the art of computer animation and are within the scope of the present invention.

  Using physics-based modeling, surround simulation can be driven with control signals derived from the input stream to obtain realistic surround field interaction. For example, a motion vector from the input video can be used to create an external wind that affects the simulated state of the surround field. As another illustrative example, sudden sounds in an audio track can be used to create ripples in a water simulation or move elements in a surround field in response to an audio cue.

  Consider the images depicted in FIGS. 19A-19D as examples. Input stream 1910 includes a light source 1920. Control signals extracted from the stream, such as the position, brightness, and color of the light source 1920, can be used to illuminate the 3D surround field element. For example, light 1925 reflected on the surface of water is noted, which represents the effect when light source 1920 is in a three-dimensional surround environment.

  It is noted that the surround field modeling may also include providing continuity between the input stream 1910 and the surround field 1930. For example, as depicted in FIG. 19B, when the light source 1920 is out of the frame of the input stream 1910, the light source 1920B becomes an element in the surround field 1930 and moves along the same motion trajectory as it was in the input video 1910. Can continue.

  In the depicted embodiment, the sudden explosive event that occurred in FIG. 19C shows how the surround field response can be modeled. Both the background and water colors can change in relation to one or more colors in the explosion shown in the video stream 1910. Furthermore, the water surface is also disturbed in connection with the explosive event depicted in FIG. 19C. It is noted that one cue, such as an explosion, can affect one or more modes in the Sarand field, in this case affecting the color and movement of the water. The actual movement of the water is determined by cues from the input stream, such as the intensity and position of the explosion, and physics-based animations, including “Practical Animation of Liquids”, Proceedings including, but not limited to, methods disclosed by Nick Foster and Ronald Fedkiw in of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, page 2330, August 2001. This document is incorporated herein by reference in its entirety. FIG. 19D further shows the surround field color changing as the explosion in the input stream subsides and the water settles.

[Surround expression]
Representing a high-resolution surround video field in real time may use computing power intensively. In an embodiment, a hybrid representation system can be used to reduce the amount of computation. In one embodiment, the hybrid representation system may use an image-based approach for the static part of the scene and a light transport-based approach for the dynamic part of the scene. The image-based method can usually perform high-speed processing using precalculated data such as a reference image. In one embodiment, the amount of computation required can be reduced by using produced content or images, such as an actual sequence of natural phenomena.

[Non-realistic surround]
It should be noted that in addition to realistic 3D surround field modeling, other surround fields may be drawn, including but not limited to unrealistic surround fields. In one embodiment, the non-photorealistic surround background can be synthesized directly from control signals derived from the input stream. For example, the surround background 2030 can be synthesized using the colors from the input scene 2010 shown in FIG. In this example, the background color of each row is obtained by computing the median color value along the input scene. A person skilled in the art may display a variety of unreal / non-photorealistic surround fields, including but not limited to creating action lines, movement or color highlights, and cartoon environments. Let's understand that. Further, as shown in FIG. 20, it is noted that the input stream is understood to include images in addition to video data.

  Those skilled in the art will appreciate that various types and styles of surround fields can be represented and are within the scope of the present invention. Those skilled in the art will understand that any particular surround field, or how to get a queue for an input stream, or how to model or act on a surround field is not essential to the present invention. In addition, an element of a surround field means a portion of the surround field or pixel, a collection of pixels, and any other image or object, including but not limited to a drawn image or group of objects. It is understood that

[Use of one or more empty display areas]
As described above, video display and surround visual fields in embodiments of the invention are displayed within the boundaries of conventional display devices such as television receivers, computer monitors, laptop computers, portable devices, gaming devices, etc. Can do.

  For example, display devices such as projectors, LCD panels, monitors, television receivers, etc. do not necessarily use all of their display capabilities. FIGS. 21A and 21B show two examples of empty display areas such as unused pixels existing when presenting content from the input stream 2110. FIG. 21A shows an input stream 2110A in letterbox format on the standard display 2100A. Indicates. Since the aspect ratio of the video content is different from the aspect ratio of the display, there are unused display areas 2130A above and below the display 2100A. FIG. 21B shows an image 2100B displayed on the wall by a projector (not shown), for example. As shown in FIG. 21B, an unused area 2130B is created around the main display area 2110B by a normal operation such as keystone effect (key-stoning) and zooming. As described above, one aspect of the present invention relates to using this unused or empty display area 2130.

  The present invention creates an immersive effect by utilizing the empty display area 2100 in the main display. The embodiment of the present invention can use all or part of the vacant display area unless used. In the embodiment, a real-time interactive edge can be displayed in the empty display area.

  In the embodiment, the edge to be displayed in the empty display area can be synthesized using the texture synthesis algorithm. The texture synthesis algorithm used is "Image quilting for texture synthesis and transfer", Proceedings of ACM SIGGRAPH 2001, Computer Graphics Proceedings, Annual Conference Series, page 341346, August 2001. Explained by Alexei A. Efros and William T. Freeman and “Graphcut textures: Image and video synthesis using graph cuts”, ACM Transactions on Graphics, 22 (3): 277286, July 2003, including but not limited to those described by Vivek Kwatra, Arno Schodl, Irfan Essa, Greg Turk, and Aaron Bobick, each of which is incorporated herein by reference. Incorporated into the book as a whole. In an embodiment, the synthesized edges can use color and edge information from the input video stream to guide the composition process. Furthermore, the synthesized texture can be animated to respond to 2D motion vectors from the input stream, which is referred to as “Texture optimization for example-based synthesis”, Similar to the approach described by Vivek Kwatra, Irfan Essa, Aaron Bobick, and Nipun Kwatra in Proceedings of ACM SIGGRAPH 2005, which is incorporated herein by reference in its entirety. Other algorithms known to those skilled in the art can also be used.

  In order to improve real-time performance, another embodiment may involve compositing a bordered spatially expanded image for frames of the input video stream. Including but not limited to the above techniques, computer graphics techniques can be used to create immersive edges that respond in real time to the input video stream.

  In one embodiment, one aspect of utilizing an empty display area around an input frame may involve displaying a background plane illuminated with a virtual light source. In one embodiment, the colors of these virtual light sources can be adapted to match one or more colors of the input stream. In one embodiment, the light source can match one or more dominant colors of the input video stream.

  As an example, consider a bump map background plate 2230 that is illuminated by the four light sources 2200x-1 through 2200x-4 depicted in FIG. In one embodiment, the illumination at each point on the plane 2230 can be affected by a texture map and a normal map. A normal map is a two-dimensional height field that is used to perturb surface normals, which affects how light reflects off the surface. Bump maps are often used in computer games to create wrinkles and depressions on the surface without the need for a true 3D model. In the illustrated embodiment, a four-point light source set 2200x-1 to 2200x-4 is used. In one embodiment, the position of the light source can correspond to four corners of the input video. It is noted that neither the number or the configuration of the light sources is critical to the present invention.

  In embodiments, the appearance of the background plate can be affected by one or more light sources. In the example shown, the background plate reflects light from the light source when the light source is brought close to it. For example, in 2200A, the light sources 2200A-1 to 2200A-4 are separated from the plate 2230. Accordingly, the light sources 2200A-1 to 2200A-4 are smaller-point light sources and appear to be limited in brightness. When the light source is virtually brought close to the plate 2230, it is illuminated more brightly. The light pattern changes, the bump mapping causes shadows to appear in areas of discontinuous depth (eg, near the continental edges), the map color can be affected, and the light sources 2200x-1 to 2200x-4 are It is noted that it can be moved independently. In an embodiment, the color of the light source may be adjusted to relate to the color of the input stream.

  The colors of the lights 2200x-1 to 2200x-4 can be obtained by sampling a part near the corner of the input image and calculating the median color. In an embodiment, color changes can be determined using simple heuristics. In other embodiments, including but not limited to mean shift, more advanced sampling techniques can be used to assign the colors of light sources 2200x-1 to 2200x-4.

  In one embodiment, the present invention can implement diffuse and specular lighting in addition to self-shadowing and bump mapping to synthesize a background image. The background image in FIG. 22 shows an example of a surround visual field synthesized using diffusion and specular illumination in addition to self-occlusion and bump mapping.

  FIG. 23 shows the surround visual field edge of the bump map resulting from illumination with a point light source for the input image. The display 2300A includes an input stream image 2310 that draws a natural scene and an empty display area 2320. A surround visual field 2330 is generated in the empty display area 2320 in order to improve the immersive effect by using the texture background of the bump map illuminated with light that is bright and colored from a part of the input stream image 2310. Can be presented. Control signals or cues can be extracted from the input image 2310 to improve the surround visual field by associating color and / or brightness with portions of the input stream image 2310. For example, the area of the surround visual field 2330A that is close to the bright portion of the image 2310A can be related in color and brightness. The self-occluded bump map background, as shown in FIG. 23, significantly improves the immersive feel of the display 2300B because light and shadows expand the field of view and respond dynamically to the input video stream. The displayed images were generated in real time and implemented on a graphics processor using the Direct3D HLSL shading language.

  In an embodiment, a surround visual field displayed in the empty display area can be used to create mood lighting, which can change or change based on one or more control signals extracted from the input stream. it can. Alternatively, the surround visual field displayed in the empty display area can have a special edge created or generated. For example, the edges may include logos, text, characters, graphics, or other elements. Such elements are associated with the input stream and can be changed or changed based on one or more control signals extracted from the input stream.

  It is noted that displaying the surround visual field using the display empty display area is not limited to the embodiments disclosed herein. The surround visual field displayed within the boundaries of the display device can use any or all of the devices or methods discussed above, including motion, images, patterns, textures, text characters, figures, changing colors, This includes but is not limited to a variety of content or effects such as light sources of varying numbers, 3D synthesis of surround visual fields, and other content and effects. Similarly, any of the embodiments described in connection with the use of the empty display area can be used in surround visual field methods and systems, including those mentioned herein.

  Embodiments of the invention can further relate to a computer product having a computer readable medium having computer code for performing various computer-implemented operations. The media and computer code may be specially designed and constructed for the purposes of the present invention, or may be of a type known and usable by those skilled in the relevant arts. Examples of computer readable media are: magnetic media such as hard disks, floppy disks, and magnetic tape, optical media such as CD-ROMs and holographic devices, magneto-optical media, and application specific integrated circuits. Includes hardware devices specifically configured to store or store and execute program code, such as (ASIC), programmable logic devices (PLD), flash memory devices, and ROM and RAM devices However, it is not limited to these. Examples of computer code include machine code as produced by a compiler, and high-level code that is executed by a computer using an interpreter.

  While the invention is susceptible to various modifications and alternative forms, specific examples thereof have been shown in the drawings and have been described in detail herein. However, it is to be understood that the invention is not limited to the particular portables disclosed, and conversely, the invention includes all modifications, equivalents, and alternatives falling within the spirit and scope of the appended claims.

A surround visual field including a projector according to an embodiment of the invention. A television receiver having a surround visual field according to an embodiment of the invention. A television receiver having a surround visual field from a projector according to an embodiment of the invention. A television receiver having a surround visual field from a projector and reflector according to an embodiment of the invention. 1 is a block diagram of a representative surround visual field controller in which a projected surround visual field relates to movement in display content, according to one embodiment of the invention. FIG. FIG. 3 illustrates a representative optical flow vector between successive frame pairs and pixels in a frame pair, according to one embodiment of the invention. FIG. 3 illustrates a representative optical flow vector between successive frame pairs and pixel blocks within a frame pair, according to one embodiment of the invention. FIG. 3 illustrates a mathematical relationship between two pixels in a global motion model that represents motion between frame pairs, according to one embodiment of the invention. FIG. 4 illustrates an example of a calculated global motion model between frame pairs according to an embodiment of the invention. FIG. 4 is an example of a video frame and overlaid optical flow vector and global motion model, according to one embodiment of the invention. FIG. 4 is a diagram illustrating extrapolation of motion vectors outside a video frame, according to one embodiment of the invention. FIG. 4 is an example of a video frame, a global motion model overlaid on the video frame, and a global motion model extrapolated beyond the boundaries of the video frame, according to one embodiment of the invention. FIG. 4 is a representative surround visual field element modified with respect to motion, according to one embodiment of the invention. FIG. FIG. 3 is a representative surround visual field element modified with respect to a plurality of motion vectors according to an embodiment of the invention. FIG. 1 is a representative surround visual field diagram for motion in a video, according to one embodiment of the invention. FIG. Functional block diagram of a typical surround visual field controller. The projected surround visual field receives one or more inputs, extracts cue or control signals from the inputs, and uses these control signals to generate a surround visual field according to embodiments of the invention. FIG. 4 is a diagram of a method for computing a pan, tilt and zoom component from a motion vector field according to one embodiment of the invention. FIG. 2 is a representative surround visual field associated with an input video stream, according to one embodiment of the invention. FIG. 2 is a representative surround visual field associated with an input video stream, according to one embodiment of the invention. FIG. 4 is a representative surround visual field associated with an input image, according to one embodiment of the invention. The figure of the typical display in which the part of a display area is unused. 1 is a representative surround visual field diagram according to one embodiment of the invention. FIG. FIG. 3 is a representative surround visual field diagram according to an embodiment of the invention.

Explanation of symbols

  500 ... surround visual field controller, 510 ... motion estimation component, 515 ... sparse optical flow estimation component, 525 ... global motion modeling component, 540 ... motion field extrapolation component, 550 ... surround visual Field animation component, 610 ... frame (k-1), 620 ... frame (k), 710 ... frame (k-1), 720 ... frame (k), 810 ... motion model, 910 ... typical global motion Model, 1110 ... Typical global motion model, 1600 ... Surround visual field controller, 1610 ... Cue / control signal extraction component, 1612 ... Video, 1614 ... Audio, 1616 ... Controller, 1618 ... Sensor, 1620 ... Embedded , 1650 ... Surround video Le field generation component / animated component.

Claims (20)

Surround visual field system
A surround visual field controller that obtains at least one control signal associated with an input stream and generates a multi-element 3D surround visual field, at least one element in the 3D surround visual field Is the controller affected by the control signal,
A display device that is communicatively coupled to a surround visual field controller, displays an input stream in a first region, and displays a surround visual field in a second region at least partially surrounding the first region. .   The at least one control signal is obtained from at least one source selected from the group consisting of video data, audio data, game controller, input device, remote control, sensor, and produced cue. The system described in   The system of claim 2, wherein the video data includes information associated with at least one selected from the group consisting of color, position, motion, and content.   The system of claim 2, wherein a motion model is used to approximate motion from the at least one control signal.   The system of claim 4, wherein the motion model is an affine motion model.   The system of claim 1, wherein the display device includes a first display device that displays an input stream in the first region and a second display device that displays a surround visual field in the second region.   The system of claim 2, wherein the surround visual field controller includes a physics-based model and the at least one element is affected in a realistic manner.   The system of claim 2, wherein the surround visual field controller utilizes an image-based method to represent at least a portion of the surround visual field. By generating a surround visual field associated with the input stream,
Extracting a control signal associated with the input stream and generating a surround visual field based on a multi-dimensional 3D environment, wherein at least one element in the 3D environment is affected by the control signal A method involving things. further,
Displaying the input stream in a first region;
The method of claim 9, comprising displaying a surround visual field in the second area at least partially surrounding the first area.   10. The control signal of claim 9, wherein the control signal is extracted from at least one source selected from the group consisting of video data, audio data, game controller, input device, remote control, sensor, and produced cue. Method. Generating a surround visual field based on a three-dimensional environment comprising a plurality of elements, wherein at least one element in the three-dimensional environment is influenced by a control signal;
Generating a motion field approximating motion from the input stream using a motion model;
Breaking the motion field around the center of the image into pan, tilt and zoom components;
Using the pan, tilt, and zoom components to control the direction of movement of the virtual camera in the three-dimensional environment.   The method of claim 12, wherein the motion model is an affine motion model. Generating a surround visual field based on a three-dimensional environment including a plurality of elements, wherein at least one element in the three-dimensional environment is influenced by a control signal;
The method of claim 9, comprising influencing the at least one element in a realistic manner using a physics-based model.   Generating a surround visual field based on a three-dimensional environment comprising a plurality of elements, wherein at least one element in the three-dimensional environment is influenced by a control signal, wherein at least a part of the surround visual field is The method of claim 9, comprising using an image based method to represent.   A computer readable medium carrying one or more instruction sequences that, when executed by one or more processors, cause the one or more processors to perform at least the steps recited in claim 9. Surround visual field controller
A control signal extraction component coupled to receive an input source associated with an input stream displayed in the first region and obtaining at least one control signal from the input source;
Coupling to a control signal extraction component and receiving the at least one control signal to receive a three-dimensional representation of the surround visual field and at least one element in the surround visual field associated with the at least one control signal A surround visual field controller including an effect to generate a surround visual field generation component.   18. The surround visual of claim 17, wherein the input source is at least one selected from the group consisting of video data, audio data, game controller, input device, remote control, sensor, and produced cue. -Field controller.   The controller of claim 17, wherein the surround visual field generation component uses a physics-based model to animate an effect on the at least one element in the surround visual field.   The controller of claim 17, wherein the generated data is used to generate a surround visual field.