Classifications

• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H1/00—Details of electrophonic musical instruments
  • G10H1/0091—Means for obtaining special acoustic effects
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06F—ELECTRIC DIGITAL DATA PROCESSING
  • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
  • G06F3/16—Sound input; Sound output
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H1/00—Details of electrophonic musical instruments
  • G10H1/0008—Associated control or indicating means
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H2210/00—Aspects or methods of musical processing having intrinsic musical character, i.e. involving musical theory or musical parameters or relying on musical knowledge, as applied in electrophonic musical tools or instruments
  • G10H2210/155—Musical effects
  • G10H2210/265—Acoustic effect simulation, i.e. volume, spatial, resonance or reverberation effects added to a musical sound, usually by appropriate filtering or delays
  • G10H2210/295—Spatial effects, musical uses of multiple audio channels, e.g. stereo
• • G—PHYSICS
  • G10—MUSICAL INSTRUMENTS; ACOUSTICS
  • G10H—ELECTROPHONIC MUSICAL INSTRUMENTS; INSTRUMENTS IN WHICH THE TONES ARE GENERATED BY ELECTROMECHANICAL MEANS OR ELECTRONIC GENERATORS, OR IN WHICH THE TONES ARE SYNTHESISED FROM A DATA STORE
  • G10H2220/00—Input/output interfacing specifically adapted for electrophonic musical tools or instruments
  • G10H2220/091—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith
  • G10H2220/101—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters
  • G10H2220/106—Graphical user interface [GUI] specifically adapted for electrophonic musical instruments, e.g. interactive musical displays, musical instrument icons or menus; Details of user interactions therewith for graphical creation, edition or control of musical data or parameters using icons, e.g. selecting, moving or linking icons, on-screen symbols, screen regions or segments representing musical elements or parameters

Definitions

• the present invention relates to display systems for playing multi-media works, and more particularly, to a sound processing system that alters a sound track in response to the cropping of an image associated with the sound track.
• Multi-media works consisting of still or moving images with narration, background sounds, and background music are becoming common. Such works may be found on the Internet or on CD-ROM. Systems for displaying motion pictures with sound on computers and other data processing systems are also common utilizing programs such as VIDEO FOR WINDOWS to reproduce the work on computers. Furthermore, 3 -dimensional modeling of sound can be specified in VRML 2.0. In a VRML 2.0 compliant browser, the sound generated by the components of a scene are specified by providing separate sound tracks for each sound source together with the location of that sound source in the scene. The sound observed by a listener facing any direction at any position relative to the sound source can then be reproduced by combining the individual sound sources.
• VIDEO FOR WINDOWS does not provide the ability to crop the motion picture image and to display the cropped image on the screen. For that reason, a conventional AVI file, which is motion picture file used by VIDEO FOR WINDOWS, normally does not include data for controlling multiple audio streams in response to the position of a cropping frame in the motion picture image.
• a conventional program such as VIDEO FOR WINDOWS lacks the ability control the audio signals decoded from the multiple audio streams in response to the user defining the position of a cropping frame in the motion picture image.
• VRML 2.0 provides the data needed to generate a sound track corresponding to the point of view of the user thereby creating a 3 -dimensional sound image that can be changed in response to cropping, etc.
• systems implementing VRML 2.0 do not alter the "sound image" in response to changes in the visual image.
• the sound model implemented by VRML 2.0 is customized to implement 3-dimensional sound effects, and is poorly suited for applications that process audio data linked to 2-dimensional images. Therefore, none of the existing programs can automatically control the audio to match the user's definition of a cropping frame in the motion picture image.
• the present invention is a display system for performing a multi-media work that includes image data representing a still or moving image, and sound data associated with the image data.
• the system includes a display for displaying an image derived from the image data, an audio playback system for combining and playing first and second audio tracks linked to the image, and a pointing system for selecting a region of the image on the display in response to commands from a user of the display system.
• the system also includes a playback processor for altering the combination of the first and second audio tracks played by the audio playback system in response to the pointing system selecting a new region of the image.
• the playback processor also alters the display such that the portion of the image selected by the pointing system is centered in the display.
• the first and second audio tracks include sound tracks to be mixed prior to playback.
• the image includes data specifying gains to be used in the mixing for the sound tracks when the selected region of the display is centered at predetermined locations in the image. If the predetermined locations do not include the center of the selected region, the playback system interpolates the data for the predetermined locations to provide the gains to be used in mixing the sound tracks.
• the multi-media work includes data for specifying images at multiple resolutions. In this embodiment, the pointing system further selects one of the resolutions in response to input from the user. The playback processor then alters the combination of the first and second audio tracks played by the audio playback system in response to both the selected region and the selected resolution.
• Figure 1 illustrates a simple multi-media display.
• Figure 2 is a schematic drawing of an image display system according to one embodiment of the present invention.
• Figure 3 is a block diagram of a sound and image processing system according to another embodiment of the present invention.
• Figure 4 illustrates the interpolation of the sound processing parameters stored for selected pixels in an image to obtain new sound processing parameters.
• FIG. 1 illustrates a simple multi-media display.
• the display consists of an image 11 of a piano 15 and a bass 16 and a sound tract of a musical work generated by the two instruments.
• the sound track is played through a stereo sound system consisting of speakers 17 and 18.
• the stereo sound track is constructed from two audio tracks, one for the piano and one for the bass.
• Each audio track has right and left hand components which are mixed to generate the signals sent to speakers 17 and 18.
• the mixing of the signals consistent with image 11 generates an "acoustical image" in which the piano appears to be located closer to speaker 17, and the bass appears to be located closer to speaker 18.
• Many playback systems allow the user to zoom into various portions of the display by defining a cropping frame around the desired portion.
• the cropped image is then re-displayed in its own frame.
• the cropped image is enlarged to fill the original frame.
• prior art systems do not alter the acoustical image to take into account the new visual image.
• the cropped image shown in cropping frame 12 would have an acoustical image in which piano 15 still appears to be located at the same position in the cropped frame as it occupied in the original frame. That is, piano 15 still appears to be closer to speaker 17 even though it is now in the middle of the new frame. This inconsistency in the acoustical and visual images is distracting to human observers.
• the present invention overcomes this problem with prior art displays by altering the acoustical image in response to the cropping of the original image.
• a cropping frame such as frame 14
• the sound tracks are remixed such that the apparent sound sources are likewise shifted in position in the acoustical image.
• the sound of the bass would be moved such that it was equidistant between speakers 17 and 18 when the viewing frame is switched from the original frame 11 to that shown in cropped frame 14.
• FIG 2 is a schematic drawing of an image display system 50 according to one embodiment of the present invention.
• the user specifies a cropping frame using, for example, a pointer 65 applied to image data 57 that is displayed on display 70.
• the cropped image boundary is input via a cropping controller 51 which sends the limits of the new frame to the appropriate cropping routine 52 in the display system.
• the new image boundaries are also sent to a gain controller 53, which controls the mixing of the right and left speaker signal components generated for each audio track.
• the audio tracks are separately processed and then mixed in the playback system 66 via summing amplifiers 58 and 59 to provide the final left and right signals that are sent to the right and left audio channels, 61 and 62, of the stereo system.
• Exemplary audio tracks are shown at 54-56.
• Each audio track includes left and right components whose relative gain is determined by the gain settings applied to a corresponding pair of amplifiers.
• the amplifiers corresponding to audio track 54 are shown at 63 and 64.
• the sound track attributes of each source are specified for each pixel at position (x, y) in the image.
• the information stored for each pixel, P(x,y) could include the left and right channel gains for each audio track in addition to the image pixel value v, i.e.,
• the data from Eq. (1) for the pixel that is now at the center of the display can then be used to recompute the audio attributes by altering the relative mixing of each sound track in accordance with location of the sound source for that sound track within the new frame created by cropping the old frame.
• a multi -resolution image is defined to be an image that can be viewed at two or more different magnifications.
• Such an image may be specified by a zoom setting.
• zoom into the image i.e., increase the magnification
• the user can point to a specific location in the image.
• the display system selects the region centered at the new position at the next highest resolution level to fill the display area.
• the display system crops the next highest resolution image at the boundaries of the display window.
• a zoom operation may alter the effective position of the viewer with respect to the image both in terms of left-right alignment and distance.
• both the volume of the various audio tracks and the relative gams of the right and left channels must be adjusted to provide a realistic sound track when the image is zoomed
• the data needed to re-compute the left-right balance and amplitude for each audio source may be specified by specifying the gains for each of the left-right amplifiers at the various resolutions That is, the attribute P(x, y, r) of the pixels in resolution layer r and position (x, y) is defined to include the channel amplifications to be used when the pixel at (x,y) becomes the center of the scene, I e ,
• the multi-media work includes image data representing a moving image that includes a sequence of frames
• the above described methods may be applied frame by frame by including the sound values for each pixel in each frame of the moving picture so that the audio tracks can be adjusted when that pixel becomes the center of the frame, 1 e , for the pixel at (x,y) in frame f,
• v is the image pixel value for the relevant pixel in the image
• Rl, LI, R2, L2, , Rn, Ln are the left and right channel gains for audio sources 1 to n, respectively Accordingly, the stereo orientation can be changed over time in response to a change m the viewing area
• audio channel amplitudes for the va ⁇ ous resolution layers may be stored for each frame to allow the relative volumes of the audio sources to be adjusted in time with changes in the visual viewing field specified by zooming in or out.
• FIG. 3 is a block diagram of a sound and image processing system 150 according to another embodiment of the present invention.
• elements of system 150 which serve analogous functions to elements shown in Figure 2 have been given reference numerals that differ by 100 from those used for the analogous functioning elements in Figure 2.
• system 150 the user again specifies a region of the image for cropping or zooming.
• the information on the new scene is converted by filter controller 153 into a set of filter coefficients that are applied to the relevant sound tracks by digital filters.
• Exemplary digital filters are shown at 163 and 164. Each digital filter coefficient changes in relation to the (x, y) coordinates of the center of the cropping region, the resolution layer, r, and the moving image frame and position.
• the music tracks in a scene of a concert hall can be altered to include echoes that are altered as the scene is zoomed in or out thereby producing a more realistic sound track.
• the processing can be customized for either binaural recording in which the sound is played back through headphones or transaural playback in which the sound is played back through stereo speakers but not headphones. In either case, the source of the sound is modified to correspond to the correct location in the modified display selected by the user.
• the left and right channel gains of each audio data are given as the P(x, y) elements described above when the coordinate at the center of the cropping region is (x, y) for specific points as shown in Figure 4 at 201-204
• the left and right channel gains for a cropped frame having a center as shown at 205 may be obtained from values stored for points 201-204 by interpolating the values shown for points 201-204.
• the present invention has been described in terms of a display system, it will be obvious to those skilled in the art from the preceding discussion that the present invention may be practiced on any general purpose data processing system equipped to playback a multi-media work. In this case, the present invention can be implement by altering the playback routines to provide the various user input function and mixing functions described above with reference to the display system embodiments of the invention.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Acoustics & Sound (AREA)
• Multimedia (AREA)
• Theoretical Computer Science (AREA)
• General Engineering & Computer Science (AREA)
• General Health & Medical Sciences (AREA)
• Human Computer Interaction (AREA)
• Audiology, Speech & Language Pathology (AREA)
• General Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• Television Signal Processing For Recording (AREA)
• Stereophonic System (AREA)
• Controls And Circuits For Display Device (AREA)
• User Interface Of Digital Computer (AREA)
• Digital Computer Display Output (AREA)

Applications Claiming Priority (3)

Publications (2)

Family

ID=16701687

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (9)

Family Cites Families (4)

• 1997
  • 1997-08-12 JP JP21728197A patent/JPH1175151A/ja active Pending
• 1998
  • 1998-08-11 CN CN98808045.1A patent/CN1126026C/zh not_active Expired - Fee Related
  • 1998-08-11 EP EP98939330A patent/EP1002266B1/de not_active Expired - Lifetime
  • 1998-08-11 DE DE69836742T patent/DE69836742T2/de not_active Expired - Lifetime
  • 1998-08-11 WO PCT/US1998/016636 patent/WO1999008180A1/en active IP Right Grant
  • 1998-08-11 KR KR1020007001369A patent/KR20010022769A/ko not_active Application Discontinuation

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events