WO2000077790A2

WO2000077790A2 - Systems and methods for facilitating the recomposition of data blocks

Info

Publication number: WO2000077790A2
Application number: PCT/US2000/016401
Authority: WO
Inventors: Donald C. Mead
Original assignee: Digital Electronic Cinema, Inc.
Priority date: 1999-06-15
Filing date: 2000-06-15
Publication date: 2000-12-21
Also published as: WO2000077790A3; AU6199000A

Abstract

Methods and systems are disclosed for facilitating the recomposition of audio/video data in audio/video environments such as motion pictures. The system and methods provide object based storage and retrieval of audio/video data blocks and their recompositing into a new audio/video environment, which includes, in part, the stored audio/video data blocks.

Description

SYSTEMS AND METHODS FOR FACILITATING THE RECOMPOSITION OF DATA BLOCKS

BACKGROUND OF THE INVENTION

Related Applications

This application claims priority to U S provisional application serial number 60/139,510, entitled "Super Compression For Motion Picture Images", filed on June 15, 1999, the entire disclosure of which is hereby incorporated by reference

Field of the Invention

The present invention is generally directed to facilitating data recomposition More particularly, the present invention is directed to facilitating the recomposition of audio/video data in audio/video environments such as motion pictures

Description of the Related Art

Recomposition of data into new formats has a wide range of commercial applications, particularly in fields involving production and processing of image and/or audio data, such as television or motion picture industries

Traditionally, most recordings of motion pictures are generally performed on regular motion picture negative reel, resulting in a realization of the motion picture storage on a photochemical medium Generally, the motion picture sequences recorded during a recording period such as a day are physically transported to a developing lab at where they are developed and distributed to the decision makers A decision is typically then made as to whether or not of the one of the on average twenty shots per scene is satisfactory enough for the final picture.

If no satisfactory shot is found, the particular scene or sequence of scenes are generally reenacted and re-recorded Once a satisfactory shot is selected, postproduction begins During post-production the scene sequence judged to be the best is physically spliced into the running spool of film that will be used for the final edition The unused sequences are usually discarded Sometimes a few of these unused sequences, called "outtakes", are saved for future reincorporation into the same or other motion pictures This current process is expensive, time consuming and adds significantly to the time frame from when the on scene recording is completed until the film can be released. One current method to reduce the post production time is by the use of computers Recent advances in computer technologies have allowed for large scale use of digitization in sequences within motion pictures Digitization is used in various forms, such as creating computer-generated version of creatures or scenery that are subsequently incorporated into a motion picture sequence containing film footage of real events Digitization can also be used to alter or remove undesirable portions of a motion picture sequence such as deleting the presence of a modern object, such as a parked car appearing in a movie set in medieval times Digitization thus allows for virtual creation of objects that may otherwise prove impossible or economically unfeasible to create in real life Digitized creations can also be reused and/or reincorporated into other motion pictures with little or no modification to the image, thus reducing the otherwise associated cost and effort of having to recreate a particular scenery or creature every time

Digitization, however, is not without its own drawbacks The digitization process generally results in generation of large volumes of data associated with the created digital images, requiring large storage areas in form of computer readable memory In addition, the number of created digital images themselves are generally exceedingly numerous, requiring a data storage and extraction system for efficient and timelv retπeval of the desired images when called for

The foregoing problems become particularly exacerbated with the growing future trend of having entire motion pictures generated and/or stored in digitized formats The storage requirements for archiving and distribution of digital uncompressed motion pictures are enormous For example, to obtain a digital representation of a 35-mιllιmeter motion picture in image frames of uncompressed, full resolution digital images approximately 45 megabytes of data storage is required for each frame in the 35-mιllιmeter motion picture Thus simple storage of a two hour movie generally requires almost 8 tera-bytes of storage data

The sheer volume of such enormous amounts of data will also result in added delays in locating and retrieving a particularly desired data sequence from the storage for further use or processing This problem becomes even further exacerbated when only portions of an image frame is desired, such as retrieval of data for only a particular creature or person without retrieving the entire frame

What is needed is a technology to allow for efficient storage and retπeval of large volumes of data and for efficient reincorporation of the retrieved data into a new environment OBJECT AND SUMMARY OF THE INVENTION

It is therefore the object of the present invention to address the foregoing deficiencies in the prior art

One object of the present invention is to provide systems and methods to facilitate recompositing of audio/video data blocks The systems and methods includes generation of at least one archiving data library containing a collection of audio/video data blocks, extraction of at least one data block from one of data libraries, and the reincorporation of the extracted data blocks into a new audio/video environment

Accordingly this object is achieved in the present invention by providing systems and methods of object-based storage and retrieval of audio/video data blocks to facilitate their recomposition into a new audio/video environment which includes, in part, the stored audio/video data blocks

In a first embodiment, the present invention is a method for facilitating the recompositing of audio/video data blocks The method includes generating at least one archiving data library containing a collection of audio/video data blocks, extracting at least one data block from one of data libraries, and reincorporating the extracted data blocks into a new audio/video environment

In another embodiment, the present invention is a system to facilitate the recomposition of audio/video data blocks The system includes an archiving data library containing a collection of audio/video data blocks, a data extraction subsystem to extract at least one data block from at least one data library, and an audio/video configuration subsystem to reincorporate the extracted data blocks into a new audio/video environment

In an exemplary embodiment of the present invention, an object-based data technology, such as Motion Picture Expert Group 7 (MPEG 7), is used to identify, compress, store and retrieve the object data blocks for reincorporation into new audio/video environments

This brief summary has been provided so that the nature of the invention may be understood quickly A more complete understanding of the invention can be obtained by reference to the following detailed description of a preferred embodiment thereof and to the drawings BRIEF DESCRIPTION OF THE DRAWINGS

Without limiting the invention in anyway, a number of exemplary embodiments of the invention are described below in association with the accompanying drawings The figures provided herein are referenced to within the specification in order to better illustrate the workings of exemplary embodiments of the claimed invention They are in no respect meant to be an exhaustive list of the various embodiments of the present invention FIG. 1 illustrates the overall architecture of the present invention FIG. 2 is a flow diagram of one subsystem of the present invention as illustrated in FIG. 1 FIG. 2a is an example of data blocks within an image frame

FIG. 3 is a flow diagram of other subsystems of the present invention as illustrated in FIG. 1

FIG. 4 is an example of data blocks as reincorporated by the present invention into the environment of a new image frame FIG. 5 is a diagram of an exemplary version of the present invention for generating lossy outputs for distribution and exhibition of data files

DETAILED DESCRIPTION OF THE INVENTION

The present invention is directed to facilitating the recomposition of audio/video data in audio/video environments, such as motion pictures, by providing object based storage and retrieval of audio/video data blocks to facilitate their recomposition into a new audio/video environment which includes, in part, the stored audio/video data blocks

Referring more particularly to the drawings, FIG. 1 illustrates the overall architecture of the present invention As illustrated in FIG. 1, incoming data such as the motion picture data 10 is entered into the data library subsystem 100, processed into data blocks 20 and archived according to a predetermined routine as described below in FIG. 2. The data extractor subsystem 200 then extracts desired data blocks 20, or any sequence of data blocks 20, for forwarding to the reincorporation subsystem 300 which recomposes the data blocks 20 into a new environment as outputted data 30 Referring to FIG. 2, the workings of the data library subsystem 100 is now disclosed in greater detail. As illustrated in FIG. 2, incoming data such as the motion picture data 10 is entered into the film to digital subsystem 101 The motion picture data 10 includes image data and generally the associated audio data in any stage in the a motion picture recording, such as daily rushes or pre-stored motion picture on a storage medium such as a video tape or any combination of digital or optical mediums. It should be noted that the film to digital subsystem 101 is not essential to the operation of the present invention if the motion picture data is already in digital form. Next, the digitized motion picture data is forwarded to object identification subsystem

102, at where the digitized motion picture is separated into data blocks 20 such as data objects. FIG 2a illustrates in greater detail the process of object selection. As shown in FIG 2a, an exemplary image frame 30 is examined for objects such as animals, 32, 33 and 34. The background 31 can itself be considered an object and stored as one. After each object is identified, it is labeled with an object locator, such as an object identification number, using the meta language techniques of for example, ISO/1EC Standard 15938-2, Multimedia Content Description Interface — Part 2: Visual (MPEG 7). The object identification number facilitates the later access of the desired objects. The identification number can also be used to further distinguish between the objects based on category, such as foreground or background objects and/or moving or stationary objects. As shown in FIG. 2 the process of object identification can involve human supervision, be done automatically or any combination of the two.

A similar process can also be performed for the audio portion of the motion picture data by examination and selection of the digitized sound bytes into audio data blocks, along with assignment of identification number to facilitate later retrieval. In an exemplary embodiment, the audio data block identification numbers are cross-referenced to the associated image data blocks for faster retrieval of both blocks if desired

Returning to FIG 2, the collection of identified objects are compressed in the object compression subsystem 103 by a compression protocol such as lossless, lossy or any combination thereof, for example, MPEG 4 techniques with lossless residual image coding as described in US patent 5,708,473. As illustrated in FIG 2, the object compression subsystem 103 is not essential to operation of the present invention if the motion picture data is not sufficiently large to exceed the available storage memory or to delay subsequent transmission, access or downloading of digitized data blocks 20

The data blocks 20, in their compressed or uncompressed forms are eventually stored in the memory storage subsystem 104, which is a library of all stored objects, along with all of the collateral information for each object. One advantage of the foregoing feature of the present invention over the prior art is that the objects identified and stored in an archiving storage system may be accessed for future use in constructing new image frames, independently of the object's original use. As the incoming flow of motion picture data into the data library subsystem 100 continues the size of storage mediums in storage subsystem 104 continues to grow In an exemplary embodiment of the present invention the data library subsystem 100 is further compressed based on a lossy or lossless compression protocol to reduce the overall size of the data library subsystem 100 In another exemplary embodiment of the present invention, memory storage subsystem 104 is an object based storage subsystem to be particularly suited to storage of data blocks 20

It is noted that the different stages and specific order of the foregoing operations are meant to be exemplary only and any process using one or more of the disclosed routines which yields an output that is consistent with the desired object-based storage of object blocks 20 is anticipated so to be within the scope of the claimed invention

FIG. 3 illustrates the recomposition of the data blocks 20, stored in the data library subsystem 100, into new audio/video environments in form of outputted data 30 As illustrated in FIG. 3 the data extractor subsystem 200 interfaces with the data hbrar\ subsystem 100 to identify and locate a desired data block 20, or any sequence of data blocks 20 that are stored in the data library subsystem 100 In an exemplary embodiment of the present invention, an extractor application is used that is particularly onented toward storage and retrieval of object- based memory data blocks One example of such extractoi application is the Motion Picture Expert Group 7 (MPEG 7) Next, the extracted data blocks are forward to the digital composition subsystem 301 of reincorporation subsystem 300 The digital composition subsystem 301 then recomposes the data blocks 20, into one or a series of data frames containing image frames and associated audio data to produce a master motion picture image data file 302 As the incoming flow of extracted data blocks into the reincorporation subsystem 300 continues the size of master motion picture image data file 302 continues to gi ow In an exemplary embodiment of the present invention, the size of master motion picture image data file 302 is reduced by a compressing the master motion picture image data file 302 using a compression protocol such as lossless, lossy or any combination thereof

As illustrated in FIG. 4 the operations of the digital composition subsystem 301 is disclosed in greater detail in the form of a new audio/video of exemplary image frame 40 The exemplary image frame 40 is composed of the extracted data objects in their compressed or uncompressed form such as animals 32, and 34 from the image frame illustrated in FIG. 2, but can also include additional objects such as human object 35, and new background object 36 In addition, because the extracted data blocks can be used independently of the object's original use in prior image frames, in constructing new image frames the extracted data blocks can be placed at any location within the new audio/video environment of the new image frame In addition, duplicates of extracted information such as animal 32 can be made with ease and likewise placed at any location within the new audio/video environment of the new image frame Other alterations to the extracted data blocks such as partial representation, magnification or reduction in image size, axial rotations, integration with other images in part or in whole, variation in color or any combination of such is anticipated to be within the scope of the present invention. Similarly, audio data blocks can be extracted and reincorporated into a data frame, alone or in association with image data One advantage of the foregoing feature of the present invention over the prior art is the speed and ease with which data, stored as objects, can be retrieved and readily recomposed along with other data to form new audio/video environments, without having to conduct an often costly and time consuming live recording or reenactment of any particular sequence

In an exemplary embodiment of the present invention, the digital composition subsystem 301 may be provided with constaict instructions to automatically produce a composite image frame, or a series of composite image frames along with associated audio data

FIG. 5 illustrates an embodiment of the present invention that provide lossy compression of the motion picture image frames when losslessly compressed, to reduce the size of extracted data from library subsystem 100 and/or master motion picture image data file 302 prior to transmission, resulting in faster transmission of desired data

Referring to FIG. 5, object based data blocks at stage 500 are forwarded to stage 501 at where the lossless variable length code is removed from each data block according to predetermined protocols The resultant data blocks are then compressed in lossy form at stage 502 and reconstructed into the overall data blocks sequences at stage 503 and outputted for transmission at stage 504

In an exemplary embodiment of the present invention, MPEG 4 techniques with lossless residual image coding as described in US patent 5,708,473 may be used to compress the data blocks Other embodiments of the invention will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims

Claims

WHAT IS CLAIMED IS

1 A method for facilitating the recompositing of audio/video data blocks, said method comprising generating at least one archiving data library containing a collection of audio/video data blocks, extracting at least one said data block fi om one said data libraries, and reincorporating said extracted data blocks into a new audio/video environment

2 The method of Claim 1, said generating further comprising digitizing at least one image and/or audio recording, examining said digitized recordings for data blocks and differentiating between said data blocks, compressing each said data block labeling each said compressed data blocks with an identifier based on said differentiating, and stoπng said identified data blocks based on said differentiating into at least one memory media of at least one archiving data library

3 The method of Claim 2, wherein said images are motion picture images being recorded during a live image recording process or pi e-stored on a storing medium

4 The method of Claim 2 wherein said differentiating between said data blocks is based on the stationary status and/or the location of an image portion represented by said data block within an image

5 The method of Claim 2, wherein said audio recording correspond to motion picture images recorded during a live image/audio recording process or pre-stored on a storing medium

6 The method of Claim 2, said compressing further comprising selecting a desired form of compi ession foi each data block wherein said desired form of compression includes loss\ and lossless forms of compression

7. The method of Claim 6, wherein said lossless compression is a residual coding lossless compression.

8. The method of Claim 2, wherein said archiving data library is further compressed based on a lossy or lossless compression protocol

9 The method of Claim 2, wherein each said data block is an audio or a video data object

10 The method of Claim 9, wherein each said video data object contains image data corresponding to at least one image area of variable size in an image frame

1 1 The method of Claim 9, wherein each said audio data object contains audio data for at least one corresponding sound byte

12 The method of Claim 2, wherein said memory media is an object based memory media

13 The method of Claim 2, wherein said storing further comprising storing said identified data blocks together with all collateral information associated with each data block

14 The method of Claim 1, said extracting further comprising accessing said data library, identifying and locating said data blocks in said data library, and retrieving said identified and located data blocks

15. The method of Claim 1, further comprising decompressing said extracted data blocks at any desired stage of said reincorporating into the new audio/video environment

16 The method of Claim 1, wherein said new audio/video environment is a composite image containing a plurality of said extracted data blocks and/or other audio/video data

17. The method of Claim 1, wherein said reincorporating include placing said extracted data block at any location within the new audio/video environment.

18. The method of Claim 1 , wherein said reincorporating include duplicating said extracted data block and placing said duplicated data blocks at any location within the new audio/video environment.

19. The method of Claim 1, further comprising: altering said extracted data blocks prior to and/or after placing said extracted data blocks in new audio/video environments wherein said altering includes partial representation, magnification or reduction in image size, axial rotations, partial or full integration with other images, or variation in color or any combination thereof.

20. The method of Claim 1, wherein said reincorporating include positioning of said extracted data blocks according to at least one construct instruction set comprising of at least one instruction for positioning of said extracted data blocks to in a composite image or a series of composite image frames in the new audio/video environment.

21. The method of Claim 20, wherein said construct instruction set is received from a remote location.

22. A system for facilitating the recompositing of audio/video data blocks, said method comprising: an archiving data library containing a collection of audio/video data blocks; a data extraction subsystem to extract at least one said data block from one said data libraries; and an audio/video configuration subsystem to reincorporate said extracted data blocks into a new audio/video environment.

23. The system of Claim 22, said archiving data library further comprising: a digitization subsystem to digitize at least one image and/or audio recording; a differentiator subsystem to examine said digitized recordings for data blocks and to differentiate between said data blocks; a compression subsystem to compress each said data block an identification subsystem to label each said compressed data blocks with an identifier based on said differentiation of said data blocks, and a storage subsystem to store said identified data blocks based on said differentiation into at least one memory media of at least one archiving data library

24 The system of Claim 23, wherein said images are motion picture images being recorded during a live image recording process or pre-stored on a storing medium

25 The method of Claim 23, wherein said differentiation between said data blocks is based on the stationary status and/or the location of an image portion represented by said data block within an image

26 The system of Claim 23, wherein said audio recording correspond to motion picture images recorded during a live image/audio recording process or pre-stored on a stoπng medium

27 The system of Claim 23, said compression subsystem further comprising a selector subsystem to select a desired form of compression for each said data block wherein said desired form of compression includes a lossy and lossless forms of compression

28 The system of Claim 27, wherein said lossless compression is a residual coding lossless compression

29 The system of Claim 23, wherein said archiving data library is further compressed based on a lossy or lossless compression protocol

30 The system of Claim 23 wherein each said data block is an audio or a video data object

3 1 The system of Claim 32, wherein each said video data object contains image data corresponding to at least one image area of variable size in an image frame

32 The system of Claim 32, wherein each said audio data object contains audio data for at least one corresponding sound byte

33 The system of Claim 23, wherein said memory media is an object based memory media

34 The system of Claim 23, wherein said storage subsystem stores said identified data blocks together with all collateral information associated with each data block

35 The system of Claim 22, said identification subsystem further comprising an accessing subsystem to access said data library a locator subsystem to identify and locate said data blocks in said data library, and a retrieval subsystem to retrieve said identified and located data blocks

36 The system of Claim 22, further comprising a decompression subsystem to decompress said extracted data blocks at any desired stage of said reincorporation into the new audio/video environment

37 The system of Claim 22, wherein said new audio/video environment is a composite image containing a plurality of said extracted data blocks and/or other audio/video data

38 The system of Claim 22, wherein said audio/video configuration subsystem to place said extracted data block at any location within the new audio/video environment

39 The system of Claim 22, wherein said audio/video configuration subsystem to duplicate said extracted data block and to place said duplicated data blocks at any location within the new audio/video environment

40 The system of Claim 22, further comprising an alteration subsystem to alter said extracted data blocks prior to and/or after placing said extracted data blocks in new audio/video environments wherein said alteration includes partial representation magnification or reduction in image size, axial rotations, partial or full integration with other images, or variation in color or any combination thereof

41. The system of Claim 22, wherein said audio/video configuration subsystem to position said extracted data blocks in the new audio/video environment according to at least one construct instruction set comprising of at least one instruction for positioning of said extracted data blocks to in a composite image or a series of composite image frames in the new audio/video environment.

42. The system of Claims 20, wherein said constaict instruction set is received from a remote location.

- ι: