JP2008048447A - Time and resolution layer structure to apply encryption and watermark processing thereto in next generation television - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a method and apparatus for performing encryption and watermark processing using temporal and resolution structure layering of a compressed image frame. <P>SOLUTION: The present invention relates to a method for encrypting a data stream of encoded and compressed video information to a basic layer and at least one enhancement layer and for performing watermark processing on the same data stream, including the steps of: selecting an encryption algorithm; selecting watermark processing; selecting at least one unit to be encrypted in the basic layer or the enhancement layer; selecting at least one unit to apply watermark processing thereto in the basic layer or the enhancement layer; applying watermark processing to each of the selected units by applying the selected watermark processing; and encrypting each of the selected units by applying the selected encryption algorithm. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to electronic communication systems, and more particularly to advanced electronic television systems that have the temporal and resolution layer structure of compressed image frames and provide the ability to encrypt and watermark.

  Currently, the NTSC standard is used for television transmission in the United States. However, proposals have been made to move from the NTSC standard to the next-generation television standard. For example, the adoption of digital standard definition and next-generation television formats at 24 Hz, 30 Hz, 60 Hz, and 60 Hz interlace rates in the United States has been proposed. It is clear that these rates are intended to continue (and therefore be compatible with) the existing NTSC television display rate of 60 Hz (or 59.94 Hz). Also, when displaying a movie with a time rate of 24 frames per second (fps), it is also clear that “3-2 pulldown” is intended for display on a 60 Hz display. However, while the above proposal provides multiple selectable formats, each format only encodes and decodes a single resolution and frame rate, respectively. Since the display or operating rates of these formats are not systematically related to each other, conversion from one format to another is difficult.

  Furthermore, this proposal does not provide compatibility between critical computer displays. These proposed image motion rates are based on historical rates dating back to the beginning of this century. If it is “blank”, these rates will not be selected. In the computer industry that has used arbitrary rates for displays over the past decade, rates in the 70 to 80 Hz range have proven to be optimal, with 72 and 75 Hz being the most common rates. Unfortunately, the proposed rates 30 and 60 Hz lack useful interoperability with 72 or 75 Hz and have poor temporal performance.

  In addition, the claim that it is necessary to have a resolution of about 1000 lines at a high frame rate requires frame interlacing, but based on that idea, it is possible to use 18 ~ One skilled in the art also points out that it is impossible to compress these images within 19 mbit / s.

  It is highly desirable that a single signal format that includes all of the desired standards and high definition resolutions can be employed. However, to achieve it within the bandwidth constraints of conventional 6 MHz broadcast television channels requires both frame rate (temporal) and resolution (spatial) compression (ie, “scalability”). One method specifically intended to provide such scalability is the MPEG-2 standard. Unfortunately, the temporal and spatial scalability features specified in the MPEG-2 standard are not efficient enough to meet the needs of next-generation television for the United States. Thus, proposals for next generation television for the United States are based on the premise that temporal (frame rate) and spatial (resolution) layer structures are inadequate and therefore a separate format is required.

  In addition to the above issues, the inventor has identified the need to protect and manage the use of valuable copyrighted audio and video media such as digital movies. The survival of all movie data delivery technologies depends on the ability to protect and manage their use. As the quality of digital compressed movie masters approaches the quality of original works, the need for protection and management techniques becomes a decisive requirement.

  When working on a system architecture for digital content protection and management, it would be beneficial to employ a variety of tools and techniques that can be applied in a modular and flexible manner. Most commercial encryption systems are ultimately damaged. Therefore, any protection system needs to be built sufficiently flexible so that it can adapt and strengthen itself if it is damaged. It is also beneficial to give each copy an informational clue by watermarking the symbol and / or serial number information in order to pinpoint exactly how the source and precautions (security) were compromised.

  Digital movie distribution to movie theaters is being realized. Copying expensive new movies has long been the target of theft or copying of today's film prints. Digital media such as DVDs have attempted poor encryption and authentication schemes (such as DIVX). For the charging of premium wired channels and pay-per-view programs and movies, analog wired scramblers (frequency band converters that mix and disrupt TV signals to prevent eavesdropping) have been used from the beginning. However, these inadequate scramblers have been widely damaged.

  One reason that digital and analog video systems have allowed such poor security systems is the value of secondary video releases and the relatively small proportion of piracy losses in the market. However, a robust security system is a necessity for digital open movies, expensive live events, and high resolution image distribution to homes and offices (in HDTV format).

  The present invention overcomes the above and other problems of current digital content protection systems.

SUMMARY The present invention provides an image compression method and apparatus that can clearly achieve a resolution superior to 1000 line resolution with high frame rate and high quality. The present invention also achieves both temporal and resolution scalability at this resolution and at high frame rates within the bandwidth available on conventional television broadcast channels. The technique of the present invention efficiently achieves more than twice the compression rate proposed for next-generation television, while providing flexible encryption and watermarking techniques.

It is preferred to capture the image material at the initial or main framing rate of 72 fps. An MPEG-2 data stream containing the following is then generated:
(1) A base layer that is preferably encoded using only MPEG-2 P-frames and includes a low resolution (eg, 1024 × 512 pixels), low frame rate (24 or 36 Hz) bitstream.
(2) An optional basic resolution time enhancement layer that is encoded using only MPEG-2 B frames and includes a low resolution (eg, 1024 × 512 pixels), high frame rate (72 Hz) bitstream.
(3) An optional base time high-resolution enhancement layer, preferably encoded using only MPEG-2 P-frames, including a high-resolution (eg, 2kx1k pixels), low-frame-rate (24 or 36 Hz) bitstream.
(4) An optional high-resolution time enhancement layer that is encoded using only MPEG-2 B-frames and includes a high-resolution (eg, 2k × 1k pixel), high-frame-rate (72 Hz) bitstream.

  The present invention provides several key technical characteristics that allow substantial improvements over current proposals, including the following: numerous resolutions and frame rates, a single layer Replacing interleaved resolutions and frame rates; no interlacing is required to achieve better than 1000 lines for 2 megapixel images at a high frame rate (72 Hz) in a 6 MHz television channel Compatible with computer displays that use the main framing rate of 72 fps; and all available bits can be assigned to a low resolution base layer when “stressful” image material appears From the proposal of the current unlayered format for next-generation television High robustness.

  The disclosed layered compression technique allows a form of modular decomposition of images. This modularity not only allows for scalable decoding and excellent stress resiliency, but has additional advantages. Modularity can also be developed as a structure that supports flexible encryption and watermarking techniques. The encryption function is to restrict viewing / showing, copying, or otherwise using an audio / video show unless one or more appropriate keys are applied to the authenticated decryption system. The watermarking function is able to track lost or stolen copies to the source, determine the nature of the theft method, improve the security of the system, and identify the people involved in the theft.

  By using layered compression, a base layer and various internal components of the base layer (such as I-frames and their DC coefficients, or P-frame motion vectors) can be used to compress compressed layered movie streams. Can be encrypted. By using such a layered subset of bits, the entire picture stream can be made unrecognizable (unless decrypted) simply by encrypting a small portion of the bits of the entire picture stream. In addition, various encryption algorithms and strengths can be applied to various parts of the layered stream including enhancement layers (viewable as a premium quality service and specially encrypted). It is also possible to change the encryption algorithm or key for each slice boundary to make the encryption and the image stream more entangled.

  The layered compression structure of the present invention can also be used for watermark processing. The goal of watermarking is to be able to identify with high reliability by detection and still be essentially invisible to the eye. For example, the low order bits in the DC coefficient in an I frame may still be invisible to the eye, but can still be used to uniquely identify a particular picture stream with a watermark. The enhancement layers can have their own unique identification watermark structure.

  The details of one or more embodiments of the invention are set forth in the accompanying drawings and the description below. Other features, objects, and advantages of the invention will be apparent from the description and drawings, and from the claims.

  Throughout this description, the following preferred embodiments and examples should be construed as exemplary rather than limiting of the present invention.

Goals of Temporal and Resolution Layered Time Rate Family After reviewing the problems of the prior art, in implementing the present invention, the goals are defined as follows to clarify the temporal features of future digital television systems: To:
• Optimal display of high resolution relics, 24 frames per second film • Smooth motion capture for fast moving image types, such as sports • On existing analog NTSC displays and computer compatible displays operating at 72 or 75 Hz Smooth display of sports and similar image movements • Appropriate and more efficient motion capture of images that do not move so fast, such as news and life dramas • All new digital images existing via a converter box Appropriate display on NTSC displays • Display all new digital images in high quality on computer-compatible displays • When 60Hz digital standard or high resolution displays appear on the market, Display a high-quality

Since 60 Hz and 72/75 Hz displays are inherently incompatible with any rate other than the movie rate of 24 Hz, it would be best to exclude either 72/75 or 60 from the display rate. 72 or 75 Hz is N.P. I. I. (National
(Information Infrastructure: National Information Infrastructure) and the rate required for computer applications, it would be most future-oriented to exclude the 60 Hz rate as an obsolete rate. However, there are many competing interests within the broadcast and television equipment industry, and there is also a strong demand that any new digital television infrastructure should be based on 60 Hz (and 30 Hz). This has sparked intense debate among the television, broadcast and computer industries.

  In addition, there is a stake in the broadcast and television industry that claims an interlaced 60 Hz format, further expanding the gap with computer display requirements. When a digital television system is applied to a computer or the like, non-interlaced display is required, so that a de-interlacer is required to display an interlaced signal. Since a deinterlacer is required for all such receivers, there is considerable debate about the cost and quality of the deinterlacer. In addition to deinterlacing, frame rate conversion also has an impact on cost and quality. For example, despite the cost of the converter between NTSC and PAL remains very high, its conversion capability is still unreliable for many common scenes. Since the discussion on interlacing is a complex and cumbersome issue, and to address the time rate issues and challenges, the present invention will be described in the context of a digital television standard without interlacing.

Choosing the optimal time rate Beat issues On a 72 or 75 Hz display, an optimal display is obtained when a camera or simulated image with an operating rate equal to its display rate (72 or 75 Hz, respectively) is formed. And vice versa. Similarly, on a 60 Hz display, optimal motion fidelity can be obtained from a 60 Hz camera or simulated image. When a 72 Hz or 75 Hz generation rate is used with a 60 Hz display, a 12 Hz or 15 Hz beat frequency is generated, respectively. Although this beat can be removed through motion analysis, motion analysis is expensive and inaccurate, often causing visible unnatural results (visible artifacts) and temporal aliasing. Without motion analysis, the beat frequency dominates the perceived display rate, and 12 or 15 Hz beats appear, resulting in motion (motion) that is less accurate than 24 Hz. Thus, 24 Hz forms a natural time common denominator between 60 and 72 Hz. Although 75 Hz produces a slightly higher 15 Hz beat than 60 Hz, the operation is still not as smooth as 24 Hz, and there is no integer relationship between 75 Hz and 24 Hz unless the 24 Hz rate is increased to 25 Hz. (In 50 Hz countries in Europe, movies are often shown 4% faster at 25 Hz, so that the film can be displayed on a 75 Hz display.)

  Without motion analysis at each receiver, 60 Hz operation on a 72 or 75 Hz display and 75 or 72 Hz operation on a 60 Hz display may not be as smooth as a 24 Hz image. Thus, either 72/75 Hz or 60 Hz operation is unsuitable for reaching a hybrid display population that includes both 72 or 75 Hz displays and 60 Hz displays.

  3-2 Pulldown An additional challenge in selecting the optimal frame rate arises from using “3-2 pulldown”, which involves video effects during the telecine (film to video) conversion process. During such conversion, the 3-2 pulldown pattern repeats the first frame (or field) three times, the next frame twice, the next frame three times, and the next frame twice. In this way, a 24 fps film is displayed on a television at 60 Hz (actually 59.94 Hz for NTSC color). In other words, 60 images per second are given by displaying 12 pairs of frames, each consisting of 2 frames on a 1 second film, 5 times each. A 3-2 pull-down pattern is shown in FIG.

  According to one estimate, more than half of the total film on the video has been adjusted to 24 fps film at a video field rate of 59.94 Hz in a significant portion of it. Such adjustments include “pan and scan”, color correction, and title scrolling. In addition, many films are timed by deleting frames or cropping the beginning and end of a scene and are adapted to fit within a given broadcast schedule. These operations may make it impossible to reverse the 3-2 pulldown process because both 59.94 Hz and 24 Hz operations exist. This makes it very difficult to compress films using the MPEG-2 standard. Fortunately, this problem is limited to existing NTSC resolution material as there is no large library of high resolution digital film using 3-2 pulldown.

  Motion blur (motion blurring) To further explore the challenge of finding common time rates higher than 24 Hz, it is useful to mention motion blur in the capture of moving images. The camera sensor and motion picture film are open to sense a moving image for a portion of the duration of each frame. In motion picture cameras and many video cameras, this exposure duration can be adjusted. Since film cameras require film advance time, they are typically limited to be open by about 210 degrees out of 360 degrees, or 58% duty cycle. Video cameras with CCD sensors often require some frame time to “read” an image from the sensor. This can vary between 10% and 50% of the frame time. Depending on the sensor, an electronic shutter must be used to block the light during this readout time. Thus, the “duty cycle” of a CCD sensor typically varies between 50 and 90%, and some cameras can be adjusted. The optical shutter can sometimes be adjusted to further reduce its duty cycle, if desired. However, for both film and video, the most common sensor duty cycle duration is 50%.

Preferred Rate With this issue in mind, it is possible to consider using only a few of the frames from an image sequence captured at 60, 72, or 75 Hz. By using one frame in groups of 2, 3, 4, etc., the subrates shown in Table 1 are derived.

  The rate of 15 Hz is an integrated rate between 60 and 75 Hz. The rate of 12 Hz is an integrated rate between 60 and 72 Hz. However, if rates higher than 24 Hz are desired, these rates are eliminated. Although 24 Hz is not common, the use of 3-2 pull-down has been accepted by the industry for display on a 60 Hz display. Thus, the only candidate rates are 30, 36, and 37.5 Hz. 30 Hz generates a 7.5 Hz beat for 75 Hz and a 6 Hz beat for 72 Hz, so it is not suitable as a candidate.

  The operating rates of 36 and 37.5 Hz are the most important candidates for providing smoother motion than 24 Hz material when displayed on 60 and 72/75 Hz displays. Both of these rates are smooth about 50% faster than 24 Hz. The 37.5 Hz rate is not suitable for use at either 60 or 72 Hz and must be eliminated, leaving only 36 Hz as having the desired time rate characteristics. (A 37.5 Hz operating rate could be used if the 60 Hz display rate of the television can be moved 4% to 62.5 Hz. 62.5 Hz is unlikely due to the stakes behind 60 Hz. There are even people who are proposing a very outdated 59.94 Hz for new television systems, however, if such changes are made, other aspects of the invention could be applied to the 37.5 Hz rate. )

  The 24, 36, 60, and 72 Hz rates remain as candidates for the time rate family. The 72 and 60 Hz rates cannot be used as delivery rates. This is because, as described above, the operation is not smooth when converting between these two rates as compared with the case where 24 Hz is used as the distribution rate. As a premise, we are looking for a rate faster than 24 Hz. As such, 36 Hz is the leading candidate for integrating motion capture and image delivery for masters used in 60 and 72/75 Hz displays.

As mentioned above, the 3-2 pulldown pattern for 24Hz material is 3 times for the first frame (or field), 2 times for the next frame, 3 times for the next frame, and 2 times for the next frame. Repeat as follows. When using 36 Hz, each pattern would be best repeated with a pattern of 2-1-2. This can be seen graphically from Table 2 and FIG.

  This relationship between 36 Hz and 60 Hz is only true for 36 Hz material. 60 Hz material can be “stored” at 36 Hz if interlaced, but 36 Hz is not properly generated from 60 Hz without motion analysis and reconstruction. However, in searching for a new rate for motion capture, 36 Hz provides somewhat smoother operation on 60 Hz than 24 Hz, and provides much better image motion smoothness on a 72 Hz display. Therefore, 36 Hz is an optimal rate that integrates the motion capture and image delivery rates for masters used in 60 and 72/75 Hz displays, and moves more smoothly than 24 Hz material when displayed on such displays. Bring.

  Although 36 Hz meets the above target, it is not the only suitable capture rate. Since 36 Hz cannot simply be extracted from 60 Hz, 60 Hz does not provide a suitable rate for capture. However, 72 Hz can be used for capture by using every other frame as the basis for 36 Hz delivery. The motion blur generated by using every other frame of 72 Hz material would be half of the motion blur for a 36 Hz capture. When we look at the appearance of motion blur in every third frame from 72 Hz, we don't like 24 Hz intermittent flash. However, using every other frame from 72 Hz for a 36 Hz display is not annoying to the eye compared to the original 36 Hz capture.

  Thus, 36 Hz can provide very smooth operation on a 72 Hz display by capturing at 72 Hz, while using alternating frames of material originally captured at 72 Hz to achieve a 36 Hz delivery rate, and 2- Extracting a 60 Hz image using 1-2 pulldown provides motion on a 60 Hz display that is superior to 24 Hz material.

Table 3 summarizes the preferred optimal time rates for capture and delivery according to the present invention.

It is also worth mentioning that this approach, which utilizes alternating frames from a 72 Hz camera to achieve a 36 Hz delivery rate, can also benefit from an increased motion blur duty cycle. The normal 50% duty cycle at 72Hz has been proven acceptable, although it yields a 25% duty cycle at 36Hz and shows a significant improvement over 24Hz on 60Hz and 72H displays. . However, if the duty cycle is increased to the 75-90% range, the 36 Hz sample will begin to approach the more common 50% duty cycle. Increasing the duty rate, for example, is not recorded time (blanking
“Auxiliary memory (backing) with short time and high duty cycle
store) "may be achieved by using a CCD design. Other methods including dual CCD multiple designs may be used.

Partially modified MPEG-2 compression For efficient storage and distribution, digital source material having a preferred time rate of 36 Hz should be compressed. The preferred compression format for the present invention is achieved using a novel variation of the MPEG-2 standard.

  MPEG-2 Basics MPEG-2 is an international video compression standard that defines video syntax that provides an efficient way to represent image sequences in a more compact coded data format. The language of the coded (encoded) bits is “syntax”. For example, an entire block of 64 samples can be expressed with several tokens. MPEG also describes a decoding (reconstruction) process in which coded bits are mapped from a compact representation to an original “raw” format image sequence. For example, a flag in the encoded bitstream indicates whether subsequent bits should be decoded (decoded) with a discrete cosine transform (DCT) algorithm or a prediction algorithm. An algorithm including a decoding process is defined by semantics defined by MPEG. This syntax can be applied to take advantage of video common features such as spatial redundancy, temporal redundancy, constant behavior, spatial masking, etc. MPEG-2 actually defines a program language as well as a data format. The MPEG-2 decoder must be able to parse and decode the incoming data stream, but as long as the data stream conforms to the MPEG-2 syntax, it can use a wide range of possible data structures and compression techniques. The present invention takes advantage of the flexibility of MPEG-2 by devising new means and methods for temporal and resolution scaling using the MPEG-2 standard.

  MPEG-2 uses a compression method within and between frames. In most video scenes, the background is relatively stable, while actions occur in the foreground. The background may move, but most of the scene is redundant. MPEG-2 starts compression by creating a reference frame called an I (Intra) frame. I frames are compressed without reference to other frames and thus contain video information for the entire frame. I-frames provide an entry point into the data bitstream for random access, but can be compressed only slightly. Typically, data representing an I frame is placed every 10 to 15 frames in the bitstream. After that, since only a small part of the frame that enters between the reference I frames is different from the I frames on both sides, only the difference is captured (captured), and compressed and stored. Two types of frames are used for the difference, which are a P (Predicted) frame and a B (Bi-directional Interpolated) frame.

  P frames are typically encoded with reference to past frames (either I frames or previous P frames) and are generally used as references for future P frames. The amount of compression that the P frame has is quite high. B-frame pictures have the highest amount of compression, but generally require both past and future references to be encoded. Bi-directional frames are not used as reference frames.

  Macroblocks within P frames may also be encoded individually using intraframe coding. Macroblocks within B frames can also be individually encoded using intra-frame coding, forward prediction coding, backward prediction coding, both forward and backward, ie bi-directional interpolation prediction coding. A macroblock is a group of 16x16 pixels consisting of four 8x8 DCT blocks, with one motion vector for the P frame and one or two motion vectors for the B frame.

  After encoding, the MPEG data bitstream contains a sequence of I, P and B frames. A sequence may consist of almost any pattern of I, P, and B frames (with a few semantic limitations that are not important for their placement). However, it is common in industry practice to have a fixed pattern (eg, IBBPBBPBBPBBPBB).

As an important part of the present invention, a base layer, at least one optional temporal enhancement layer and an optional resolution enhancement layer
MPEG-2 data stream including the layer) is created. Each of these layers will be described in detail later.

Time Scalability Base Layer The base layer is used for the transmission of 36 Hz source material. In the preferred embodiment, one of two types of MPEG-2 frame sequences, IBPBPBP or IPPPPPP, may be used for the base layer. The latter pattern is most preferred because it only requires the decoder to decode P-frames and can reduce the memory bandwidth required when a 24 Hz movie is decoded without using B-frames.

  72 Hz Time Extension Layer When using MPEG-2 compression, if the spacing between P frames is regular, it is possible to embed the 36 Hz time extension layer as a B frame within the MPEG-2 sequence of the 36 Hz base layer. This can support both 36 Hz and 72 Hz displays in a single data stream. For example, both layers may be decoded to generate a 72 Hz signal for computer monitoring, while only the base layer may be decoded and converted to generate a 60 Hz signal for television.

  In the preferred embodiment, both IPBBBPBBBBPBBBP and IPBPBPBPB MPEG-2 coding patterns are made 36 Hz to 72 Hz by placing every other frame containing only temporally extended B frames in a separate stream. Yes. These coding patterns are shown in FIGS. 2 and 3, respectively. The coding pattern in which the P interval in FIG. 3 is 2 frames is required when a 36 Hz decoder only needs to decode P frames and a 24 Hz movie is decoded without B frames. It has the further advantage that the memory bandwidth can be reduced.

  Experiments using high-resolution images have shown that the interval of P in FIG. 3 is 2 frames is optimal for most types of images. That is, the configuration of FIG. 3 seems to provide an optimal temporal structure that supports both 60 and 72 Hz, and gives excellent results on a modern 72 Hz computer compatible display. This configuration achieves 72 Hz with two digital streams: 36 Hz for the base layer and 36 Hz for the enhancement layer B frame. This is illustrated in FIG. FIG. 4 is a block diagram showing that the 36 Hz base layer MPEG-2 decoder 50 simply decodes the P frame to produce a 36 Hz output, and that output can be easily converted to either a 60 Hz or 72 Hz display. is there. An optional second decoder 52 simply decodes the B frame to produce a second 36 Hz output, and when that output is combined with the 36 Hz output of the base layer decoder 50, a 72 Hz output is obtained. (The combination method will be discussed later). In an alternative embodiment, one high speed MPEG-2 decoder 50 can decode both base layer P frames and enhancement layer B frames.

  Optimal Master Format A considerable number of companies produce MPEG-2 decoding chips that operate at about 11 Mpixels / second. The MPEG-2 standard defines several “profiles” for resolution and frame rate. These profiles are strongly biased towards computer incompatible format parameters such as 60Hz, non-square pixels, and interlace, but many chipmakers appear to be developing decoder chips that operate at "main profile, main level" It is. This profile is defined to have a horizontal resolution of up to 720 pixels, a vertical resolution of up to 576 lines up to 25 Hz, and a vertical resolution of up to 480 lines up to 30 Hz. A wide range of data rates from about 1.5 Mbit / sec to about 10 Mbit / sec is defined. However, an important issue from the chip perspective is the rate at which pixels are decoded. The pixel rate of the main level and main profile is about 10.5 Mpixels / second.

  Despite variations among chip manufacturers, most MPEG-2 decoder chips will actually operate at up to 13 Mpixels / second when given fast support memory. There will also be some decoder chips that are as fast as 20 Mpixels / second or faster. Considering the tendency of CPU chips to improve performance by 50% or more every year at a predetermined cost, it is possible to predict the pixel rate flexibility of the MPEG-2 decoder chip in the near future.

Some desirable resolutions and frame rates and their corresponding pixel rates are shown in Table 4.

  All of these formats are available with an MPEG-2 decoder chip that can generate at least 12.6 Mpixels / second. The highly desirable 640x480 format at 36 Hz can be implemented on almost all current chips because its rate is 11.1 Mpixels / second. Widescreen 1024x512 images can be compressed to 680x512 with a 1.5: 1 compression ratio and can be supported at 36Hz if 12.5M pixels / second can be handled. A highly desirable 1024 × 512 square pixel widescreen template can be realized at 36 Hz when the MPEG-2 decoder chip can handle about 18.9 Mpixels / second. This is more feasible when 24 Hz and 36 Hz material is coded using only P frames and B frames are required in a 72 Hz time enhancement layer decoder. A decoder that uses only P frames requires less memory and memory bandwidth, making it easier to achieve the goal of 19 Mpixels / second.

  The 1024x512 resolution template will most often be used for films with aspect ratios of 2.35: 1 and 1.85: 1 at 24 fps. This material requires only 11.8M pixels / sec and should fit within the limits of most existing main level-main profile decoders.

  All of these formats are shown in FIG. 6 in a “master template” for the base layer at 24 or 36 Hz. The present invention thus provides a unique method of accommodating a wide range of aspect ratios and temporal resolutions compared to the prior art. (Master templates are discussed further below).

  The time extension layer of the B frame that generates 72 Hz uses a chip with twice the pixel rate as defined above, or in parallel so that the second chip can additionally access the decoder memory. Can be decrypted. Under the present invention, there are at least two ways to combine the enhancement and base layer data streams to insert alternating B-frames. First, the combination can be made invisible to the decoder chip using the MPEG-2 transport layer. An MPEG-2 transport packet for two PIDs (program IDs) can be recognized as including a base layer and an enhancement layer, and the contents of those streams both go to a decoder chip with double rate capability. Or it can simply be sent to a properly configured pair of standard rate decoders. Second, it is also possible to use a “data partitioning” function in the MPEG-2 data stream instead of the transport layer of the MPEG-2 system. The data partitioning function marks B frames as belonging to different classifications in the MPEG-2 compressed data stream and is therefore flagged to be ignored by 36 Hz decoders that support only the temporal base layer rate.

  As defined by MPEG-2 video compression, temporal scalability is not as good as the simple B-frame splitting of the present invention. The temporal scalability of MPEG-2 is only referenced in the forward direction from the previous P or B frame, and thus is obtained by the proposed B frame encoding referenced in both the forward and reverse directions. It is not as efficient as possible. Therefore, simply using a B frame as a time enhancement layer provides a simpler and more efficient time scalability compared to the time scalability defined in MPEG-2. Nevertheless, using B frames as a temporal scalability mechanism in this way is perfectly consistent with MPEG-2. The two methods of recognizing these B frames as enhancement layers by data partitioning for B frames or alternating PIDs are also perfectly consistent.

  50/60 Hz time extension layer In addition to or as an alternative to the 72 Hz time extension layer described above (which encodes a 36 Hz signal), a 60 Hz time extension layer (which encodes a 24 Hz signal) can be used in a similar manner. Can be added to the base layer. The 60 Hz time enhancement layer is particularly useful for encoding existing 60 Hz interlaced video material.

  Most of the existing 60 Hz interlaced material is analog, D1 or D2 format NTSC videotape. Japanese HDTV (SMPTE240 / 260M) is also present in a small amount. There are also cameras that operate in this format. Any 60 Hz interlace format may be processed in a known manner where the signal is deinterlaced and frame rate converted. This process includes a technique for understanding very complex images similar to robot vision. Even with very high-performance techniques, temporal aliasing typically causes “misunderstanding” by the algorithm, and sometimes produces artifacts. A typical 50% duty cycle of image capture means that the camera is “not looking” for half the time. “Wagon wheel reverse rotation” in movies is an example of temporal aliasing with temporal undersampling as a common practice. Generally, such artifacts cannot be removed without human-assisted reconstruction. Therefore, there will always be cases that cannot be corrected automatically. However, the behavioral conversions possible with current technology should yield some results on most materials.

  The price of a high-definition camera or tape machine would be equivalent to the cost of such a converter. Therefore, the cost of such conversion is reasonable for a studio with several cameras and tape machines. However, adequately performing such processing is currently exceeding the budget for home and office products. Therefore, the complicated process of removing the interlace and converting the frame rate for the existing material is preferably accomplished in the production studio. This is shown in FIG. FIG. 5 shows a 36 Hz signal (36 Hz base layer only) and a 72 Hz signal (36 Hz base layer + 36 Hz) from camera 60 or other source (such as non-film video tape) 62 including deinterlacer and frame rate conversion functions. It is a block diagram which shows the 60-Hz interlace input to the converter 64 which can output a (time extension layer).

As an alternative to outputting a 72 Hz signal (36 Hz base layer + 36 Hz time extension layer), this conversion process is adapted to generate a second MPEG-2 time extension layer of 24 Hz on the 36 Hz base layer, thereby deinterlacing. But the original 60Hz signal could be reproduced. If similar quantization is used for 60 Hz time enhancement layer B frames, the data rate should be slightly below the 72 Hz time enhancement layer rate due to the small number of B frames.
> 60I → 36 + 36 = 72
> 60I → 36 + 24 = 60
> 72 → 36,72,60
> 50I → 36, 50, 72
> 60 → 24, 36, 72

  The overwhelming majority of materials that benefit the United States are low resolution NTSCs. Currently, most NTSC signals are viewed with substantial deterioration on most home televisions. In addition, viewers have been tolerant of the worsening of time associated with the use of 3-2 pulldown to perform the film on television. Almost all Golden Hour televisions are made on 24 frames per second film. Therefore, only sports, news, and other video original shows need to be processed in this way. Artifacts and losses associated with converting these shows to 36/72 Hz format would be offset by improvements associated with high quality deinterlacing of the signal.

  The motion blur inherent in the 60 Hz (or 59.94 Hz) field should be very similar to the motion blur in the 72 Hz frame. Thus, this approach of providing base and enhancement layers should look similar to the 72 Hz original in terms of motion blur. Therefore, when interlaced 60Hz NTSC material is processed into a 36Hz base layer and 24Hz from the time extension layer is added and displayed at 60Hz, most viewers may notice a slight improvement. You will not notice the difference. However, those who purchase a new 72Hz non-interlaced digital television will notice a small improvement when viewing NTSC, and a significant improvement when viewing new material captured or produced at 72Hz. right. Even when the decoded 36 Hz base layer is displayed on a 72 Hz display, the interlace artifacts will be replaced by a slow frame rate and will look equivalent to a high quality digital NTSC.

  Similar processing can be applied to converting existing 50 Hz PAL material to the second MPEG-2 enhancement layer. PAL videotapes are best decelerated to 48 Hz before such conversion. Raw PAL requires conversion using rates of 50, 36 and 72 Hz, which are unrelated. At present, such a converter unit is profitable only with the source of the broadcast signal, and at present, it is not practical for home and office receivers.

Resolution Scalability In order to achieve higher resolution on the base layer, it is possible to extend the base resolution template using hierarchical resolution scalability using MPEG-2. The use of extensions can achieve 1.5 times and twice the resolution of the base layer. The double resolution may be realized in 2 steps using 3/2 and 4/3, or may be 1 step with a factor of 2. This is shown in FIG.

  The resolution enhancement process can be achieved by generating a resolution enhancement layer as an independent MPEG-2 stream and applying MPEG-2 compression to the enhancement layer. This approach is different from “spatial scalability” defined in MPEG-2 and proven to be extremely inefficient. However, MPEG-2 includes all of the tools for constructing an effective layered resolution to provide spatial scalability. A preferred layered resolution encoding process of the present invention is shown in FIG. A preferred decoding process of the present invention is shown in FIG.

8. Coding of Resolution Layer In FIG. 8, a 2k × 1k original image 80 is filtered by a conventional method so that the resolution of each dimension is halved, and a 1024 × 512 basic layer 81 is generated. Then, the base layer 81 is compressed according to the conventional MPEG-2 algorithm, and an MPEG-2 base layer 82 suitable for transmission is generated. During this compression step, MPEG-2 full motion compensation (full
It is important that MPEG-2 motion compensation) can be used. The same signal is then decompressed using a conventional MPEG-2 algorithm and returned to a 1024 × 512 image 83. The 1024 × 512 image 83 is expanded (eg, by pixel replication or preferably by a better filter such as spline interpolation) to become a 2k × 1k first magnified image 84.

On the other hand, as an optional step, the filtered 1024 × 512 base layer 81 is expanded into a 2k × 1k second enlarged image 85. This 2kx1k second enlarged image 85 is subtracted from the 2kx1k original image 80, and an image representing the top octave of the resolution between the original high-resolution image 80 and the original base layer image 81 is generated. The resulting image is arbitrarily multiplied by a sharpness factor or weight and added to the difference between the 2kx1k original image 80 and the 2kx1k second magnified image 85 to give a 2kx1k center weighted enhancement layer. A source image 86 is generated. This enhancement layer source image 86 is then compressed according to a conventional MPEG-2 algorithm to generate another MPEG-2 resolution enhancement layer 87 suitable for transmission. During this compression step, MPEG-2 full motion compensation (full
It is important that MPEG-2 motion compensation is available.

  Resolution Layer Decoding In FIG. 9, the base layer 82 is decompressed using a conventional MPEG-2 algorithm and returns to a 1024 × 512 image 90. The 1024 × 512 image 90 is expanded into a 2k × 1k first image 91. On the other hand, the resolution enhancement layer 87 is decompressed using the conventional MPEG-2 algorithm and returns to the 2k × 1k second image 92. Then, the 2k × 1k first image 91 and the 2k × 1k second image 92 are added to generate a 2k × 1k high-resolution image 93.

Improvements from MPEG-2 In essence, the enhancement layer is formed by extending the decoded base layer, taking the difference between the original image and the decoded base layer, and compressing it. . However, optionally, the compressed resolution enhancement layer may optionally be added to the base layer after decoding to create a higher resolution image in the decoder. The layered resolution encoding process according to the present invention differs from MPEG-2 spatial scalability in several respects. Ie:
The enhancement layer differential picture is compressed as its own MPEG-2 data stream along with I, B and P frames. This difference represents the main reason that the resolution scalability proposed in this document is effective even when the spatial scalability of MPEG-2 is not effective. Spatial scalability defined in MPEG-2 is the difference between the upper layer picture and the extended base layer, or as a motion compensated MPEG-2 data stream of the actual picture, or a combination of both As a matter of course, it is possible to code higher layers. However, none of these encodings are efficient. Although the difference from the base layer can be considered as a difference I frame, it is inefficient compared to a motion compensated difference picture as in the present invention. The encoding of the upper layer defined in MPEG-2 is also inefficient because it is equivalent to completely encoding the upper layer. Therefore, as in the present invention, motion compensation coding of difference pictures is much more efficient.
Since the enhancement layer is an independent MPEG-2 data stream, the transport layer (or other similar mechanism) of the MPEG-2 system must be used to multiplex the base layer and the enhancement layer.
The expansion and resolution reduction filter processing may be a Gaussian or spline function, and is more preferable than the bilinear interpolation defined in the MPEG-2 spatial scalability.
In a preferred embodiment, the image aspect ratio must be consistent between the lower and upper layers. MPEG-2 spatial scalability allows expansion to width and / or height. Such stretching is not allowed in the preferred embodiment, according to efficiency requirements.
-Due to efficiency requirements and due to the very large amount of compression used in the enhancement layer, not all areas of the enhancement layer are coded. Usually, the area excluded from expansion will be a border area. Accordingly, the 2kx1k enhancement layer source image 86 in the preferred embodiment is center weighted. In a preferred embodiment, a fading function (such as linear weighting) is used to avoid abrupt changes in the image by “blurring” the enhancement layer toward the center of the image as it moves away from the border edge. In addition, you can use manual or automatic methods to determine the areas with details you will follow, and select areas that require detail, and exclude areas that do not require excessive detail. . The entire image has basic layer level details and all of the image is present. Only areas of special interest will benefit from the enhancement layer. In the absence of other criteria, frame edges or boundaries may be excluded from expansion, as in the center weighted embodiment described above. Using the “Lower_Layer_Prediction_Horizontal & Vertical Offset” parameter, which is an MPEG-2 parameter and used as a negative integer, in combination with the value of “Horizontal & Vertical_Subsampling_Coefficient m & n” It is possible to specify the overall size of the enhancement layer rectangle and its placement within the extended base layer.
Add the sharpness coefficient to the enhancement layer to offset the sharpness loss that occurs during quantization. Care must be taken to use this parameter only to restore the sharpness and sharpness of the original picture and not to enhance the image. As described in connection with FIG. 8, the sharpness coefficient is the “high octave” of the resolution between the original high resolution image 80 and the original base layer image 81 (after expansion). In addition to including high octave resolution sharpness and detail, this high octave image is considerably noisy. If this image is added too much, the motion compensation coding of the enhancement layer may become unstable. The amount to be added depends on the noise level in the original image. A typical weight is 0.25. For noisy images, sharpness should not be added, rather it may be prudent to suppress noise in the original image for the enhancement layer using a traditional noise suppression technique that preserves the detail before compression. Absent.
Temporal scalability and resolution scalability are mixed by utilizing B frames for temporal expansion from 36 to 72 Hz in both the base layer and the resolution enhancement layer. In this way, since there are options available at two levels of temporal scalability, four levels of decoding capability can be obtained with two layers of resolution scalability.

  These differences represent an essential improvement from the spatial and temporal scalability of MPEG-2. However, these differences are still consistent with the MPEG-2 decoder chip, although additional logic may be required in the decoder to extend and add in the resolution extended decoding process shown in FIG. Such additional logic is almost the same as that required by the spatial scalability of MPEG-2, which is inferior in effect.

  Non-MPEG-2 Coding of Arbitrary Resolution Enhancement Layer A compression technique different from MPEG-2 can be used for the resolution enhancement layer. Furthermore, it is not necessary to use the same compression technique for the resolution enhancement layer as for the base layer. For example, when a difference layer is coded, motion compensated block wavelets can be utilized to match and track details very efficiently. Even if the most efficient position to place each wavelet jumps on the screen due to a change in the amount of difference, the low amplitude enhancement layer will not notice it. Furthermore, it is not necessary to cover the entire image, only the wavelet is placed on the detail. It is also possible to guide the arrangement of wavelets by the detail area in the image. Their arrangement may be biased away from the edge.

  Multiple resolution enhancement layers 2M pixels (2048 × 1024), 72 frames / second are encoded at 18.5 mbit / s, the bit rate described herein, the base layer (1024 × 512, 72 fps) and one resolution enhancement layer Only succeeded. However, the improved efficiency expected to be made possible by further refinement of the resolution enhancement layer coding should allow for multiple resolution enhancement layers. For example, it is conceivable that a 512 × 256 base layer can be extended to 1024 × 512, 1536 × 768, and 2048 × 1024 by four layers. This is possible with the existing MPEG-2 coding as long as the frame rate of the movie is 24 frames per second. At a high frame rate such as 72 frames per second, MPEG-2 cannot perform the encoding of each resolution enhancement layer with sufficient efficiency, and at the present time, this multiple layer cannot be realized.

Mastering Format Using a template of 2048 x 1024 pixels or near, it is possible to create a single digital video master format source corresponding to various public formats. As shown in FIG. 6, a 2k × 1k template can efficiently support typical widescreen aspect ratios of 1.85: 1 and 2.35: 1. The 2kx1k template can accommodate 1.33: 1 and other aspect ratios.

  In resolution layering, integers (especially factor 2) and fractions (3/2 and 4/3) are the most efficient step sizes, but any resolution layer structure required using any ratio is achieved. Is possible. However, the use of 2048x1024 templates or close to them not only provides a high-quality digital master format, but many other convenient resolutions can be provided from the base layer of factor 2 (1kx512), which is Includes John standard NTSC.

  It is also possible to scan the film with higher resolution, such as 4kx2k, 4kx3k or 4kx4k. With arbitrary resolution extensions, higher resolutions can be formed from a central master format resolution around 2kx1k. An enhancement layer for such a film would consist of image detail, graininess and other sources of noise (such as scanner noise). Because of this noise, an alternative to MPEG-2 type compression would be required to use compression techniques in the enhancement layer for these very high resolutions. Fortunately, there are other compression techniques that can be used to compress such noisy signals while maintaining the desired details in the image. An example of such a compression technique is a motion compensated wavelet or motion compensated fractal.

  Preferably, from an existing movie, a digital mastering format should be created at the film frame rate (ie, 24 frames per second). Using both 3-2 pulldown and interlace together would not be appropriate for a digital film master. New digital electronic materials will not use 60 Hz interlace in the near future and are expected to be replaced by more computer compatible frame rates such as 72 Hz as proposed herein. The digital image master should be made at the frame rate at which the image is captured, whether 72 Hz, 60 Hz, 36 Hz, 37.5 Hz, 75 Hz, 50 Hz, or any other rate.

  The concept of a mastering format as a single digital source picture format for all electronic public formats is that PAL, NTSC, letterbox, pan and scan, HDTV, and other masters are all generally independent of the original of the film It is different from the existing practice of being created. The use of a mastering format allows both film and digital / electronic shows to be published in various resolutions and formats once mastered.

Combined resolution enhancement layer and temporal enhancement layer As mentioned above, both temporal and resolution enhancement layering can be combined. Time extension is provided by decoding B frames. The resolution enhancement layer also has two temporal layers and therefore contains B frames.

  For 24 fps film, the most efficient and lowest cost decoder may use only P frames. This minimizes both memory and memory bandwidth, and simplifies the decoder by eliminating B-frame decoding. Therefore, according to the present invention, it is possible to use a decoder that does not have B frame processing capability for the decoding of movies of 24 fps and the decoding of next-generation television of 36 fps. Then, as shown in FIG. 3, the B frame is used between each P frame to produce a higher 72 Hz temporal layer that can be decoded by a second decoder. The second decoder could also be simplified because it only needs to decode the B frame.

  Such layering also applies to the extended resolution layer, and only P and I frames can be used for 24 and 36 fps rates as well. By adding decoding of B-frames within the resolution enhancement layer, the resolution enhancement layer can further achieve a full time rate of 72 Hz at high resolution.

  The combined resolution and temporal scalable options of the decoder are shown in FIG. This example also shows the distribution of the data stream rate of approximately 18 mbit / s to realize the space-time layered next generation television of the present invention.

  In FIG. 10, a base layer MPEG-2, 1024 × 512 pixel data stream (including only P frames in the preferred embodiment) is provided to the base resolution decoder 100. P frames require a bandwidth of approximately 5 mbit / s. The basic resolution decoder 100 can decode at 24 or 36 fps. The output of the basic resolution decoder 100 includes a low resolution, low frame rate image (1024 × 512 pixels at 24 or 36 Hz).

  B frames from the same data stream are parsed and provided to the base resolution time enhancement layer decoder 102. Such B frames require a bandwidth of approximately 3 mbit / s. The output of the basic resolution decoder 100 is also coupled to the time enhancement layer decoder 102. The time enhancement layer decoder 102 can decode at 36 fps. The combined output of the time enhancement layer decoder 102 includes a low resolution, high frame rate image (1024 × 512 pixels, 72 Hz).

  Also in FIG. 10, a resolution enhancement layer MPEG-2, 2 k × 1 k pixel data stream (including only P frames in the preferred embodiment) is provided to the base time high resolution enhancement layer decoder 104. These P frames require a bandwidth of approximately 6 mbit / s. The output of the basic resolution decoder 100 is also connected to the high resolution enhancement layer decoder 104. The high resolution enhancement layer decoder 104 can decode at 24 or 36 fps. The output of the high resolution enhancement layer decoder 104 includes a high resolution, low frame rate image (24 or 36 Hz at 2 k × 1 k pixels).

  B frames from the same data stream are parsed and provided to the high resolution temporal enhancement layer decoder 106. Such B frames require a bandwidth of approximately 4 mbit / s. The output of the high resolution enhancement layer decoder 104 is coupled to the high resolution time enhancement layer decoder 106. The output of the time enhancement layer decoder 102 is also coupled to the high resolution time enhancement layer decoder 106. The high resolution time enhancement layer decoder 106 can decode at 36 fps. The combined output of the high resolution temporal enhancement layer decoder 106 includes a high resolution, high frame rate image (2 k × 1 k pixels at 72 Hz).

The compression rate achieved through this scalable coding mechanism is very high, indicating very high compression efficiency. Table 5 shows the compression ratio for each of the temporal and scalability options in the example of FIG. These compression rates are based on a source RGB pixel of 24 bits / pixel. (When calculating the conventional 16 bits / pixel 4: 2: 2 encoding or the conventional 12 bits / pixel 4: 2: 0 encoding, the compression rate is 3/4 and 1/2 of the indicated value, respectively. Will be.)

These high compression ratios are made possible by two factors.
1) High temporal coherency (coherence) of 72Hz images with high frame rate
2) High spatial coherency (coherence) of high resolution 2kx1k images
3) Apply resolution detail enhancement to important parts of the image (eg, central center) and not to less important parts (eg, frame boundaries)

  These elements are utilized by utilizing the strength of the MPEG-2 coding syntax in the layered compression method of the present invention. These strengths include bi-interpolated B frames for temporal scalability. MPEG-2 syntax also provides efficient motion representation by using motion vectors in both basic and extended layers. MPEG-2 also encodes detail, not noise, efficiently in the enhancement layer through motion compensation in conjunction with DCT quantization up to a threshold with high noise and quick image changes. If this threshold is exceeded, data bandwidth is best allocated to the base layer. These MPEG-2 mechanisms work together when used in accordance with the present invention to produce highly efficient and effective coding that is both temporally and spatially scalable.

  Compared to 5 mbit / s CCIR601 digital video encoding, the compression rates in Table 5 are much higher. One reason for this is the loss of some coherence due to interlacing. Interlacing adversely affects not only the correlation between vertically adjacent pixels, but also the ability to predict both subsequent frames and fields. Thus, most of the increase in compression efficiency described here is due to the absence of interlace.

The large compression ratio achieved by the present invention can be considered in terms of the number of bits available to code each MPEG-2 macroblock. As described above, a macroblock is a group of 16 × 16 pixels consisting of four 8 × 8 DCT blocks, with one motion vector for P frames and one or two motion vectors for B frames. Table 6 shows the bits available per macroblock for each layer.

  The number of bits available to code each macroblock is less in the enhancement layer than in the base layer. This is appropriate because it is desirable for the base layer to be as high quality as possible. The motion vector requires about 8 bits, leaving 10-25 bits for the macroblock type code and DC and AC coefficients for all 4 8x8 DCT blocks. This can only afford a few “strategically available” AC coefficients. Thus, statistically, most of the information available for each block must come from the preceding frame of the enhancement layer.

  It is easy to see why the MPEG-2 spatial scalability is not effective at these compression rates. This is because there is not enough data space available to code enough DC and AC coefficients to represent the high octave of the detail represented by the extended difference image. High octaves are represented primarily in the fifth through eighth horizontal and vertical AC coefficients. If only 2-3 bits per DCT block are available, these coefficients cannot be reached.

  The system described here gains its efficiency by utilizing motion compensated prediction from past extended difference frames. This is clearly effective in producing excellent results in encoding temporal and resolution (spatial) layer structures.

  Graceful Degradation The temporal and resolution scaling techniques described here work well for materials that normally operate at 72 frames per second using a 2kx1k original source. These approaches work well for film-based materials that operate at 24 fps. However, at high frame rates, if very noisy images are coded, or if there are too many shooting cuts in the image stream, the enhancement layer is necessary for effective coding. You may lose coherence between frames. Such loss is easily detected because the typical MPEG-2 encoder / decoder buffer occupancy / rate control mechanism attempts to set the quantizer to a very coarse setting. When this condition is encountered, all the bits normally used for encoding the resolution enhancement layer can be assigned to the base layer. This is because the base layer requires as many bits as possible to code the stressful material. For example, between about 0.5 and 0.33 Mpixels per frame for the base layer, and 72 frames per second, the resulting pixel rate would be 24 to 36 Mpixels / sec. Giving all available bits to the base layer gives about 500,000 to 670,000 additional bits per frame at 18.5 mbit / s, which encodes very well even for stressful material Should be enough.

  Even in the more extreme cases where all frames are very noisy and / or cuts occur every few frames, there is no further loss of resolution in the base layer. It is possible to perform full degradation. This can be achieved by removing the B frames that encode the time enhancement layer, so that all of the available bandwidth (bits) can be used for 36 fps base layer I and P frames. This increases the amount of data available for each frame of the base layer to approximately 1.0-1.5 mbit / frame (depending on the resolution of the base layer). Also, even under extremely stressful coding conditions, a fairly good motion representation rate of 36 fps will be achieved with a fairly high quality resolution of the base layer. However, if the base layer quantizer is still operating at a coarse level of about 18.5 mbit / s, 36 fps, the base layer frame rate drops dramatically to 24, 18, or 12 frames per second. However, it would be possible to handle even the most unusual moving image types. A method for changing the frame rate in such a situation is known.

  Current proposals for next-generation television in the United States do not allow these graceful degradation methods, and therefore cannot perform as well as the system of the present invention for stressful materials.

  In most MPEG-2 encoders, the adaptive quantization level is controlled by the output buffer occupancy. At high compression rates in the resolution enhancement layer of the present invention, this mechanism may not work optimally. Various techniques can be used to optimize data allocation to the optimal image area. The simplest conceptual approach is to perform an encoding pre-pass on the resolution enhancement layer, gather statistics, and search for details to maintain. The prepass result can be used to set the appropriate quantization to optimize the maintenance of details in the resolution enhancement layer. It is also possible to artificially bias the quantization settings so that they are non-uniform on the image, assign image details to the main screen area, and bias them away from the macroblock at the edge of the frame. is there.

  Existing decoders work well without such improvements, so none of these adjustments are necessary except to leave the enhancement layer boundaries at a high frame rate. However, these further improvements can be exploited with a slight extra effort on the enhancement layer encoder.

Conclusion It seems optimal to choose 36 Hz as the new common base time rate. To demonstrate the use of this frame rate, it can be seen that there is a significant improvement from 24 Hz for both 60 Hz and 72 Hz displays. A 36 Hz image can be created using every other frame from a 72 Hz image capture. Thereby, a 36 Hz base layer (preferably using a P frame) and a 36 Hz time extension layer (using a B frame) can be combined to realize a 72 Hz display.

  The “future-oriented” rate of 72 Hz is not compromised by the approach of the present invention. A transition for analog NTSC display for 60 Hz is provided. The present invention also allows a transition for display for 60 Hz, even if other 60 Hz formats dedicated to other passive entertainment being considered (computer incompatible) are accepted.

  Resolution scalability can be achieved by using a separate MPEG-2 image data stream for the resolution enhancement layer. Resolution scalability can take advantage of the B-frame approach to provide temporal scalability in both basic and extended resolution layers.

  The invention described herein achieves many highly desirable features. Some people in the US next-generation television processing claim that neither resolution nor temporal scalability can be achieved with high-definition resolution within approximately 18.5 mbit / s available for terrestrial broadcasting. . However, the present invention achieves both temporal scalability and space-resolution scalability within this available data rate.

  It has also been argued that within the available data rate of 18.5 mbit / s, 2M pixels at high frame rates cannot be achieved without using interlacing. However, the present invention not only achieves resolution (spatial) scalability and temporal scalability, but can realize 2M pixels at 72 frames / second.

  In addition to providing these capabilities, the present invention is also very robust, especially compared to current proposals for next generation television. This is made possible by assigning most or all bits to the base layer when encountering very stressful image material. Such stressful materials are noise-like in nature and change very quickly. Under these circumstances, the details associated with the resolution enhancement layer are not visible. Since the bits are devoted to the base layer, the playback frame is much more accurate than current proposed next generation television systems that use a single, constant, higher resolution.

  In this way, the system of the present invention optimizes both perceptual efficiency and coding efficiency while providing maximum visual impact. This system provides very sharp images with resolution and frame rate capabilities that have been considered impossible for many people. It is believed that the system of the present invention is likely to outperform the next generation television format currently proposed. In addition to this expected great performance, the present invention also provides very valuable features of temporal and resolution layer structures.

Encryption and Watermarking Overview Layered compression allows a form of modular decomposition of images that supports flexible encryption and watermarking techniques. By using layered compression, the base layer and various internal components of the base layer can be used to encrypt and / or watermark the compressed layered movie data stream. Encrypting and watermarking the compressed data stream reduces the amount of processing required compared to a high resolution data stream that must be processed at the rate of the original data. The computation time required for encryption and watermarking depends on the amount of data that must be processed. If the computational resources are at a certain level, reducing the amount of data through layered compression can produce improved cryptographic strength and / or reduced encryption / decryption costs.

  Encryption protects the compressed image (and audio) data and makes it easy for only the user with the key to access the information. Layered compression breaks an image into components: temporal and spatial base layers, and temporal and spatial enhancement layers. The base layer is the key to decoding the visible picture. Therefore, only the time and space base layers need to be encrypted, thereby reducing the amount of computation required. The temporal and spatial enhancement layers are worthless without the decrypted and decompressed base layer. Therefore, by using such a layered subset of bits, the entire picture stream can be made unrecognizable by encrypting only a small portion of the bits of the entire stream. Different encryption algorithms and strengths can be applied to different parts of the layered stream including the enhancement layer. Also, the encryption algorithm or key may be changed at each slice boundary (data stream structure for signal error recovery) to further entangle the encryption and picture stream.

  A copy (reproduction) of a work is marked by watermarking invisible (or almost invisible). This concept stems from the practice of placing identifiable symbols in paper to ensure that a document (eg, money) is authentic. Watermarking can track copies that can be removed from the ownership of an authorized owner or licensee. Thus, watermarking helps to track lost or stolen copies to their source, can determine the nature of the theft method, and allows identification of those involved in the theft.

  The concept of watermarking has been applied to images by attempting to place faint image symbols or signatures on the actual image being displayed. The most widely recognized concept of digital watermarking is a low amplitude visible image applied over a high amplitude visible image. However, this approach slightly alters the quality of the original image, similar to the process of applying a network logo to the corner of the screen on the television. Such a change is undesirable because it degrades picture quality.

  In the compression domain, the signals can be modified so that watermark symbols or codes are applied on those signals, but these watermark modifications are not applied directly to the visual domain. For example, DCT transform operates in a frequency transform space. Any alterations in this space may be much less visible (or completely invisible), especially if corrected from frame to frame. Preferably, the watermarking uses the low order bits of a particular coefficient in a particular frame of the layered compressed movie stream to provide reliable identification while invisible or nearly invisible to the eye. Watermarking can be applied to the base layer of the compressed data stream. However, the enhancement layer is initially very subtle in detail and can be protected to a much greater extent than the base layer. Each enhancement layer can have its own unique identification watermark structure.

  In general, care must be taken to ensure that encryption and watermarking are mixed so that watermarking cannot be easily removed from the stream. For this reason, it is beneficial to apply the watermark to various useful locations within the layered data stream. However, since watermarking is most useful for detecting pirates and piracy paths, it must be assumed that encryption has been fully or partially weakened, so watermarking uses a simple procedure. So that these various watermarks cannot be removed, they should be stubbornly deeply embedded in the data stream. A preferred approach is to have a master representation of the securely stored work and provide random variations from that master to create each watermark independently. Such random variations cannot be removed because there is no way to detect what those variations were from the final stream. However, to protect against the additional random variations (possibly by adding visible level noise to the image) that are added to the looted stream to disrupt the watermark, various other techniques for defining the watermark (see below) It is useful to have a vector second best technique etc.

The encryption preferably operates to perturb as many frames as possible by the smallest possible encryption unit (scramble) or at least visually damage. Various types of compression systems, such as MPEG and motion compensated wavelets, use a range of frames ("Groups"
In order to decode “of Pictures” (ie GOP), it uses units of hierarchical information that have to be cascaded. This feature provides an opportunity to encrypt to scramble a wide range of frames from a few parameters early in the range of the concatenated decrypted units. Furthermore, in order to protect the work commercially, it is not necessary for all units to be encrypted or disturbed by higher level unit encryption. For example, if every minute frame of frames, or particularly important plots or action scenes are encrypted or perturbed, the film will be worthless for piracy.

  In contrast, the goal of watermarking is to stream a symbol and / or serial number identification mark that is detectable by analysis but invisible or nearly invisible (ie, does not cause significant visual damage) in the image. Is to place on top. Thus, watermarking is preferably applied to each part near the end of each unit's hierarchy in the decoding unit chain so as to minimize the effect on each frame in the group of frames.

  For example, FIG. 11 is a diagram showing the range covered by encryption and watermark processing as a function of unit dependency on I, P and B frames. No matter which frame is encrypted, all subsequent dependent frames are disturbed. Therefore, when the first I frame is encrypted, all P and B frames derived from that I frame are disturbed. In contrast, the watermark on that I frame is usually not carried over to subsequent frames, so it is better to watermark the larger number of B frames to spread the watermark more widely in the data stream.

Video Information Units A compressed MPEG type or motion compensated wavelet bitstream is parsed by successfully extracting and processing various basic units of compressed information in the video. This is true for the most efficient compression systems such as MPEG-2, MPEG-4 and motion compensated wavelets (when the wavelet is considered to have equivalent to I, P and B frames). Such units include multi-frame units (such as GOP), single frame units (eg, I, P and B frame types, and motion compensated wavelet equivalents), subframe units (AC and DC coefficients, macroblocks, and motion vectors). ), And "distributed units (distributed
unit) ”(described later).

  When GOPs are used as encryption units, each GOP can be encrypted with an independent method and / or key. In this way, each GOP benefits from its own processing and modularity, and in parallel with other GOPs in non-real-time or near real-time (slightly delayed by a few seconds) applications (such as electronic movies and broadcasts). Or can be decoded and / or decrypted out of order. The final frames need only be arranged in the final display order.

As mentioned above, the encryption of a particular unit may disrupt the proper decryption of other units that are dependent on the information obtained from that encrypted unit. In other words, when certain information in one frame is required for decoding video information of the subsequent frame and only the previous frame is encrypted, the decoding of the subsequent frame that is not otherwise encrypted is disturbed. May be. Thus, when selecting a unit to encrypt, it is beneficial to note how encrypting a particular unit disrupts the availability of other related units. For example, multiple frames spanning one GOP are affected at various levels as shown in Table 7.

Furthermore, it is not necessary to encrypt the entire frame to disturb some or all of the GOP. The sub-unit of the frame may be encrypted, and the disturbance effect is still exhibited while reducing the processing time for encryption and decryption. For example, encryption of a specific intra-frame unit affects subsequent frames at various levels as shown in Table 8.

  In many applications (such as broadcast and digital movies) delays can be applied to encrypt collections of items from similar units before transmission. This realizes a “distributed unit” in which the bits including the encryption / decryption unit are physically allocated everywhere in the data stream in the conventional unit of the type described above, and can be decrypted without knowing the key. Can be much more difficult. To decode, a sufficient number of conventional units are collected (eg, in a buffer) and decoded as a group. For example, DC coefficients can be collected in groups for the entire frame or GOP. Similarly, motion vectors are differentially coded and predicted throughout the frame, such as from one motion vector to the next motion vector, from one macroblock to the next, and so on. Can be encrypted and decrypted with. Variable length coding tables are also collected in the group, and a modular unit can be formed between “start codes”. Further examples of units or subunits that can be aggregated, encrypted, and whose encrypted bits can be separated or distributed within a data stream include motion vectors, DC coefficients, AC coefficients, and quantum Includes the scale factor of the generator.

Application of Encryption In a preferred embodiment, one or more of the above units (or other data stream units with similar characteristics) may be selected for encryption, each unit being (MPEG-1, It can be encrypted independently (as in MPEG-2 and MPEG-4) rather than as a combined stream. Different units may use different keys with different strengths (eg, number of bits per key) and different encryption algorithms may be used.

  Encryption can be applied independently for each individual copy of the work (when a physical medium such as a DVD-RAM is used), so that each copy has its own key. Alternatively, the encryption algorithm may be a stream assembled with a significant portion of the stream removed or modified from its data stream before encryption (eg, by setting the left macroblock's motion vector to all zeros). Can be applied to. This defines the shape of the bulk (mass) distribution copy. The removed or modified parts can then be encrypted separately and independently for each display site, thereby separating them into individual sites in a convenient manner (eg satellite transmission, modem, internet, etc.) A custom distribution copy to be sent to is defined. This technique, for example, is where most of the work is delivered on a medium such as a DVD-ROM, while unique copies of smaller critical compression units are sent separately to independent recipients with their own keys. This is useful when used (eg, via satellite, internet, modem, express delivery, etc.). Only after the custom part is decrypted and recombined with the decrypted bulk delivery copy can the entire work be decoded as a video signal. The larger the bandwidth (size capacity) of such custom information, the larger the image portion that can be custom encrypted. This technique can be used in combination with watermark processing.

  A variation of this approach is to encrypt a subset of the units of the data stream as a custom distribution copy and not encrypt the rest of the units at all. The remaining units may be delivered in bulk form separately from the custom delivery copy. Only after the custom part has been decrypted and recombined with the decrypted bulk delivery copy can the entire work be decrypted as a video signal.

  One or more overall encryptions may be concatenated or combined with special customized encryptions for various important units of video decryption information. For example, the entire video data stream is “lightly” encrypted (eg, using a short key or a simple algorithm) while the unit that holds a particular key in the data stream is “heavy” (eg, a long key or It may be encrypted (using a more complex algorithm). For example, in one embodiment, the highest order resolution and / or temporal layer may be more heavily encrypted to create a premium signal shape that provides the best looking image when properly decrypted. The lower layers of the image are not affected by such encryption. This approach will enable various grades of signaling services for end users.

  If each unit is encrypted independently of each other, one or more parallel processing decryption methods may be used privately on separate units in the compressed image stream, and decryption may be performed in parallel.

Application of watermarking For each unit discussed above and other units with similar characteristics, different locations in the compressed video data stream are suitable for applying watermarks in different ways, such locations. Includes the following:
• In transformation space or real space, or a combination thereof.
In the least significant bit (LSB) of the DC coefficient. For example, the DC coefficient can have extra bits (10 and 11 bits for MPEG2 and up to 14 bits for MPEG4 are allowed). The low order bits can encode a particular watermark identifier without any visual degradation of the image. Further, such low order bits may only need to be in I frames, since a clear watermark need not be present in every frame.
In the noise pattern in the AC coefficient LSB.
• At low frequencies of the entire low-amplitude picture, it is coded from frame to frame to form an imaging pattern that is not visually detectable. This may be, for example, a small number of low signal amplitude letters or numbers on each frame, where each letter may be very large and soft. For example, where one pixel should have the binary value “84”, the watermarking can set the value to “83” instead, and the watermark has a value of “1” at this position. Will have. The difference is essentially invisible to the eye but forms code within the compressed data stream. Such an imaging pattern subtracts the decoded image from the undisturbed (unwatered) decompressed original (and from the uncompressed original source work) and greatly increases its amplitude Will be detected. Then a very large and hazy series of letters or numbers will appear.
Use marks with very low visibility in non-propagating frames (I frame, last P frame before I frame, B frame, etc.). These frames are also only displayed for a short time.
-At the slice boundary (usually the left end of the macroblock line)

  The watermarks at these locations are generally added with a pattern of small variations in pixel data. These variations may form images or symbols that are invisible or nearly invisible to the eye due to the very low amplitude of bit variations in pixel brightness and color and / or due to the short display. For example, FIGS. 12A and 12B are diagrams of an image frame 1200 with different types of watermarks. FIG. 12A shows a frame 1200 with a single symbol (“X”) 1202 in one corner. FIG. 12B shows a frame 1200 with a set of marks (dots in this example) 1204, where the marks 1204 are dispersed on the frame 1200. These watermarks can only be detected by data comparison and generate a watermark signal. For example, a precision decoder can detect LSB fluctuations between an original work and a watermarked work that is invisible to the eye but performs its own watermarking on a customized copy of the original work. .

Although no specific image or symbol is added, other watermarking formats that form a unique pattern in the data stream may be used. For example, certain coding decisions are almost invisible, but can be used to watermark the data stream. For example, small fluctuations in rate control are invisible to the eye, but can be used to mark each copy so that each copy has a slightly different number of AC coefficients at several locations. Other examples of such decisions include the following:
• Rate control fluctuations in I frames • Rate control fluctuations in P and B frames • Assignment of specific AC coefficients that affect LSB

  Similarly, the second best motion vector, which is substantially equivalent to the optimal motion vector, may be selected to create a watermark code. Alternatively, the second best one may be selected where exactly the same SAD (absolute difference sum, common motion vector matching criterion) occurs and occurs. If necessary, other non-optimal (eg, third and higher rank) motion vector matches can be used with little visual damage. Such second-choice (and higher) motion vectors only need to be used in a coherent pattern from time to time (eg 2-3 紺 per frame) to form a watermark code. It is.

  Image variations are less visible near the periphery of the frame (i.e., near the top, bottom, right edge, and left edge). Thus, if there is a possibility that even a small image or symbol type watermark may be visible, it is better to apply the selected watermark to the edge region of the image. Watermarking methods with very low visibility (such as the second optimal motion vector or rate control variation) can be used everywhere on the image.

  The watermark processing can be coded as a unique (unique) serial number type code for each copy subjected to the watermark processing. Thus, 1000 copies of the original work will each be watermarked slightly differently using one or more of the above techniques. By tracking where each watermarked copy was shipped, it is possible to determine which copy was the source of unauthorized duplication if a watermark was found during unauthorized copying.

Watermark detection Most of these watermarking methods use the decompressed original image as a reference for comparison with each watermarked copy to make it visible (decrypted). Require to do. The difference between the two images reveals the watermark. Therefore, it is necessary to store the master stretched source in a safe place. Security is required because possessing a master stretched source copy provides enough information to ruin many of the watermarking methods. However, the theft of the watermark processing comparison master itself can be detected. This is because the watermark processing comparison master is automatically subjected to “watermark processing” so as to perfectly match itself. If a watermarking comparison master was used to disturb the copy (ie find and remove the watermark), it indicates that you own the master.

  The advantage of using a low-amplitude large blurred symbol or image as a watermark is that it can be detected not only by comparison to a stretched master source, but also by comparison to an uncompressed original work Have Thus, by storing the uncompressed original work in a separate and secure environment, the low amplitude watermark can be used in the original (unmodified otherwise) compressed master source. Thus, even if either the original work or the compression / decompression master source is stolen, the reference standard for watermark comparison will remain. However, owning both would be able to ruin both watermarks.

Watermark Vulnerability When using watermarking, it is important to understand how it can be used to ruin or disrupt the detection of such marks. Some watermark processing methods are disturbed by adding a small amount of noise to the image. This degrades the image quality somewhat, and the degradation may be visually small. However, it may be sufficient to disrupt watermark interpretation. Watermarking techniques that are vulnerable to disturbances that add noise include those that use LSBs in DC or AC coefficients.

  There are also watermarking methods that are more difficult to disturb with noise. A watermarking technique that is resistant to noise disturbances, but still easy to detect, includes low-frequency image fluctuations throughout the low-amplitude picture (very low-amplitude, low-amplitude superimposed on the image). Large words, etc.) including the second optimal motion vector and small variations in rate control.

Therefore, to avoid the simple method of trying to disturb watermark detection,
It is highly valuable to use a plurality of watermark processing methods. Furthermore, the use of encryption can ensure that the watermark cannot be altered unless the encryption is compromised. Therefore, preferably, the watermark processing is used in combination with encryption having a strength suitable for the application.

Tool Kit Approach The various concepts of encryption and watermarking including such aspects of the present invention are preferably embodied as a tool set that can be applied to the task of protecting valuable audio / video media. The tools can be variously combined as desired by the content developer or distributor to create a layered compressed data stream protection system.

  For example, FIG. 13 is a flowchart showing one method to which the encryption method of the present invention is applied. A unit to be encrypted is selected (step 1300). This may be any of the above units (eg, a distributed unit, a multi-frame unit, a single frame unit, or a sub-frame unit), or any other unit with similar characteristics. An encryption algorithm is selected (step 1302). This may be a single algorithm applied throughout the encryption session, as described above, or may be selected per unit. Suitable algorithms are well known and include both secret and public key algorithms such as DES, Triple DES, RSA, Blowfish, etc. Next, one or more keys are generated (step 1304). This entails selection of both key length and key value. Again, this may be a single selection applied throughout the encryption session, as described above, or a truly unit-by-unit selection. Finally, the unit is encrypted using the selected algorithm and key (step 1306). Then, the process for the next unit is repeated. Of course, some steps, in particular steps 1300, 1302 and 1304, may be performed in a different order.

  For decompression, the associated key will be applied to decrypt the data stream. The data stream is then decompressed and decoded as described above to produce a displayable image.

  FIG. 14 is a flowchart showing one method of applying the watermark processing method of the present invention. A unit to be watermarked is selected (step 1400). This may be any of the above units (for example, a distributed unit, a multi-frame unit, a single frame unit, or a sub-frame unit), or other unit having similar characteristics. Then, one or more watermark processing methods are selected (step 1402), such as a method that is resistant to noise and a method that is not resistant to noise. This may be one choice applied throughout the watermarking session, or truly per unit (or unit classification if more than one watermarking technique is applied to different types of units). May be selected. Finally, watermark processing is performed on the selected unit using the selected method (step 1404). Then, the process is repeated for the next unit. Of course, some steps, in particular steps 1400 and 1402, may be performed in a different order.

Key Management The encryption / decryption key can be associated with various items of information to form a more secure or synchronized key. For example, public or private encryption and decryption keys may be generated or derived from including any of the following components:
・ Past key.
The serial number of the destination device (eg a cinema projector with a secure serial number).
A date or time range (uses a safe clock) and ensures that the key only works for a specific time (eg only for a specific day of the week, only for a relative period such as a week). For example, the encryption system may plan to use a secure GPS (Global Positioning Satellite) in the decoder as a time source. The decryption processor can decrypt the image file or stream only by accessing the secure time source.
-The position of the decoding processor. The GPS capability allows fairly accurate real-time location information to be incorporated into the key. A known destination static Internet Protocol (IP) address can also be used.
・ Accounting record of the number of past screenings of the work reported by each movie theater (manually or automatically).
A “PIN” (personal identification number) of a specific authenticator (eg, movie theater administrator).
Using a physical customized encrypted movie (eg a DVD movie, where each movie is uniquely keyed to a particular theater) and keyed at its intended site Ownership of the encrypted movie itself by the holder can be used as a key authentication format for subsequent movies. For example, playing a portion of a movie and transmitting that portion to a remote key generation site can be part of a key authentication protocol. Further, if the distribution copy is stored on an erasable medium such as a hard disk or DVD-RAM, the use of encrypted movie data as a key element can be associated with a secure media erasure key. In this way, past movies are erased as part of the key process for obtaining a new movie.
Also, the key is valid for a specific number of screenings or other natural number of usage units, after which a new key can be requested.

Various methods for managing the distribution of the decryption key can be applied. Various key management tactics can be applied to each usage method and each data distribution method (either network data transfer, satellite, or physical disk or tape media). An example of key distribution and management procedures is shown below.
The key is stored on a medium (eg floppy disk (floppy is a registered trademark), CDROM) and physically sent to the destination on the next day delivery, or electronically or in document form (eg facsimile, e-mail, direct connection data) (By transmission, internet transmission, etc.).
-The public key method can be used with a local unique key in addition to key verification by an authorized third party.
-By pre-defining key decryption and application rules for each destination (eg movie theater), the key itself is encrypted and transmitted electronically (eg, direct data transmission, Internet transmission, e-mail, etc.) May be.
-You may require ownership of the current key as a condition for obtaining or using a new key. The current key value may be transmitted to the key management site by any suitable means as described above, and the new key can be returned by one of the above means.
Use of a decryption key may require a “key handshake” with a key management site that verifies or authenticates the application of the key in all cases of decryption. For example, a decryption key may need to be combined with an additional symbol maintained by the key management site, although that particular symbol changes with each use. The use of the key handshake can be used for each screening, for each length of usage time, or for each other unit of natural values. Since such usage may be billed in a natural number unit, the key management logs the number of times of use or usage time and appropriately charges the key holder (for example, rental fee for each movie show) It can also be integrated with a billing system. For example, both key management and usage log records can be tied to a key authentication server system that can simultaneously handle billing for each authenticated show or usage time.

  Some keys may be pre-authenticated keys to keys that are authenticated on-site. Pre-authentication keys will generally be issued one at a time by the key management site. With on-site key authentication, the key management site issues a set of keys to the movie theater so that on-site managers can respond to audience demands for movies that are more popular than originally anticipated. It may be possible to authorize additional decryption (ie screening). If such a key is used, the system is designed to send a signal to the key management site for additional screening (eg, via email or data record sent over the Internet, or by modem) for billing purposes. Is preferred.

CONCLUSION Various aspects of the invention that are believed to be novel include, but are not limited to, the following concepts.
• Encryption applied to layered compression • Watermarking applied to layered compression • Applied to each layer of a layered system, requiring a different key, authentication, or algorithm for unlocking each independent layer Unique encryption • Applied to each layer and uses a subframe unit of the compressed image stream for unique watermarking / encryption or watermarking (using serial number or other methods) to identify a specific layer・ Use multiple watermark processing methods simultaneously to protect against detection of specific types of watermark detection. ・ Use multiple encryption methods and strengths simultaneously. Thereby requiring multiple independent decoding systems to decode various units in a single layer or layered compressed image stream; one or more for different units in a compressed image stream Deciphering in parallel using decryption methods-Tying keys to billing systems-Tying encryption to specific media and / or specific target locations or serial numbers-Encrypting a safe clock and range of use dates・ Use encryption as a key to get a new movie or key ・ Use as a key to get a new movie Delete movie data from physical media at the end of the authorized use period・ Continue refinement of key usage to improve flexibility, usability, and safety using a flexible key toolkit approach ・ The second best watermarking technique ( Or use the third (optimal etc.) motion vector and use small variation of rate control as watermarking technique (any combination of I, B and / or P type frames and their equivalents) Applies to motion compensated wavelets)
Use low order bit fluctuations in DC and / or AC coefficients as a watermarking technique (applies to I, B, and / or P type frames and their equivalents)
Use low amplitude blur characters or numbers that are uniquely added to each copy of the image during compression to uniquely watermark each copy.Bitstream effects that affect a large portion of the image stream. Apply encryption to parts (high impact on encryption)
Apply global encryption to the majority of the work and apply customized encryption to selected unitsEncrypt a small portion of the data stream and identify each of these in a point-to-point manner Sending to a location (including binding to a serial number, key, staff code, IP address, and other unique identifiers at that particular location)
Apply watermarking to parts of the bitstream that have a low impact on other frames to minimize visibility. To minimize visual impact, the edge region (top, bottom, Use potentially visible watermarks (such as low-amplitude letters and numbers, or LSB in DC or AC coefficients) for the left and right edges) • Left column (start slice) motion vector, I frame Extracting the influence points of subframe units for independent encryption such as DC and AC coefficients, prediction mode bits, control codes, etc.

Computer Implementation The present invention can be implemented in hardware (eg, an integrated circuit) or software, or a combination of both. However, the present invention executes on one or more programmable computers including at least a processing unit, a data storage system (including volatile and non-volatile memory and / or storage elements), an input device, and an output device. Preferably implemented in a computer program. By applying the program code to the input data, the functions described in this specification are executed and output information is generated. The output information is applied to one or more output devices in a known manner.

  Each such program may be implemented in any desired computer language (including machine language, assembly language, or high-level imperative words, logic words, or object-oriented programming languages) and communicate with a computer system. . In either case, the language may be a compiled language or a translated language.

  Each such computer program is stored on a storage medium or device (e.g., ROM, CD-ROM, or magnetic or optical medium) readable by a general purpose or special-purpose programmable computer system, and the storage medium or device is a computer system. Preferably, the computer is configured and operated to perform the procedures described herein. The system of the present invention is also considered to be implemented as a computer-readable storage medium configured with a computer program, which causes the computer system to operate in a specific predefined manner with the storage medium configured as such. The functions described in the specification may be performed.

  While several embodiments of the present invention have been described, it will be appreciated that various modifications can be made without departing from the spirit and scope of the invention. For example, although the preferred embodiment uses MPEG-2 encoding and decoding, the present invention will work with any similar standard that provides equivalents and layers for I, B, and P frames. Therefore, it will be understood that the invention is not limited to the specific embodiments described but is only limited by the scope of the appended claims.

It is a timing chart showing a pull-down rate for displaying materials of 24 fps and 36 fps at 60 Hz. A first preferred MPEG-2 coding pattern is shown. A second preferred MPEG-2 coding pattern is shown. FIG. 6 is a block diagram illustrating temporal layer decoding according to a preferred embodiment of the present invention. FIG. 6 is a block diagram illustrating a 60 Hz interlaced input to a converter capable of outputting both 36 Hz and 72 Hz frames. FIG. 4 shows a “master template” for a basic MPEG-2 layer at 24 or 36 Hz. FIG. 3 is a diagram illustrating an extension of a basic resolution template using hierarchical resolution scalability using MPEG-2. It is a figure which shows a preferable layered resolution encoding process. It is a figure which shows a preferable layered resolution decoding process. FIG. 4 is a block diagram illustrating a combination of resolution and temporal scalable options for a decoder according to the present invention. It is a figure showing the range which encryption and a watermark process cover as a function of unit dependence. FIG. 6 is an illustration of an image frame with a certain type of watermark. FIG. 4 is a diagram of image frames with different types of watermarks. It is a flowchart which shows one method of applying the encryption method of this invention. It is a flowchart which shows one method of applying the watermark processing method of this invention.

Explanation of symbols

  50 ... MPEG-2 decoder, 52 ... second decoder, 60 ... camera, 62 ... other source, 64 ... converter, 1200 ... frame, 1202 ... symbol, 1204 ... mark.

Claims (3)

A method for encrypting and watermarking a data stream of video information encoded and compressed into a base layer and at least one enhancement layer, comprising:
(A) selecting at least one encryption algorithm;
(B) selecting at least one watermarking technique;
(C) selecting at least one unit to be encrypted from the base layer or the at least one enhancement layer;
(D) selecting a unit to be subjected to at least one watermark processing of the base layer or the at least one enhancement layer;
(E) applying said at least one selected watermarking technique to subject each of said selected units to a watermarked unit;
(F) applying the at least one selected encryption algorithm to encrypt each of the selected units into an encrypted unit;
Including methods. A system for encrypting and watermarking a data stream of video information encoded and compressed into a base layer and at least one enhancement layer,
(A) means for selecting at least one encryption algorithm;
(B) means for selecting at least one watermark processing technique;
(C) means for selecting at least one unit to be encrypted of the base layer or the at least one enhancement layer;
(D) means for selecting a unit to be subjected to at least one watermark processing of the base layer or the at least one enhancement layer;
(E) means for applying a watermarking process by applying the at least one selected watermarking technique to each of the selected units as a unit to be subjected to the watermarking process;
(F) means for applying each of the at least one selected encryption algorithm to encrypt each of the selected units into an encrypted unit;
Including system. A computer program for encrypting a data stream of video information stored in a computer readable medium and encoded and compressed into a base layer and at least one enhancement layer and watermarking the data stream, the computer comprising:
(A) selecting at least one encryption algorithm;
(B) selecting at least one watermark processing method;
(C) selecting at least one unit to be encrypted from the base layer or the at least one enhancement layer;
(D) selecting a unit to be subjected to at least one watermark processing of the base layer or the at least one enhancement layer;
(E) applying the at least one selected watermarking technique to cause each of the selected units to be watermarked as a unit to be watermarked;
(F) applying the at least one selected encryption algorithm to encrypt each of the selected units into an encrypted unit;
A computer program containing instructions.

Images