WO1997024880A1

WO1997024880A1 - Color video data reduction

Info

Publication number: WO1997024880A1
Application number: PCT/US1996/020489
Authority: WO
Inventors: Oleg Rashkovskiy; Jerry Hauck
Original assignee: Intel Corporation
Priority date: 1995-12-29
Filing date: 1996-12-27
Publication date: 1997-07-10

Abstract

Raw digital video data is generated by a digital video camera and transmitted over a bus (27) to a computer (28). The raw digital video data is generated from the analog output of a charge coupled device (23) in the camera, which preferably includes color filter overlays. The bus can be a relatively low-speed bus, having a bandwidth of not greater than 12 Mbits/second. The video camera preferably processes the raw digital video data prior to transmitting it over the bus to decrease the resolution of the image defined by the raw digital video data. Such processing can be downsampling in both the horizontal and vertical dimensions. The computer includes a processor capable of converting the raw digital video data into usable digital video data, such as YUV or YIQ digital video data.

Description

COLOUR VIDEO DATA REDUCTION

BACKGROUND OF THE INVENTION 1. Field of the Invention

This invention relates to the field of video color format and compression.

2. Description of the Related Art

The advent of powerful microprocessors has expanded the quantity and quality of computer applications. Multimedia and other video related applications in particular are increasingly prominent. These applications typically involve transmitting video data into and out of personal computers (PCS). In typical video related applications, as shown in Fig. 1 , a digital video camera 10 is used to capture video 18. Digital video camera 10 typically includes a lens 12, a single plate charge coupled device (CCD) sensor 13, correlated double sampling and automatic gain control device 14, analog to digital (A/D) converter 15, digital signal processor

(DSP) 16 and microcontroller 17.

As noted, typical video cameras for PC and home applications use a single plate CCD. Therefore, to produce a color video signal, the CCD's are covered with a thin film color filter, which can have various color formats. For example, there are filters having primary colors: red, green and blue (RGB); and filters with complimentary colors: cyan, yellow, green, magenta (Cy,Y,G,Mg) and white, cyan, yellow and green (W,Cy,Y,G).

The correlated double sampling and automatic gain control device 14 conditions the analog signal output by the CCD 13 and passes the conditioned signal to A/D converter 15. The analog video signal is converted to a digital video signal by analog/digital converter 15 and the digital signal is passed to DSP 16. DSP 16 performs color transformation to, for example, the

YUV, which is the predominant choice both in the digital television industry and for PC digital video and teleconferencing applications, or YIQ format and further manipulates image quality, as is known to those having ordinary skill in the art. In fact, traditional video cameras perform a large amount of analog and digital signal processing. The hardware that implements this processing is responsible for a considerable amount of camera cost.

Microcontroller 17 controls the operation of DSP 16 as understood by those having ordinary skill in the art. The YUV format nominally includes a total of 24 bits of data, 8 bits for each of the Y, U and V components. Typically, the YUV signal is subsampled, for example by DSP 16 prior to transmission to PC 1 1 to reduce the number of bits which must be transmitted through bus 19. Popular subsampled YUV formats include YUV 16, YUV 12 and YUV9.

In YUV9, for example, pixels are grouped into 4 x 4 blocks, for a total of 16 pixels. Each of the 16 pixels maintains its separate 8 bit Y luminance information. Each pixel, however, shares a single 8 bit U and a single 8 bit V chrominance component. The total number of bits which must be sent to represent each pixel is 9: ((16 Y * 8 bits) + (1U * 8 bits) + (1 V * 8 bits))/ 16 pixels = 144 bits/ 16 pixels = 9 bits per pixel (9bpp). YUV9 subsampling thereby reduces the data required to represent one pixel from 24 bits to 9 bits without a severe effect on image quality, since human viewers have poor acuity for color transitions in an image but relatively high acuity to intensity level changes.

YUV 16 and YUV 12 color subsampling employ similar techniques on a smaller matrix of pixels.

After converting digitized raw CCD color information to, for example, a YUV format and subsampling the data, the camera must transmit the video data to PC 1 1, for example for storage or display on an attached monitor (not shown in Fig. 1). The video data typically is transmitted to PC 1 1 through a parallel or serial port and over Universal Serial Bus (USB) 19. Unfortunately, even the most advanced USB has a theoretical bandwidth limit of 12Mbit second, the maximum sustainable usable rate is more accurately about 6 Mbit/second. The bandwidth limitations imposed by the USB have a significant adverse effect on the quality of full motion PC video. For example, the following table shows maximum sustainable frame rates at various resolutions for YUV9 and YUV 12 data through a USB:

Colon YUV9 YUV 12 Resolution

160 * 120 34.7 fps 26.0 fps

320 * 240 8.7 fps 6.5 fps

Table 1. Video Transmission Frame Rates for YUV Color Formats

The minimum refresh rate for acceptable full motion video is at least 30 fps. As can be seen from the above table, a USB bus is capable of delivering only YUV9 subsampled video data at low resolution at the rate required for acceptable full-motion video. A system and method are needed, therefore, for transmitting digital video signals from a digital video camera at a rate sufficient to reproduce full motion video over transmission media having bandwidth constraints.

SUMMARY OF THE INVENTION

An image is captured by a camera which has a sensor for generating a plurality of analog video signals in response to the image. The analog video signals are converted into a plurality of raw digital video signals defining the image. The raw digital video signals are transmitted to a computer over a bus. In an embodiment, the resolution of the image defined by the raw digital video signals is decreased prior to being transmitted over the bus.

BRIEF DESCRIPTION OF THE DRAWINGS The following detailed description will be more fully understood with reference to the accompanying drawings in which:

F g. 1 is a block diagram of a prior art digital camera to PC via bus coupling; F g. 2 is a block diagram of the digital camera to PC via bus coupling of the invention; F g. 3a is a horizontal element of a downsampling filter; F g. 3b is a vertical element of a downsampling filter; F g. 4 is a block diagram of an output of a CCD sensor in a primary color layout; F g. 5a is another embodiment of a horizontal element of a downsampling filter; F g. 5b is another embodiment of a vertical element of a downsampling filter; and F g. 6 is a block diagram of an output of a CCD sensor in a secondary color layout.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Referring to Fig. 2, an embodiment of the invention is shown. As is evident from Fig. 2, digital video camera 20 does not include any DSP hardware. Any necessary transformation of raw digitized CCD video data to NTSC (YIQ) or PAL (YUV) format preferably is done using appropriate software running on PC 28. In an embodiment PC 28 further performs additional manipulations of the video data as discussed further below. Video camera 20 has lens 21 for focusing on a scene or the like denoted by optical path

22. CCD sensor 24 generates an analog video signal which is passed to correlated double sampling and automatic gain control device 23 for conditioning. The conditioned raw CCD analog video signal is converted into a raw CCD digital video signal by A/D converter 26. Camera 20 also preferably includes microcontroller 25.

Color filters are preferably placed over CCD 24 to generate color analog video signals. As noted above, the color filters may be in primary or secondary color layouts. When using a primary color layout, the sensor output typically produces alternating lines of red, green, red, green, red, green . . . (R,G,R,G,R,G...) and green, blue, green, blue, green, blue . . . (G,B,G.B,G,B...) colors represented as analog color video signals. The raw analog color signals from CCD 24 are converted into raw digital signals by analog-to-digital (A D) converter 26.

In one complimentary color layout, CCD 23 typically produces alternating lines of G+Cy, Mg+Ye. G+Cy, Mg+Ye, G+Cy, Mg+Ye . . . and Mg+Cy, G+Ye, Mg+Cy, G+Ye, Mg+Cy, G+Ye

. . . color video signals. The analog color video signals are digitized by A/D converter 26. The raw digital CCD color video signals in a secondary color layout are further processed such as to convert them into the RGB color space. In one embodiment, camera 20 includes the necessary processing hardware and/or software required for converting the secondary colors into an RGB format of video data. In another embodiment, PC 28 converts the raw digital CCD video data in complimentary color layout to RGB data, for example under software control.

For example, in the latter embodiment, digital camera 20 is coupled through a bus having limited bandwidth, for example USB bus 27 to PC 28. Digitized raw CCD sensor video data from A D converter 26 is transmitted over bus 27 for processing prior to display or storage. For example, PC 28 preferably generates raw luminance and color difference signals from the raw digital CCD signals in the complimentary color layout. PC 28 converts the color difference signals to RGB values which then can be transformed into a composite video format, such as NTSC (YIQ) or PAL (YUV). Additional processing may be performed by PC 28 on, for example, the RGB color signals, such as white balancing and gamma correcting. The video data may be displayed, for example on a video monitor associated with PC 28 (not shown in Fig. 2).

For a further disclosure of the conversion of raw digital CCD color signals in a complimentary color layout to RGB signals, and processing of RGB color signals, see commonly owned U.S. Application No. 08/520,61 1, filed on August 30, 1995, incoφorated herein by reference. The raw digitized CCD video data transmitted over bus 27, whether in a primary or secondary color layout, has 8 bits of color information per pixel. The 8 bit size of each raw digitized CCD color signal is of course less than video data transmitted in, for example, YUV9, YUV 12 or YUV 16 formats, which effectively have 9, 12 and 16 bits per pixel, respectively. However, even though the raw digitized CCD video information has fewer bits per pixel than any of the YUV formats, it has more color information in those 8 bits than any of the YUV formats, which have a single pair of 8 bit color difference numbers, U and V, shared by 16, 4 and 2 pixels, respectively. It can be seen, therefore, that the conversion of the RGB color signal into a YUV or YIQ format by DSP 16 in conventional systems as shown in Fig. 1 inefficiently generates redundant information which nevertheless must be transmitted over bus 19, thereby occupying valuable bus bandwidth.

In a preferred embodiment shown in Figs. 3a and 3b, the digital video signals which represent raw CCD digital video signals are subjected to low-pass or downsampling filtering by filter 30 to decrease the resolution of the video image being transmitted over bus 27.

Downsampling as herein described further decreases the bit-rate and enhances an ability to use a limited bandwidth bus 27 for transmission of video data to, for example, a PC at a sufficiently high refresh rate.

The horizontal position 31 of a combination downsampling filter 30 is shown in Fig. 3a. A/D converter 26 of camera 20 having primary color layout produces alternating lines of

RGRGRG and GBGBGB digital video data, representing raw sensor output as shown, for example, in Fig. 4. The digital video data is horizontally downsampled by horizontal downsampling filter 31 , which decreases by approximately 33% the horizontal resolution of the video image. For example pixel element 0,0 is multiplied by 0.5 (or divided by 2) by multiplier 32.1 and the result is passed to adder 33.1. Similarly, pixel element 0,2 also is multiplied by 0.5

(or divided by 2) by multiplier 32.3 and the resultant value is transmitted to adder 33.1. Multipliers 32.1 and 32.3 may be hardware devices, for example a shifter, for shifting right the digital value of pixel elements 0,0 and 0,2. In another embodiment, the multiplication or division may be implemented in software and under processor control. Pixel element 0,1, which is a green pixel element is multiplied by multiplier 32.1 having a coefficient of 0 so that the red pixel elements are separately downsampled.

Adder 33.1 combines the values from multipliers 32.1, 32.2 and 32.3 to provide a single digital value 34 representing downsampled pixel elements 0,0 and 0,2.

Similar downsampling filtering is performed on groups of pixel elements 0,2 - 0,4; 0,4 - 0.6; 0.6 - 0,8 . . . for the entire first horizontal row 0 as shown in Fig. 3a. The entire filter matrix is then shifted a single pixel element to the right and the green pixel elements of the first horizontal row are downsampled. For example, pixel element 0, 1 is multiplied by 0.5, pixel element 0,2 is multiplied by 0 and pixel element 0,3 is multiplied by 0.5. The results of the multiplications (or divisions as described above) are added to provide, for example, a single digital value representing downsampled pixel elements 0,1 and 0,3. Downsampling filtering is correspondingly performed on pixel element groups 0,3 - 0,5; 0,5 - 0,7; 0,7 - 0,9... for the entirety of row 1.

After downsampling horizontal row 0, rows 1 - n are downsampled in an identical fashion.

After performing horizontal downsampling, a preferred embodiment of the invention vertically downsamples using a vertical downsampling filter, such as vertical downsampling filter 39 of downsampling filter 30, for example to decrease by approximately 33% the vertical resolution of the image to be transmitted over bus 27. An embodiment of a vertical downsampling filter 39 is shown in Fig. 3b. Pixel elements 0,0, 1 ,0, and 2,0 are multiplied by multipliers 35.1, 35.2 and 35.3, respectively, having coefficients of 0.5, 0 and 0.5, and the resulting values are added by adder 36.1 to provide a single digital value 37. As discussed above, an alternative to multiplying by 0.5 is to divide by 2, a mathematical operation easily accomplished in digital electronics by shifting right.

The value from adder 36.1 is the downsampled value of pixel elements 0,0 - 2,0 and more particularly the red pixel elements 0,0 and 2,0. After downsampling all of the red pixel elements in column 0, the entire filter matrix is shifted down one pixel element and downsampling is run on pixel groups 1,0 - 3,0; 3,0 - 5,0; 5,0 - 7,0; 7,0 - 9,0, which has the effect of isolating and downsampling the green pixel elements in column 0. After down sampling column 0, downsampling is performed on columns 1 - n in the manner described above.

Downsampling filters of varying configurations are possible, as is evident to a person of ordinary skill relying on this disclosure, to decrease the resolution of an image output by digital camera 20 by varying percentages. For example, Figs. 5a and 5b show downsampling filter 50 having horizontal and vertical elements 51 and 59 respectively, for further reducing, as compared to the filter of Fig. 3, the resolution of an image in two dimensions. Downsampling filter elements 51 and 59 have multipliers with coefficients of 1/3 and 0 for filtering colors independently, in the manner described above with reference to Figs 3a and 3b. A decrease in resolution typically enables an increase in a frame refresh rate.

A digital camera 20 having a secondary color layout will generate the color pattern shown in Fig. 6. In a preferred embodiment of the invention, downsampling is performed on a complimentary color layout pattern in the same manner as described above with reference to a digital video camera having a primary color layout. Any necessary conversion from a secondary color layout to, for example, a linear RGB format and an NTSC or PAL format preferably occurs in PC 28 after the downsampled, raw, digitized secondary color video signals are transmitted over bus 27. Conversion to RGB can be done under software control as will be understood by those skilled in the art, particularly when referring to Application No. 08/520,61 1, filed on August 30, 1995, and incoφorated herein by reference.

The invention takes advantage of the smaller number of bits per pixel of video data associated with the digitized, raw output from a CCD of a digital camera as compared to the bits per pixel of an identical signal which has been converted to a YUV or YIQ format. The invention enables transmission of digital video data from a digital camera to a PC through a bus having limited bandwidth by moving any YUV or YIQ conversion steps out of the camera and into a PC, where they can be done by a high-speed PC microprocessor efficiently and without occupying many processor cycles. Advantageously, any other video data manipulations which are required also can be performed by the PC. For example, software running on an INTEL

PENTIUM processor operating at 100 MHz was able to do the work needed to convert digitized CCD sensor signals into high quality video signals suitable for video conferencing and other video processing applications. At typical video conferencing resolution and frame rate, the CPU workload was less than 5% of the 100 MHz PENTIUM cycles. By eliminating a need for sophisticated processing hardware in a digital video camera, the invention provides the additional advantage of enabling the manufacture of less expensive digital video cameras.

It should be understood that various changes in the details, materials, and arrangements of the parts which have been described and illustrated in order to explain the nature of this invention may be made by those skilled in the art without departing from the principle and scope of the invention as expressed in the following claims.

Claims

CLAΪ S What we claim is:

1. A digital video camera, comprising:

(a) means for generating a plurality of raw analog video signals in response to an image; (b) means for converting the raw analog video signals into a plurality of raw digital video signals defining the image; and

(c) means for transmitting the raw digital video signals to a computer over a bus.

2. The camera of claim 1 , further comprising: (d) means for decreasing the resolution of the image defined by the raw digital video signals.

3. The camera of claim 2, wherein means (a) comprises a charge coupled device.

4. The camera of claim 3, wherein means (a) further comprises a color filter.

5. The camera of claim 4, wherein the color filter includes one of the sets of colors of red, green and blue, and magenta, yellow, cyan, and green.

6. The camera of claim 2. wherein means (b) comprises at least one analog-to-digital converter.

7. The camera of claim 2, wherein the bus has a bandwidth of not greater than 12 Mbits per second.

8. The camera of claim 2, wherein means (d) comprises a downsampling filter.

9. The camera of claim 8. wherein means (d) further comprises:

(i) a horizontal downsampling filter to enable a reduction of resolution in a horizontal dimension; and (ii) a vertical downsampling filter to enable a reduction of resolution in a vertical dimension, and wherein each of the horizontal and vertical downsampling filters includes coefficients of 0.

10. The camera of claim 2, wherein the computer has processor means for converting the plurality of raw digital video signals into a plurality of usable digital video signals.

1 1. The camera of claim 10, wherein the usable digital video signals are one of YUV and YIQ signals.

12. A method for generating digital video signals using a digital video camera, comprising the steps of:

(a) generating a plurality of raw analog video signals in response to an image; (b) converting the raw analog video signals into a plurality of raw digital video signals defining the image; and

(c) transmitting the raw digital video signals to a computer over a bus.

13. The method of claim 12, further comprising the step of (d) decreasing the resolution of the image defined by the raw digital video signals.

14. The method of claim 13, wherein step (a) comprises the step of focusing the image onto a charge coupled device.

15. The method of claim 14, wherein the charge coupled device is overlaid with one or more color filters.

16. The method of claim 15 wherein the one or more color filters include one of the sets of colors of red, gTeen and blue, and magenta, yellow, cyan, and green.

17. The method of claim 13, wherein step (b) comprises the step of processing with at least one analog-to-digital converter.

18. The method of claim 13, wherein the bus has a bandwidth of not greater than 12 Mbits per second.

1 . The method of claim 13, wherein step (d) comprises the step of downsample filtering.

20. The method of claim 19, wherein step (d) further comprises the steps of:

(i) horizontal downsample filtering to enable a reduction of resolution in a horizontal dimension; and

(ii) vertical downsample filtering to enable a reduction of resolution in a vertical dimension, and wherein each of the horizontal and vertical downsample filtering uses coefficients of 0.

21. The method of claim 13, wherein the computer comprises a processor for converting the plurality of raw digital video signals into a plurality of usable digital video signals.

22. The method of claim 21, wherein the usable digital video signals are one of YUV and YIQ signals.

23. A device for processing video signals, comprising: (a) means for receiving over a bus a plurality of raw digital video signals defining an image; and

(b) means for converting the plurality of raw digital video signals into a plurality of usable digital video signals, wherein the raw digital video signals are a digitized version of raw charge coupled device output signals.

24. The device of claim 23, wherein the plurality of raw digital video signals have been generated by a digital video camera equipped with means for decreasing the resolution of the image defined by the raw digital video signals.

25. The device of claim 23, wherein the bus has a bandwidth of not greater than 12

Mbits/second.

26. The device of claim 23, wherein the usable video signals are one of YUV and YIQ signals.

27. The device of claim 23, wherein the digital video camera has an analog to digital converter and the raw digital video signals generated by the camera are a raw analog output of the charge coupled device which has been processed by the analog to digital converter.

28. The device of claim 23, wherein means (b) comprises a computer processor.