WO2001001675A2 - Camera video a fonctions principales mises en oeuvre dans un logiciel hote - Google Patents

Camera video a fonctions principales mises en oeuvre dans un logiciel hote Download PDF

Info

Publication number
WO2001001675A2
WO2001001675A2 PCT/US2000/018046 US0018046W WO0101675A2 WO 2001001675 A2 WO2001001675 A2 WO 2001001675A2 US 0018046 W US0018046 W US 0018046W WO 0101675 A2 WO0101675 A2 WO 0101675A2
Authority
WO
WIPO (PCT)
Prior art keywords
data
video camera
camera
pixel
value
Prior art date
Application number
PCT/US2000/018046
Other languages
English (en)
Other versions
WO2001001675A3 (fr
Inventor
Mark Hsu
Mitchell Norcross
Georges Auberger
Remy Zimmermann
Sergio Maggi
George Sanchez
Bryed Billerbeck
Wei Li
Junien Labrousse
Original Assignee
Logitech, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from US09/343,934 external-priority patent/US6580828B1/en
Priority claimed from US09/345,167 external-priority patent/US6833862B1/en
Priority claimed from US09/464,364 external-priority patent/US7009644B1/en
Priority claimed from US09/602,547 external-priority patent/US6704359B1/en
Application filed by Logitech, Inc. filed Critical Logitech, Inc.
Priority to DE20080319U priority Critical patent/DE20080319U1/de
Publication of WO2001001675A2 publication Critical patent/WO2001001675A2/fr
Publication of WO2001001675A3 publication Critical patent/WO2001001675A3/fr

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H04N25/61—Noise processing, e.g. detecting, correcting, reducing or removing noise the noise originating only from the lens unit, e.g. flare, shading, vignetting or "cos4"
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/16—Analogue secrecy systems; Analogue subscription systems
    • H04N7/162—Authorising the user terminal, e.g. by paying; Registering the use of a subscription channel, e.g. billing
    • H04N7/165—Centralised control of user terminal ; Registering at central
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/414—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance
    • H04N21/4143—Specialised client platforms, e.g. receiver in car or embedded in a mobile appliance embedded in a Personal Computer [PC]
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/80—Generation or processing of content or additional data by content creator independently of the distribution process; Content per se
    • H04N21/83—Generation or processing of protective or descriptive data associated with content; Content structuring
    • H04N21/84—Generation or processing of descriptive data, e.g. content descriptors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/60—Noise processing, e.g. detecting, correcting, reducing or removing noise
    • H04N25/68—Noise processing, e.g. detecting, correcting, reducing or removing noise applied to defects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/47—End-user applications
    • H04N21/482—End-user interface for program selection

Definitions

  • the present invention relates to video cameras, and in particular to video cameras connected to a computer or other intelligent device by a shared interface, such as the universal serial bus (USB).
  • a shared interface such as the universal serial bus (USB).
  • Video cameras for providing both still pictures and motion pictures have been developed which can connect to a computer to provide a digitized image.
  • Such cameras can be connected to the computer via a shared bus, such as the USB. This limits the bandwidth of data that can be sent, requiring compression of the video data.
  • cameras will do some of the preprocessing of the image in the camera electronics, thus both off-loading processing from the CPU of the computer and potentially reducing the amount of data that has to be compressed and sent over the bus.
  • some functions are required to be done in the camera. For example, a CCD or CMOS sensor in the camera may have defective detector positions which do not properly collect light.
  • FIG. 1 is a block diagram of a typical video camera which can connect over a bus to a host.
  • a lens 10 focuses light onto a sensor 12 with associated control logic for reading out the sensor data.
  • the data is provided to an analog-to-digital converter (ADC) 14, where it is digitized.
  • ADC analog-to-digital converter
  • the data is then stored in a frame buffer memory 16.
  • Pixel correction can be applied to the data with a circuit 18.
  • Correction data may be stored in a programmable memory 20. This memory may be programmed during the manufacturing process, before the camera is even shipped.
  • a number of other functions are typically performed in the camera, shown in Fig. 1 in a particular order, although they may be in other orders or connected via a bus to vary when they are invoked.
  • a color processing circuit 22 may perform a number of color functions, such as converting the received data from one color format, such as YUV, to another format, such as RGB.
  • a sealer function 24 can scale the image to fit the display desired at the host.
  • a compression circuit 26 will compress the data to reduce the amount of data to be transferred over the shared bus.
  • a bus interface circuit 28 implements the protocols of a bus 30 for transferring data to a remote host.
  • Huffman coding is one of the algorithms most widely used to compress video data. Huffman coding has been used in various image and video compression standards such as the standards specified by the Joint Photographic Experts Group (JPEG), Motion Picture Experts Group (MPEG), MPEG-LI, H.261, H.263, and H.323. These standards are included herein by reference, in there entirety and for all purposes. Huffman coding is a lossless entropy coding technique. Huffman coding utilizes probability to select the shortest code word for the data that occur most frequently. For example, when encoding English text, "E" which is a very popular character in the English language can be represented by a 2-bit code word, whereas "A" which is not used as frequently can be represented by an 8-bit code word.
  • Huffman coding compresses data by assigning shorter code words to more frequently occurring data and longer code words to less frequently occurring data.
  • the assigned code words can be maintained in a table which is then used by both recipients and senders. Further details regarding Huffman encoding can be found in "A Method for the Construction of Minimum Redundancy Codes," Proceedings of the IRE, Vol. 40, No. 9, September 1952, pp. 1098-1101, by D. A. Huffman, which is incorporated herein by reference in its entirety and for all purposes.
  • Huffman code words are constructed in such a way that no code word is the prefix of another code word. This assures the unambiguous decoding of each symbol from a bitstream of data despite the variable lengths of the codes. Additional description of the characteristics of Huffman codes can be found in Data Compression Book, M&T Books, 1996, by M. Nelson and J. L. Gailly, which is also incorporated herein by reference in its entirety and for all purposes. On pages 31-35, this reference proposes a decoding algorithm based on tree-tracing. This algorithm is, however, not suitable for fast decompression using most of the currently available microprocessors such as Intel ® 's Pentium ® family.
  • Patent No. 4,884, 140 shows an analog circuit for providing vignetting compensation for a video camera using a zoom lens which causes vignetting.
  • Patent No. 5,434,902 shows measuring the vignetting effect for an x-ray examination apparatus by using a constant brightness image. A correction factor is then stored in the memory for each pixel.
  • Patent No. 5,576,797 shows the detection of vignetting effects in a camera with a focus detecting device.
  • Patent No. 5,381,174 shows the correcting of vignetting due to operation of a zoom lens by using a field frequency sawtooth signal.
  • Patent No. 4,816,663 shows the detection of vignetting between a photo taking lens and focus detecting optical system.
  • Digital cameras for use with a personal computer for personal teleconferencing have become cheaper and cheaper. This puts pressure on the camera manufacturers to use cheaper lenses, which have more of a vignetting effect. In addition, the price pressure forces manufacturers to use fewer semiconductor chips, thus making the use of semiconductor memory to store vignetting corrections undesirable.
  • the present invention provides a low cost camera by implementing the major functions in host software. This is accomplished by sending raw, digitized data from the camera directly to the host. The increased volume of raw data is handled by either an improved compression/decompression scheme using lossless compression, using lossy compression or using a shared bus with higher bandwidth. By moving such functions as color processing and scaling to the host, the pixel correction can also be moved to the host. This in turn allows the elimination of the frame buffer memory from the camera. Finally, the camera can use a low cost lens by implementing vignetting correction with a vignetting correction value stored in a register of the camera for later access by the host to perform corrections.
  • the host decompresses the transmitted data by using a processor with the capability of simultaneous operations on multiple packed pixel values, such as the Intel MMXTM technology. This maintains a sufficient decompression speed for a larger amount of data with minimal impact on the frame rate.
  • a group of bits from the data stream are duplicated and provided to multiple positions in a register, where they can be simultaneously compared to multiple maximum, values. This allows a quick determination of how many bits of the variable bit encoding correspond to a pixel value.
  • vignetting, gamma, distortion or aliasing correction and pixel correction are performed in the camera itself, since the ability to correct would be degraded by the loss of information during the lossy compression/decompression process.
  • the color processing, scaling and other operations are still performed in the host, achieving a low cost camera which does not require a frame buffer memory and color processing and scaling circuitry.
  • the camera can be made low-cost by using a low-cost lens even though it has vignetting distortion.
  • a memory element such as a register
  • This memory element is programmed at the time of manufacture .with a value corresponding to the amount of vignetting or correction required.
  • the register can then be read by the host during operation to determine the amount of correction required in a vignetting correction algorithm executed in the host on the received data prior to any other processing.
  • the memory element can store a correction or value factor for other defects in the lens or other aspects of the camera. For example, an indication of bad pixel sites could be stored for subsequent reading and correction by the host.
  • FIG. 1 is a block diagram of a prior art video camera.
  • Fig. 2 is a block diagram of one embodiment of the invention with major functions moved to a host.
  • Fig. 3 is a diagram illustrating the operation of Huffman decoding operations in parallel.
  • Fig. 4 illustrates an example of a computer system used to execute methods of embodiments of the present invention
  • Fig. 5 illustrates a simplified system block diagram of a typical computer system 100 used to execute the methods of embodiments of the present invention
  • Fig. 6 is a simplified block diagram of a SIMD system 300 in accordance with an embodiment of the present invention.
  • Fig. 7 illustrates a simplified block diagram of a system 400 ' in accordance with an embodiment of the present invention
  • Fig. 8 illustrates a sequential decoding method 500 for decoding Huffman encoded data in accordance with an embodiment of the present invention.
  • Fig. 9 illustrates a parallel decoding method 600 for decoding Huffman encoded data in accordance with another embodiment of the present invention.
  • Fig. 10 is a drawing of a pixel array and brightness curve illustrating the vignetting effect.
  • Fig. 1 1 is a block diagram of one embodiment of vignetting correction according to the invention.
  • Fig. 12 illustrates a simplified flow chart of a method of detecting and correcting defective pixels according to one embodiment of the present invention.
  • Fig. 13 is a block diagram of a video camera system according to one embodiment of the present invention.
  • Fig. 14 is timing diagram of the pulses provided to a CMOS sensor array in accordance with one embodiment of the present invention.
  • FIG. 2 is a block diagram of one embodiment of the invention.
  • a camera
  • a correction register 34 is added to store a value corresponding to the vignetting of lens 10.
  • the camera connects over a shared bus 30 to host 36.
  • the blocks shown are programming blocks executed by the processor of host 36. These are a decompression block 38, a vignetting correction block 40, a pixel correction block 42, a color processing block 44 and a scaling block 46.
  • a statistics memory 48 which can be a portion of the host memory, for storing statistics information on pixels needing correction.
  • the processor used by host 36 includes the ability to perform operations on multiple packed pixels in a single register in parallel.
  • the Intel MMXTM technology provides a number of instructions for such operations on packed pixels in the microprocessor registers. Accordingly, it is desirable to take advantage of such functions to increase the speed of processing.
  • certain operations do not lend themselves to such simultaneous operation.
  • Huffman compression is often used for video data, and is used in the embodiment of this invention. This is a variable length code compression mechanism. In other words, the number of bits corresponding to a pixel value could be 1, 2, 3, 4, etc., and can vary from pixel to pixel.
  • Fig. 3 is a diagram illustrating in general how the parallel Huffman decoding proceeds.
  • An incoming data stream 50 is illustrated by a number of letters. Each letter is an arbitrary symbol indicating a different number of bits.
  • the first of number of bits, A is duplicated four times and placed into four positions in a first register 52.
  • a second register 54 stores four different masks. The masks enable a comparison with different portions of the bits in bit segment A. For example, the number of bits of A examined could be 1, 2, 3 and 4, respectively. These would then be compared to the maximum values for the Huffman codes for 1, 2, 3 and 4 bits. These four different maximum values are stored in another register 56.
  • FIG. 4 illustrates an example of a computer system used to execute the software of the present invention.
  • Fig. 4 shows a computer system 100 which includes a monitor 104, screen 102, cabinet 108, keyboard 214 (see Fig. 5), and mouse 110.
  • the mouse 110 can have one or more buttons such as mouse buttons 112.
  • the cabinet 108 can house a CD-ROM drive 106 and a hard drive (not shown) which can be utilized to store and retrieve software programs incorporating the present invention.
  • the CD-ROM 106 is shown as removable media, other removable tangible media including floppy disks, tape drives, ZIP ® drives, and flash memory can be utilized.
  • the cabinet 108 can also house familiar computer components (not shown) such as a processor, memory, and the like.
  • Fig. 4 shows a computer system 100 which includes a monitor 104, screen 102, cabinet 108, keyboard 214 (see Fig. 5), and mouse 110.
  • the mouse 110 can have one or more buttons such as mouse buttons 112.
  • FIG. 5 illustrates a simplified system block diagram of a typical computer system 100 used to execute the software of embodiments of the present invention.
  • the computer system 100 can include the monitor 104.
  • the computer system 100 can further include subsystems such as I/O controller 204, system memory 206, central processor 208, speaker 210, removable disk 212, keyboard 214, fixed disk 216, and network interface 218.
  • Other computer systems suitable for use with the present invention may include additional or fewer subsystems.
  • another computer system could include more than one processor 208 (i.e., a multi-processor system) or a cache memory.
  • Arrows such as 220 represent system bus architecture of the computer system 100. However, these arrows 220 are illustrative of any interconnection scheme serving to link the subsystems.
  • a local bus could be utilized to connect the central processor 208 to the system memory 206.
  • an image capture device such as a charge-coupled device (CCD) camera can be connected to the computer system 100 for capturing image data.
  • the image capture device can be connected to the computer system 100 via the same or another bus architecture such as a Universal Serial Bus (USB) and the like.
  • USB can provide plug and play support for more than 100 connected peripherals by using an identification number which is matched against a database of device drivers.
  • the USB can also be connected to the computer system 100 through the I/O controller 204 or the network interface 218.
  • the computer system 100 can be configured to communicate with the Internet via, for example, the I/O controller 204 or the network interface 218.
  • data can be transmitted to and from the computer system 100 by a variety of devices.
  • the computer system 100 shown in Fig. 4 is but an example of a computer system suitable for use with the present invention.
  • Other configurations of subsystems suitable for use with the present invention will be readily apparent to one of ordinary skill in the art.
  • the present invention can be embodied in any microprocessor capable of single instruction multiple data (SIMD) execution.
  • SIMD single instruction multiple data
  • the Intel ® MMXTM microprocessors the Sun ® UltraSPARC with Visual Instructions Set, Advance Micro Device, Inc.'s ® 3DNow!TM, Intel ® MMXTM with streaming SIMD extensions, and the like can be utilized.
  • an implementation using the Intel ® MMXTM will be described.
  • the MMXTM technology was formally introduced in January 1997 to accelerate the CPU demanding multimedia applications.
  • the MMXTM technology provides a set of instructions (57 instructions) introduced by Intel ® Corporation. It can perform addition, subtraction, multiplication, logic, and arithmetic shifts on a unit of 64 bits in the same instruction.
  • the unit of 64 bits can also be treated as 8 bytes, 4 words, 2 double words, or 1 quad word. Instructions for saturation arithmetic and packing/unpacking data are provided as well as those which transfer data between MMXTM registers and integer registers. Further information regarding MMXTM technology can be found in "Introduction to the Intel ® Architecture MMXTM Technology Developer's Manual,” Intel ® (1996), which is incorporated herein by reference in its entirety and for all purposes.
  • Fig. 6 is a simplified block diagram of a SIMD system 300.
  • the system 300 includes an MMXTM processor 302 which presently supports a set of 57 instructions.
  • the MMXTM processor can be implemented within a CPU or can be alternatively implemented in a separate chip.
  • the MMXTM processor 302 receives data from registers 304a-h. Each of the registers 304a-h are 64 bits wide. Each of these registers can hold 8x8, 16x4, 32x2, 64x1 bits of data.
  • the MMXTM processor 302 receives the data from the registers 304a-h and performs the required operations on the data in one cycle. The processed data is then either provided to a memory 306 or back to registers 304a-h.
  • the memory 306 can be implemented within the MMXTM processor 302 or it can alternatively be shared memory. As a result, the MMXTM processor 302 can perform the same operations on different data at the same time, hence it has an SIMD architecture.
  • Fig. 7 illustrates a simplified block diagram of a system 400 using the
  • a camera 402 captures images and provides them to a bitstream conversion block 404.
  • the camera 402 can include circuitry for conversion of analog data to digital data.
  • the camera 402 can also include circuitry to perform data compression and/or encoding.
  • the camera 402 and the bitstream conversion block 404 are connected via connector 406.
  • the connector 406 can be selected from a number of connectors or bus architectures which are well known to those with ordinary skill in the art.
  • the connector 406 is preferably a USB connection. In some implementations, USB provides a 12 Mb/sec bandwidth.
  • Another choice for connector 406 can be Firewire (IEEE 1394) which can provide a bandwidth of 100 Mb/sec, 200, 400, or 800 Mb/sec.
  • USB connection is preferred for some embodiments because it is a powered bus which currently provides up to 500 mA and 5 V.
  • the power provided by USB can be utilized to run the devices connected to the bus, such as the camera 402. Additionally, USB is less costly to implement.
  • USB since USB does not have the same bandwidth as Firewire, USB can be utilized where the data is first compressed prior to transmission on the USB.
  • the camera 402 can include compression circuitry to compress the captured images before sending the data to the bitstream conversion block 404.
  • the camera 402 can be any number of devices for capturing images including a CCD, complementary metal oxide semiconductor (CMOS), and the like.
  • the bitstream conversion block 404 can be configured to convert serially transmitted data into packets of data.
  • the bitstream conversion block 404 can accumulate data for each image frame and send the accumulated data to a decompression block 408.
  • the frame of data can be any size, but is preferably 352x288 pixels.
  • the frame can also be a block of 320x240, 176x144, or 160x120 pixels. In some embodiments, the frame of data can be a block of 640x480 pixels.
  • the decompression block 408 decompresses and/or decodes the data received from the bitstream conversion block 404.
  • the decoding can be in accordance with Huffman coding, arithmetic coding, other types of entropy coding, and the like.
  • the bitstream conversion block 404 can also include buffers for storage of the data received from the camera 402 and the data sent to the decompression block 408.
  • the decompressed data from the decompression block 408 is then provided to a color conversion block 410.
  • the data from the decompression block 408 can be in any format but is preferably in YUV format, where Y is luminance, U is chrominance red (also known as CR), and V is chrominance blue (also known as CB).
  • the conversion block 410 converts the YUV format data to a format suitable for a display 412, including RBG (red, green, and blue).
  • the display 412 can be any output device including a printer, a handheld device, and the like.
  • the system 400 also includes a memory 414 which can provide storage for the display 412, the color conversion block 410, and the bitstream conversion block 404.
  • the memory 414 can be any type of storage such as dynamic random access memory (DRAM), extended output DRAM (EDO DRAM), synchronous DRAM (SDRAM), video ram (VRAM), static ram (SRAM), and the like.
  • DRAM dynamic random access memory
  • EDO DRAM extended output DRAM
  • SDRAM synchronous DRAM
  • VRAM video ram
  • SRAM static ram
  • the bitstream conversion block 404, the color conversion block 410, and the display 412 can have their own local memory.
  • Fig. 8 illustrates a sequential decoding method 500 for decoding Huffman encoded data.
  • the sequential decoding method 500 uses the following four tables to produce the decoded value: MINCODE [IJ - minimum value of code words with length /; MAXCODE [I] - maximum value of code words with length /; HUFFVALfJJ - table of symbol values corresponding to the j" 1 code word; and VALPTRflJ - the index to the start of the list of values in HUFFVAL which are decoded by code words of length I.
  • the minimum length of all code words is MinLength and the maximum length of all code words is MaxLength.
  • the size of tables MINCODE, MAXCODE, and VALPTR is equal to [MaxLength-MinLength+l].
  • the size of HUFFVAL depends on the number of code words, and is denoted as N.
  • the three tables MINCODE, MAXCODE, and VALPTR are used to decode a pointer to the HUFFVAL table for each valid Huffman code.
  • bitstream data is received for decoding.
  • a current code length / is set to MinLength.
  • Step 504 also initializes a variable CODE to NEXTBITS [MinLength] which contains the code words from the bitstream provided by the step 502.
  • a step 506 compares the value of CODE to MAXCODEflJ. If the value of CODE is greater than the value of MAXCODEfl], it is indicated that the current code word has a length larger than I, which was first initialized to MinLength in step 504, and the step 506 is followed by a step 508. In the step 508, the value of I is incremented by one. The step 508 also reads the next bit of the bitstream into CODE.
  • the step 508 updates the value of CODE by performing a shift logic left (SLL) on current value of CODE by one bit, and filling the least significant bit of CODE with the next bit from the bitstream (NEXTBIT). On the other hand, if in the step 506, it is determined that the value of
  • CODE is not greater than the value of MAXCODEflJ, it is indicated that the current code word has a length equal to or less than /, and the step 506 is followed by a step 510.
  • the step 510 decodes the symbol value.
  • the step 510 computes a code value pointer J and then uses J to compute the symbol value.
  • the step 510 sets Jto ⁇ VALPTRfl-MinLengthJ+CODE-MINCODEfl-MinLength] ⁇ .
  • the step 510 then computes the symbol value by setting VALUE to HUFFVALfJJ . Accordingly, the step 510 performs three table look-ups, including VALPTR, MINCODE, and HUFFVAL, to compute the symbol value.
  • the calculated symbol value (VALUE) is then provided to a step 512 to be output.
  • the sequential decoding method 500 is repeated for the next code word.
  • the Huffman table used for encoding and decoding can be customized for each implementation. But, most coding standards provide a default Huffman table. Table 1 below illustrates a recommended Huffman table for luminance DC difference in accordance with the JPEG standard. Further information regarding the JPEG standard can be found in "JPEG - Still Image Data Compression Standard," Appendix A. ISO DIS 10918-1, Requirements and Guidelines, pp. F-26, Van Nostrand Reinhold, 1993, by William B. Pennebaker and Joan L. Mitchell, which is incorporated herein by reference, in its entirety and for all purposes.
  • MinLength is 2 and MaxLength is 16. Code words are further sorted to be grouped in length and in ascending value order in each length group.
  • the values in MINCODE and MAXCODE are signed 16 bit integers.
  • CodeWord represents the actual code words from the bitstream; CodeLength is the length of each code word; HuffCode is the value of each CodeWord; and HuffValue is the Huffman code for each symbol.
  • MINCODE ⁇ ] ⁇ 0, 2, 14, 30, 62, 126, 254, 510 ⁇
  • MAXCODE ⁇ 8] ⁇ 0, 6, 14, 30, 62, 126, 254, 510 ⁇
  • VALPTR[S] ⁇ 0, 1, 6, J, 8, 9, 10, 11 ⁇ .
  • the step 504 will set /to 2 and CODE to "01."
  • the step 506 will return YES because "01" is greater than MAXCODEf2-2J which is 0.
  • / is incremented to 3 and CODE is set to "010.”
  • the step 506 is repeated again and this time it will return NO because "010" is less than MAXCODE [3-2] which is 6.
  • the step 510 ill set J equal to 1.
  • the step 510 also looks up the symbol value for HUFFVAL f 1] and outputs this symbol value (VALUE) in the step 512.
  • the sequential method 500 repeatedly shifts the bitstream into CODE and compares the current code with the maximum code of the same length. Once code length is known, the decoding of each value needs two additions and three table look-ups. This algorithm is sequential in nature because code lengths are checked sequentially. That is, before checking length /, there is no indication of whether the actual code length will be 1, 1+1, 1+2, etc.
  • Fig. 9 illustrates a parallel decoding method 600 for decoding Huffman encoded data in accordance with an embodiment of the present invention.
  • an implementation of the parallel decoding method 600 such as discussed with respect to Fig. 6, will be described.
  • sample JPEG values discussed with respect to Fig. 8 and Table 1 are utilized in the sample embodiment.
  • the parallel decoding method 600 can, however, be performed on any processor capable of SIMD executions including Intel ® MMXTM microprocessors, the Sun ® UltraSPARC ® with Visual Instructions Set, Advance Micro Device, Inc.'s ® 3DNow!TM, Intel ® MMXTM with streaming SIMD extensions, and the like.
  • MMXTM technology can perform operations on a unit of 64 bits in the same instruction.
  • the unit of 64 bits can also be treated as 8 bytes, 4 words, 2 double words, or 1 quad word.
  • the 64 bits can be treated as four 16-bit registers (or 4 words) because the maximum code word length in Table 1 is 9.
  • the MMXTM instructions PSRL, PSLL, and PSRA are utilized to shift the contents of selected variables.
  • PSRL denotes pack shift right logic.
  • PSLL denotes packed shift left logical and PSRA denotes packed shift right arithmetic.
  • the parallel ⁇ decoding method 600 receives the bitstream in a step 602.
  • bitstream is arranged in 4-bit portions (or quadbits) with little-endian format.
  • the original bitstream is of the form b0,bl,b2,b3, ... (where bi stands for byte ), then the converted bitstream will be of form: bl,b0,bl,b0,bl,b0,bl,b0 ,bl,b0 ,b3,b2,b3, b2,b3,b2,b3,b2,b3,b2, ...
  • a step 606 values of/, MASK, and CODE4 are initialized. As discussed above for the JPEG example, /is initialized to 2.
  • the step 606 initializes CODE4 to the first set of 4x16-bit codes (or quadbits) from the bitstream.
  • the parallel decoding method 600 uses MASK to mask out the bits for a comparison operation in a step 608.
  • the initial value for MASK is set to "1100,0000,0000,0000,1110,0000,0000,0000,1111,0000,0000,0000,1111, 1000,0000,0000 b" or "0xC000,E000,F000,F800.” This is the bit mask for code lengths 2 through 5 which will be checked in the first iteration of the parallel decoding method 600.
  • CODE4 is masked off by the value of MASK.
  • the step 608 also sets MAXCODE4 to MAXCODE4_PfI ⁇ I+3] which represents values of
  • MAXCODE4JP for /, 1+1, 1+2, and 1+3. Accordingly, MAXCODE4 will hold the maximum values for the four successive code lengths. Because the four values for MAXCODE4 will be loaded in one single instruction, the MAXCODE table for the parallel decoding method 600 can be generated as follows:
  • MAXCODE_P are unsigned 16-bit integers.
  • the MAXCODE table only has to be generated once for each Huffman table.
  • MAXCODE ⁇ P ⁇ ] ⁇ 61441, 57345, 49153, 1, 65281, 65025, 64513, 63489 ⁇
  • the dimension of this table must be a multiple of 4. Also, zero is assumed for those code lengths that do not exist in the Huffman code table.
  • a step 610 the value of MAXCODE4 is compared with CODE4 which was initialized and masked in the steps 606 and 608.
  • this comparison operation can be performed by one unsigned substraction with saturation and one comparison with zero.
  • the values o ⁇ MAXCODE+1 can be stored in MAXCODE4. If CODE4 is found to be greater than MAXCODE4, then in a step 612, / is incremented by 4 and the MASK is updated by shifting its value right arithmetically by 4 bits. The new value of MASK will create the bit mask for the next remaining 4 code word lengths (6 through 9).
  • the steps 608 and 610 are repeated. Accordingly, for the example of Table 1, the steps 608 and 610 are repeated twice, once for code word lengths 2 through 5 and once for code word lengths 6 through 9.
  • a step 614 determines which one of the four lengths being checked contains the symbol. Accordingly, the step 614 computes a value of DIF which is set to [MAXCODE4-CODE4-1] shifted logically right by 4 bits. The step 610 also performs a table look-up to provide the found symbol value (VALUE). The step 614 looks up the symbol value (VALUE) in a table HUFFVAL_P.
  • a HUFFVAL_P table can be created using [I+l6*(MAXCODEfI]-CODE)] as index.
  • the HUFFVAL J 1 table will have a size of [16*(M+2)].
  • COENUMfl denote the number of code words whose code length is (1+ MinLength).
  • CODEfJ denotes the code value of the J code with code length /.
  • the entries of the large HUFFVAL table can be computed as follows:
  • values in the HUFFVAL_P table will be the maximum size of unsigned 16-bit integers depending on the number of the Huffman codes.
  • a step 616 the calculated symbol value (VALUE) is output.
  • the current 4x16-bit code word can be left shifted and the same number of bits from the next 4x16-bit code word can be shifted in. The decoding of next Huffman code can then be started.
  • the parallel decoding method 600 reads in 4 bits of data at a time from the bitstream. Each consecutive 4-bit lengths are checked in parallel to see if the current code word falls in that range. Instead of the required 16 checks to cover the range 1 through 16 in the sequential algorithm, only 4 checks are needed to cover the same range. Another significant improvement is to combine the three table look-ups for decoding into one single table look-up. This has the effect of minimizing computations as well as register usage, which can be very valuable in many situations. SAMPLE EMBODIMENT FOR LENGTHS 1 THROUGH 8
  • the decoding can be done on units of 8 bits. A maximum of one check instead of 8 is needed for each code word. This requires that the bitstream be expanded into 8x8 bits. For example, if the input bitstream is "b0,bl, -- (where bi stands for byte i), the converted bitstream will be "b0,b0,b0,b0,b0,b0,b0,b0,b0,b0,b0,b 1 ,b 1 ,b 1 ,b 1 ,b 1 ,b 1 ,b 1 ,b 1 ,b 1 ,b 1 ,b 1 ,b 1 ,b 1 ,b 1 , ... " The initial bit mask will now be " 10000000, 11000000, 11100000,
  • HUFFVAL_P table will be unsigned 8-bit integers.
  • MAXCODE _P table will be unsigned 8-bit integers.
  • bitstream will be extended into 2x32 bit form as follows: Input bitstream: b0,bl,b2,b3,b4,b5,b6,b7,...
  • the initial bit mask will now be "10000000,00000000,00000000, 00000000,11000000,00000000,00000000,00000000,00000000 b" or "0x8000,0000, C000,0000.”
  • the table entries will also be computed as follows:
  • MAXCODE _PfI] (2 2 - mn ngth - r) *MAXCODE[J ⁇ ) + 1
  • HUFFVAL_P[I+32*(MAXCODEfI]+l-CODEfJ])] HUFFVAL[VALPTR[I]+CODE[J] -MINCODE f I] ⁇
  • the values for MAXCODE JP table will be unsigned 32-bit integers.
  • Values of table HUFFVAL_P will be the maximum size of unsigned 32-bit integers.
  • DCT differential parse code modulation
  • wavelet wavelet
  • sub-band transform vector quantization
  • MMXTM 166 MHz Intel ® Pentium ® processor with MMXTM
  • Table 2 demonstrates that the parallel algorithm significantly reduces the worst case timing while giving superior performance for the average case. Also, note that even better results may be achieved by optimizing the assembly code further.
  • the present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof.
  • the techniques of the present invention can also be applied to other SIMD systems such as the Sun ® UltraSPARC ® with Visual Instructions Set, Advance Micro Device, Inc.'s ® 3DNow!TM, and the like can be utilized.
  • Intel ® ' s new Pentium ® III also know as Katmai
  • Pentium ® II adds streaming SIMD extensions.
  • Pentium ® III provides 70 new instructions. Many instructions are for floating point SIMD operations which are targeted at 3-D applications. Several instructions for fixed point number operations have also been added.
  • the techniques of the present invention can be implemented in a computer system.
  • the methods 500 and 600 can be implemented on a peripheral component interconnect (PCI) card.
  • PCI peripheral component interconnect
  • the PCI card can be installed onto the PCI bus of a personal computer.
  • other bus technologies such as NUBUS, ISA, EISA, Universal Serial Bus (USB), 1394 Firewire, and Accelerated Graphics Port (AGP) can also be utilized.
  • the techniques of the present invention can be implemented by utilizing the available routines and features such as caching, new instruction sets, multi processor systems, and their equivalents.
  • Register 34 of Fig. 2 allows a vignetting correction to be applied, thus allowing a cheaper lens 10 to be used in camera 32. This allows the cost of the camera to be driven even lower.
  • the processor in host 36 will interrogate the camera to read the value of register 34. This will then be used in the vignetting correction block 40 of the host to correct for vignetting defects of the camera. This is done prior to pixel correction of block 42, to avoid vignetting being mistaken as a defective pixel.
  • the correction can be accomplished by adding an offset value to the pixel brightness value provided.
  • Fig. 10 illustrates a pixel array 11 and a corresponding brightness curve 13 over the Y axis.
  • a central pixel 17 with value (x 0 , yo) corresponds to the point of maximum brightness of the lens.
  • a second example pixel 15 is near the minimum brightness, showing a vignetting effect of having the difference between curve 13 and a maximum brightness value 19. The present invention corrects this vignetting effect in both X and Y directions.
  • a threshold indicated by r ⁇ is illustrated within which the curve 13 is near the maximum value 19, and no vignetting correction need be applied.
  • the correction can be applied to all of the pixels regardless to avoid any transition effects at the points rx.
  • vignetting amount can be used to provide a constant in a register 72.
  • different lenses could be used in manufacturing a camera by simply programming a different constant value into register 72. This allows multiple sources to be used for the lenses, and allows compensation for changes in lens quality from lot to lot or manufacturer to manufacturer.
  • the vignetting correction can be done by either hardware/software in the camera, or software in a host.
  • the vignetting defect of a low quality lens may be modeled as pass through in the middle of the lens and a parabolic curve near the edge of the lens.
  • the pixel aspect ratio is not taken into account.
  • the equation is exact, for 12/11 pixel aspect ratio, the result is slightly off.
  • a will be scaled by l A if the sensor format is non-scaled QCTF (subsampled QCIF from CIF by leaving out every other pixels in both horizontal and vertical directions).
  • FIG. 11 is a block diagram of one embodiment of a hardware system or software blocks for implementing the equation described above.
  • an optional selection circuit/block 80 is applied to the select input of a multiplexer 82. Where the pixel position is less than the threshold r 2 , the input pixel pi on line 84 is simply passed through to the output pixel, p 0 , on line 86. If it is greater than the threshold, a vignetting correction is applied by implementing the formula above with the hardware shown in Fig.
  • the current x value of the pixel, x ; is applied to arithmetic circuit/block 88, where the central pixel, x 0 , is subtracted from it. This value is then squared by applying its input twice, through a multiplexer 90, to a multiply circuit/block
  • the y value y is provided to a circuit block 96, where the difference from the center y value, y 0 , is determined.
  • This is similarly applied as two values through mux 90 to a multiplier 92, where it is squared and provided to add/accumulator 94, where it is added to the squared x value.
  • the use of the multiplexer simply allows, by time division multiplexing, the same multiplier 92 to be used for both calculations. Obviously, an alternative would be to provide two separate multiply units.
  • the output of accumulator 94 is then provided to a second multiplier 96, where it is multiplied by the constant value from register 72.
  • the value of one is then subtracted in a unit 98, and this value is multiplied by the pixel value, p;, in a multiplier
  • the output is provided on a line 102 through multiplexer 82 to output line 86.
  • Optional circuit/block 80 provides a comparator 104 which compares the output of add accumulator 94 (the radius value of the x 2 +y 2 values) to the threshold radius in a register 106.
  • the constant a has the same number of bits as the pixel value
  • correction is provided where the image is cropped before the vignetting correction.
  • correction can be done for the pixel aspect ratio by multiplying the y value by 1 1/12 where a TV will be used.
  • the present invention thus allows variation in lenses and also cheaper lenses to be used. It improves the AGC and AWB in the host computer. Additionally, it allows better object motion tracking. Object motion is usually done by assuming the luminance value is the same as the object moves to the edge of a sensor, and thus tracking by looking for the same luminance value. Obviously, vignetting effects can thwart the efforts to locate the object. By applying the vignetting correction close to the sensor, this can be overcome. The invention also provides overall improved video quality.
  • Block 42 of Fig. 2 performs pixel correction in the host. This allows for defective detector locations on the CMOS or CCD sensor array to be corrected. In general, this is accomplished by comparing a brightness value to the brightness value of the neighboring detector elements. If the difference is more than a threshold value, it is assumed that this is due to a defective sensor element. Accordingly, a corrected value will be substituted, typically an average of the surrounding pixel values.
  • the algorithm is kept simple. This is accomplished through two primary features. First, the defective pixel detection is not done on any frame, but only on a subsample of the frames. For example, the detection may be done only every 32-128 frames. Second, the statistics kept are simple, to reduce the amount of processing required. In particular, the statistics may include simply the location of the pixel element, and the frequency or number of times a defective pixel has been detected.
  • the system works by reading off or scanning the raw image data from an image pickup device for pixels that vary more than a specific amount in intensity from their neighboring pixels.
  • the raw image sensor data is the unprocessed brightness data output obtained from the image sensor and which has not gone through any lossy compression or color processing.
  • the image sensor reads analog voltage or current, converts to digital and sends the signal to a host with no further processing or compression.
  • the photosites on the image sensor are used to capture either color or monochrome digital still or video images.
  • the raw image data is sent to the intelligent host over a bus with a data transfer rate which is determined by the bus protocol of the particular bus such as a universal serial bus (USB) or a parallel port.
  • USB universal serial bus
  • the raw image sensor data, the location and frequency of occurrence of each defective pixel, and the results of all intermediate computations performed by the computer program are all stored in memory. Other algorithms are then used to average the values of an anomalous pixel's neighboring pixels to replace the data from the defective pixel.
  • the corrected data can then be further processed and ultimately displayed on the monitor.
  • the process includes video subsampling, meaning that the detection is carried out and repeated at various frame intervals.
  • the video subsampling is carried out on one of every 128 (1/128) frames. Alternately, the video subsampling can be carried out on every 1/64 video frames.
  • the video subsampling is carried out on every l/(n times X) frames, where n is an integer and X is not equal to 50 or 60.
  • 50 and 60 correspond to 50 Hz and 60 Hz, which are AC lighting frequencies used in the United States and Europe respectively. This way, it is ensured that anomalous raw data pixels are not artifacts of the artificial lighting systems.
  • the use of video subsampling allows for rapid and optimum corrections without the need to scan every frame which would adversely impact the processing speed of the processor and the system. Naturally, no video subsampling is employed when detecting and correcting defective pixels in still images.
  • Fig. 12 illustrates a simplified flow chart describing the functioning of the software program implemented in a system for detecting and correcting defective pixels according to one embodiment of the present invention.
  • the first step in defective pixel detection and correction includes acquiring a portion of a frame of raw image data from an image sensor array, step 1210.
  • the raw image data may be data corresponding to a live scene being digitized or it may be data corresponding to a calibration or "black” background image.
  • the "black” background image may be obtained by reducing the integration time of the sensor array or by reducing the gain of the image signal.
  • the raw data from the "black” image can be used to detect over active photosites.
  • the raw data corresponding to the "black” image must also be black, and if any pixel is not, then it corresponds to an overactive photosite.
  • the use of the "black” image can enhance defective pixel detection by removing any brightness deviations that originate from the scene itself.
  • the algorithm begins its function as soon as data from three lines from the image sensor has arrived at the host. Three lines of data from the image sensor array will allow for the processing of data for a pixel and all its surrounding neighbors.
  • video subsampling is used so that not every frame of the video captured by the image sensor array is required for the defective pixel detection operation.
  • defect detection is carried out at predetermined frame intervals as described above and the correction is applied to all video frames. The judicious use of video subsampling allows for the method to be very fast and efficient.
  • no video subsampling is used such that the defective pixel detection is carried out on every video frame.
  • the use of video subsampling is a function of the intelligent host's processor speed. If a processor is fast enough to allow detection of anomalous pixels in every frame, then video subsampling is not used. If, on the other hand, the processor is not fast enough to allow video processing at a desired frame rate, then video subsampling is used to ensure data transfer at that desired frame rate. Without video subsampling, anomalous pixel correction is immediate, such that defects are corrected in the very frame in which they are detected. With video subsampling, anomalous pixel correction is delayed until a frame is sampled to detect anomalous pixels. Therefore, the choice of using video subsampling, and the rate of subsampling are a function of the processor speed and a trade off between processor power and the delay before correction ⁇
  • the local pixel brightness value and also the average brightness value of all its immediately neighboring pixels are computed and stored, step 1212.
  • a deviation threshold value is established, step 1214.
  • the deviation threshold establishes the acceptable level of variance between a pixel's brightness value and the average brightness value of all its immediately neighboring pixels.
  • a local brightness deviation is computed, step 1216.
  • the local brightness deviation is the absolute value of the difference between a pixel's brightness value and the average of the brightness value of all its immediately neighboring pixels.
  • step 1218 For each pixel whose data that has been acquired, its local brightness deviation is compared to the deviation threshold, step 1218. Any pixel whose local brightness deviation exceeds the threshold deviation value is then flagged as a defective pixel.
  • the physical location and the frequency of occurrence of each defective pixel is then recorded in a statistical database, step 1220.
  • the statistical database is then queried, to determine whether the defective pixel's data value should be corrected, step 1222.
  • the statistical database by storing the location and frequency of defective pixels, develops over time trends which confirm which of the defective pixels warranted correction.
  • the logic of the trends from the statistical database initially warrant correction of all flagged defective pixels as a default, and over time warrant pixel correction only if a particular pixel has an occurrence frequency of at least two out of the last four queries.
  • the defective pixels that have passed through the statistical database filter are corrected next, step 1224.
  • the erroneous raw brightness data for a defective pixel is replaced by that pixel's local average brightness value, which is the average brightness value of all its immediately neighboring pixels.
  • the corrected data from the defective pixels as well as data from non-defective pixels is prepared to be sent for subsequent processing, step 1226.
  • Subsequent processing may include compression, color processing and encoding to data formats suitable for display.
  • the defective pixel detection and correction is carried out on the raw data from the image sensor array because it is preferred to correct the data before any subsequent processing has occurred since that processing itself can introduce artifacts which are hard to distinguish from artifacts which have been produced as a result of defective photosites.
  • the software algorithm is intentionally kept very simple so that the processor load and/or the video frame rate is minimally affected.
  • the software algorithm is kept simple because it only performs three functions, namely the detection, correction and statistics functions.
  • the statistics routine is kept simple because only position information and frequency of occurrence of anomalous pixels are tracked.
  • the software algorithm is kept simple so as to have a minimal impact on the rate at which data is transferred to the host, so that while a frame is being scanned for defective pixels, subsequent frames are not held up.
  • the maximum impact of the software algorithm is to at worst reduce the video data transfer rate from 10 frames per second ( ⁇ s) to 9 ⁇ s.
  • the software algorithm is kept simple such that the host processor's load is not increased by more 1 % when executing the algorithm. The anomalous pixels are detected in one frame and the corrections are later instituted on subsequent frames as they are read off the image sensor.
  • the statistical analysis segment of the defect detection and correction algorithm is an optional one. It is aimed at increasing the efficiency of the pixel corrections, so as not to correct anomalies that were not caused by defective photosites, and hence save processing time and load. However, the gains in efficiency must be balanced against the load imposed by the statistical analysis portion itself. In an alternate embodiment not employing the statistical analysis portion, all pixels that get flagged as defective get corrected. As in the choice of using or not using the video subsampling, the decision to employ the statistical analysis portion of the defect detection and correction algorithm depends on a trade off between efficiency and processor power. If the processor is fast enough, then efficiency concerns are not so important.
  • the present invention may be embodied in other specific forms without departing from the essential characteristics thereof.
  • the rate of video subsampling could be varied, or not done at all.
  • the logic of the optional statistical database could be altered from one of correcting defective pixels as a default to one where defective pixels are corrected only if warranted by the trends from the statistical database.
  • a small buffer in the bus interface is used.
  • a small buffer is achievable by controlling the timing of when the sensor is read to correspond to when the host bus is available, and reading only an amount of data which can be buffered until the next bus availability.
  • Fig. 13 is a block diagram of a video camera according to the present invention.
  • This system includes a lens 1310 and a sensor array, such as a CMOS sensor array 1312, and is connected to a USB 1320.
  • a sensor array such as a CMOS sensor array 1312
  • the digitized signals from the CMOS sensor chip are provided directly to a digital processing circuit 1348. They are first provided to a video digital signal processor 1350 which performs the compression, and optionally order functions to minimized data transfer, such as cropping, scaling and digital filtering. Once processed, the digital data is provided to a bus interface 1354.
  • Bus interface 1354 includes a bus controller buffer 1356 and a bus controller 1358.
  • Buffer 1356 stores at least a single line of data from a CMOS sensor row.
  • bus controller buffer 1356 is capable of storing two USB frames at the maximum possible rate, or 2 Kbytes.
  • Bus controller 1358 provides a control signal on a line 1360 to timing and control generator 1326.
  • Timing generator 1326 provides clocking signals on line 1364 to CMOS sensor array 1322.
  • Clocking signals 1364 include the row and column transfer pulses.
  • the column transfer pulses are provided as in the prior art, periodically loading the charged values from the CMOS sensor array.
  • the row transfer pulses are varied to match the reading of the data out of the CMOS sensor array with the processing by the following circuitry and the transferring of data to the USB.
  • the digital processing circuit 1348 is integrated onto the same semiconductor chip substrate as CMOS sensor array 1312, timing generator 1326, ADC 1328, and the row and column logic and buffers.
  • CMOS sensor array 1312 timing generator 1326, ADC 1328, and the row and column logic and buffers.
  • the elimination of the frame buffer allows this single chip. This allows a more compact, less expensive video camera to be built.
  • Fig. 14 illustrates the timing of the row transfer pulses provided on line 1364. Rather than a continuous stream of pulses, the bits are clocked out as needed by the processing circuit in groups as illustrated. A first group of three lines 1466 is issued, and then there is a delay while these pixels are processed and transferred to the bus. Then, a next of group of two lines 1468 may be provided. There might then be a delay depending upon the availability in the line buffer and the timing of transfer of data onto the USB. A single line 1469 is shown being transferred next, followed by a subsequent group of four lines transferred by a number of pulses 1470. After an entire image has been transferred, a vertical transfer pulse 1472 is provided. The example of Fig. 14 is intended to be simply illustrative.
  • the row transfer pulses are provided between the same column pulses, in order to allow the gaps shown, the pulses are provided much closer together.
  • bursts of small groups of pulses are provided to quickly send pixel data as fast as the circuitry can handle it.
  • an entire image may be clocked out in a series of closely spaced pulses, allowing the processing to complete before the next column transfer pulse. If the processing still is not completed at the time of the next column transfer pulse, there can be a delay, which conforms to the timing of the processing and the availability of the USB.
  • the row pulses may all be sequentially provided, but at a faster rate than in the prior art and the timing of this grouping of pulses after the column pulse can be varied to be right after, or just before the next column pulse, in order to accommodate the processing and bus timings.
  • the compression block 26 and decompression block 38 of Fig. 2 can be eliminated. This may be done, for example, when a new USB is developed with higher bandwidth. However, prior embodiments may still be desirable if increased sensitivity sensors are developed, allowing higher resolution video image data to be transmitted, using up the additional bandwidth made available. However, one embodiment takes advantage of the additional bandwidth by eliminating the need for compression, thus further simplifying the video camera and reducing its cost.
  • the same major functions shown in Fig. 2 would be performed in the host, with the exception of the decompression.
  • the camera would still include the correction register 34 in one embodiment, allowing for the inexpensive lens 10 to be used.
  • Lossy compression means that the recovered image may lose some resolution.
  • a disadvantage of lossy compression is that any vignetting or pixel correction done in the host would be degraded. Accordingly, in this alternate embodiment, the vignetting correction and the pixel correction is done in the video camera itself. This allows a lossy compression block to be used for block 26 of Fig. 2. The color processing and scaling operations are still performed in the host, thus providing a simpler camera than the prior art of Fig. 1.
  • the correction register can correct for features of a camera that can vary from camera to camera on manufacture, other than features of the lens.
  • the data can be wirelessly transmitted from the camera to the host, or to a receiver connected to the host.
  • the host itself can be a personal computer, an intelligent Internet device, or any other appliance or component with some sort of processor or processing circuitry.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Security & Cryptography (AREA)
  • General Engineering & Computer Science (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)
  • Studio Devices (AREA)
  • Surface Acoustic Wave Elements And Circuit Networks Thereof (AREA)
  • Color Television Image Signal Generators (AREA)

Abstract

L'invention concerne une caméra bon marché dont les fonctions principales sont mises en oeuvre dans un logiciel hôte. On effectue cette mise en oeuvre en envoyant des données brutes numérisées directement de la camera vers l'hôte. Le volume de données brutes augmenté est manipulé au moyen d'un mécanisme de compression/décompression amélioré par compression sans perte, par compression avec perte ou à l'aide d'un bus partagé à largeur de bande plus élevée. Du fait que des fonctions telles que le traitement de la couleur, et la mise à l'échelle sont déplacées vers l'hôte, la correction de pixel peut également se déplacer vers ledit hôte, ce qui permet ensuite d'éliminer la mémoire tampon de trame de la caméra. Ladite caméra peut, en outre, utiliser une lentille bon marché permettant d'effectuer une correction des effets de vignettage, de distortion, de gamma ou de repliement, à l'aide d'une valeur de correction stockée dans un registre de la caméra afin que l'hôte puisse y accéder ultérieurement pour exécuter des corrections.
PCT/US2000/018046 1999-06-30 2000-06-29 Camera video a fonctions principales mises en oeuvre dans un logiciel hote WO2001001675A2 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
DE20080319U DE20080319U1 (de) 1999-06-30 2000-06-29 Videokamera, bei der die Hauptfunktionen in der Hauptrechnersoftware implementiert werden

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US09/343,934 US6580828B1 (en) 1999-06-30 1999-06-30 Fast decoding
US09/345,167 1999-06-30
US09/345,167 US6833862B1 (en) 1999-06-30 1999-06-30 Image sensor based vignetting correction
US09/343,934 1999-06-30
US09/464,364 1999-12-15
US09/464,364 US7009644B1 (en) 1999-12-15 1999-12-15 Dynamic anomalous pixel detection and correction
US09/602,547 US6704359B1 (en) 1999-04-15 2000-06-21 Efficient encoding algorithms for delivery of server-centric interactive program guide
US09/602,547 2000-06-23

Publications (2)

Publication Number Publication Date
WO2001001675A2 true WO2001001675A2 (fr) 2001-01-04
WO2001001675A3 WO2001001675A3 (fr) 2001-07-05

Family

ID=27502681

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2000/018046 WO2001001675A2 (fr) 1999-06-30 2000-06-29 Camera video a fonctions principales mises en oeuvre dans un logiciel hote

Country Status (3)

Country Link
CN (2) CN2567940Y (fr)
DE (1) DE20080319U1 (fr)
WO (1) WO2001001675A2 (fr)

Cited By (19)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP1447977A1 (fr) * 2003-02-12 2004-08-18 Dialog Semiconductor GmbH Compensation du vignettage
WO2004093438A1 (fr) * 2003-04-17 2004-10-28 Nokia Corporation Telephone mobile a module photographique perfectionne
EP1274224A3 (fr) * 2001-06-28 2004-11-17 Nokia Corporation Procédé et dispositif d'amélioration d'image
EP1530375A2 (fr) * 2003-11-06 2005-05-11 OmniVision Technologies, Inc. Correction d'objectif utilisant des données YUV traitées
NL1023711C2 (nl) * 2002-06-20 2006-02-21 Hewlett Packard Co Werkwijze en inrichting voor correctie van niet-uniformiteit in de kleur van een digitale camera.
NL1023709C2 (nl) * 2002-06-20 2006-02-21 Hewlett Packard Co Werkwijze en inrichting voor correctie van niet-uniformiteit in de kleur van een digitale camera.
CN100367763C (zh) * 2005-07-19 2008-02-06 华晶科技股份有限公司 清除数字影像噪音的方法及装置
KR100848589B1 (ko) 2005-10-11 2008-07-28 노키아 코포레이션 휴대용 카메라폰, 이미지 레코딩 방법 및 카메라 모듈
KR100879832B1 (ko) * 2001-07-12 2009-01-22 도 랩스 영상 처리 수단에 포맷화된 데이터를 표준 포맷에 따라제공하는 방법 및 시스템
US9655189B2 (en) 2013-03-13 2017-05-16 Federal Law Enforcement Development Services, Inc. LED light control and management system
US9654163B2 (en) 2009-04-01 2017-05-16 Federal Law Enforcement Development Services, Inc. Visible light transceiver glasses
US9660726B2 (en) 2007-05-24 2017-05-23 Federal Law Enforcement Development Services, Inc. LED light broad band over power line communication system
US10051714B2 (en) 2007-05-24 2018-08-14 Federal Law Enforcement Development Services, Inc. LED light control assembly and system
US10205530B2 (en) 2013-05-06 2019-02-12 Federal Law Enforcement Development Services, Inc. Network security and variable pulse wave form with continuous communication
US10250329B1 (en) 2007-05-24 2019-04-02 Federal Law Enforcement Development Services, Inc. LED light fixture
US10448472B2 (en) 2015-08-11 2019-10-15 Federal Law Enforcement Development Services, Inc. Function disabler device and system
US11265082B2 (en) 2007-05-24 2022-03-01 Federal Law Enforcement Development Services, Inc. LED light control assembly and system
WO2022174881A1 (fr) * 2021-02-16 2022-08-25 Huawei Technologies Co., Ltd. Compensation de variation d'éclairement de capteur d'événement
US11783345B2 (en) 2014-01-15 2023-10-10 Federal Law Enforcement Development Services, Inc. Cyber life electronic networking and commerce operating exchange

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8188878B2 (en) 2000-11-15 2012-05-29 Federal Law Enforcement Development Services, Inc. LED light communication system
ITMI20030341A1 (it) * 2003-02-26 2004-08-27 Easydur Italiana Di Renato Affri Sistema per controllare la fabbricazione di una molla.
CN101277436B (zh) * 2007-03-26 2010-07-14 创惟科技股份有限公司 多串流式网络摄影机控制系统及其控制方法
US9258864B2 (en) 2007-05-24 2016-02-09 Federal Law Enforcement Development Services, Inc. LED light control and management system
US9294198B2 (en) 2007-05-24 2016-03-22 Federal Law Enforcement Development Services, Inc. Pulsed light communication key
US8103121B2 (en) * 2007-08-31 2012-01-24 Adobe Systems Incorporated Systems and methods for determination of a camera imperfection for an image

Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4774565A (en) * 1987-08-24 1988-09-27 Polaroid Corporation Method and apparatus for reconstructing missing color samples
US5231485A (en) * 1991-11-19 1993-07-27 Scientific-Atlanta, Inc. Method and apparatus for transforming between fixed-rate vector quantized data and variable rate vector quantized data
US5280361A (en) * 1990-07-03 1994-01-18 Canon Kabushiki Kaisha Data processing apparatus
US5327246A (en) * 1991-01-24 1994-07-05 Matsushita Electric Industrial Co., Ltd. Pixel defect removing circuit for solid-state image pickup device
US5436659A (en) * 1991-05-10 1995-07-25 Eastman Kodak Company Method and apparatus for determining defective pixel location
US5475441A (en) * 1992-12-10 1995-12-12 Eastman Kodak Company Electronic camera with memory card interface to a computer
US5477264A (en) * 1994-03-29 1995-12-19 Eastman Kodak Company Electronic imaging system using a removable software-enhanced storage device
US5712680A (en) * 1993-12-24 1998-01-27 Canon Kabushiki Kaisha Image pickup device for obtaining both moving and still images
US5808672A (en) * 1993-10-21 1998-09-15 Hitachi, Ltd. Apparatus for removably mounting an electronic camera to a computer system
US5867214A (en) * 1996-04-11 1999-02-02 Apple Computer, Inc. Apparatus and method for increasing a digital camera image capture rate by delaying image processing
US5907359A (en) * 1995-08-30 1999-05-25 Sanyo Electric Co., Ltd. Highly-integrated image sensing apparatus producing digitized output
US6005613A (en) * 1996-09-12 1999-12-21 Eastman Kodak Company Multi-mode digital camera with computer interface using data packets combining image and mode data

Patent Citations (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4774565A (en) * 1987-08-24 1988-09-27 Polaroid Corporation Method and apparatus for reconstructing missing color samples
US5280361A (en) * 1990-07-03 1994-01-18 Canon Kabushiki Kaisha Data processing apparatus
US5327246A (en) * 1991-01-24 1994-07-05 Matsushita Electric Industrial Co., Ltd. Pixel defect removing circuit for solid-state image pickup device
US5436659A (en) * 1991-05-10 1995-07-25 Eastman Kodak Company Method and apparatus for determining defective pixel location
US5231485A (en) * 1991-11-19 1993-07-27 Scientific-Atlanta, Inc. Method and apparatus for transforming between fixed-rate vector quantized data and variable rate vector quantized data
US5475441A (en) * 1992-12-10 1995-12-12 Eastman Kodak Company Electronic camera with memory card interface to a computer
US5808672A (en) * 1993-10-21 1998-09-15 Hitachi, Ltd. Apparatus for removably mounting an electronic camera to a computer system
US5712680A (en) * 1993-12-24 1998-01-27 Canon Kabushiki Kaisha Image pickup device for obtaining both moving and still images
US5477264A (en) * 1994-03-29 1995-12-19 Eastman Kodak Company Electronic imaging system using a removable software-enhanced storage device
US5907359A (en) * 1995-08-30 1999-05-25 Sanyo Electric Co., Ltd. Highly-integrated image sensing apparatus producing digitized output
US5867214A (en) * 1996-04-11 1999-02-02 Apple Computer, Inc. Apparatus and method for increasing a digital camera image capture rate by delaying image processing
US6005613A (en) * 1996-09-12 1999-12-21 Eastman Kodak Company Multi-mode digital camera with computer interface using data packets combining image and mode data

Cited By (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7426316B2 (en) 2001-06-28 2008-09-16 Nokia Corporation Method and apparatus for image improvement
US8103123B2 (en) 2001-06-28 2012-01-24 Nokia Corporation Method and apparatus for image improvement
EP1274224A3 (fr) * 2001-06-28 2004-11-17 Nokia Corporation Procédé et dispositif d'amélioration d'image
US7724977B2 (en) 2001-07-12 2010-05-25 Do Labs Method and system for providing formatted data to image processing means in accordance with a standard format
KR100879832B1 (ko) * 2001-07-12 2009-01-22 도 랩스 영상 처리 수단에 포맷화된 데이터를 표준 포맷에 따라제공하는 방법 및 시스템
US7502057B2 (en) 2002-06-20 2009-03-10 Hewlett-Packard Development Company, L.P. Method and apparatus for color non-uniformity correction in a digital camera
US7233352B2 (en) 2002-06-20 2007-06-19 Hewlett-Packard Development Company, L.P. Method and apparatus for color non-uniformity correction in a digital camera
NL1023709C2 (nl) * 2002-06-20 2006-02-21 Hewlett Packard Co Werkwijze en inrichting voor correctie van niet-uniformiteit in de kleur van een digitale camera.
NL1023711C2 (nl) * 2002-06-20 2006-02-21 Hewlett Packard Co Werkwijze en inrichting voor correctie van niet-uniformiteit in de kleur van een digitale camera.
EP1447977A1 (fr) * 2003-02-12 2004-08-18 Dialog Semiconductor GmbH Compensation du vignettage
US7692700B2 (en) 2003-02-12 2010-04-06 Digital Imaging Systems Gmbh Vignetting compensation
WO2004093438A1 (fr) * 2003-04-17 2004-10-28 Nokia Corporation Telephone mobile a module photographique perfectionne
EP1530375A3 (fr) * 2003-11-06 2008-03-26 OmniVision Technologies, Inc. Correction d'objectif utilisant des données YUV traitées
EP1530375A2 (fr) * 2003-11-06 2005-05-11 OmniVision Technologies, Inc. Correction d'objectif utilisant des données YUV traitées
CN100367763C (zh) * 2005-07-19 2008-02-06 华晶科技股份有限公司 清除数字影像噪音的方法及装置
KR100848589B1 (ko) 2005-10-11 2008-07-28 노키아 코포레이션 휴대용 카메라폰, 이미지 레코딩 방법 및 카메라 모듈
US10050705B2 (en) 2007-05-24 2018-08-14 Federal Law Enforcement Development Services, Inc. LED light interior room and building communication system
US10820391B2 (en) 2007-05-24 2020-10-27 Federal Law Enforcement Development Services, Inc. LED light control assembly and system
US9660726B2 (en) 2007-05-24 2017-05-23 Federal Law Enforcement Development Services, Inc. LED light broad band over power line communication system
US9755743B2 (en) 2007-05-24 2017-09-05 Federal Law Enforcement Development Services, Inc. LED light global positioning and routing communication system
US9768868B2 (en) 2007-05-24 2017-09-19 Federal Law Enforcement Development Services, Inc. LED light dongle communication system
US9967030B2 (en) 2007-05-24 2018-05-08 Federal Law Enforcement Development Services, Inc. Building illumination apparatus with integrated communications, security and energy management
US11664897B2 (en) 2007-05-24 2023-05-30 Federal Law Enforcement Development Services, Inc. LED light fixture
US10051714B2 (en) 2007-05-24 2018-08-14 Federal Law Enforcement Development Services, Inc. LED light control assembly and system
US11664895B2 (en) 2007-05-24 2023-05-30 Federal Law Enforcement Development Services, Inc. LED light control assembly and system
US10250329B1 (en) 2007-05-24 2019-04-02 Federal Law Enforcement Development Services, Inc. LED light fixture
US10374706B2 (en) 2007-05-24 2019-08-06 Federal Law Enforcement Development Services, Inc. LED light broad band over power line communication system
US11265082B2 (en) 2007-05-24 2022-03-01 Federal Law Enforcement Development Services, Inc. LED light control assembly and system
US11201672B2 (en) 2007-05-24 2021-12-14 Federal Law Enforcement Development Services, Inc. LED light fixture
US10911144B2 (en) 2007-05-24 2021-02-02 Federal Law Enforcement Development Services, Inc. LED light broad band over power line communication system
US10812186B2 (en) 2007-05-24 2020-10-20 Federal Law Enforcement Development Services, Inc. LED light fixture
US11424781B2 (en) 2009-04-01 2022-08-23 Federal Law Enforcement Development Services, Inc. Visible light communication transceiver glasses
US10763909B2 (en) 2009-04-01 2020-09-01 Federal Law Enforcement Development Services, Inc. Visible light communication transceiver glasses
US10411746B2 (en) 2009-04-01 2019-09-10 Federal Law Enforcement Development Services, Inc. Visible light communication transceiver glasses
US9654163B2 (en) 2009-04-01 2017-05-16 Federal Law Enforcement Development Services, Inc. Visible light transceiver glasses
US9655189B2 (en) 2013-03-13 2017-05-16 Federal Law Enforcement Development Services, Inc. LED light control and management system
US10205530B2 (en) 2013-05-06 2019-02-12 Federal Law Enforcement Development Services, Inc. Network security and variable pulse wave form with continuous communication
US11824586B2 (en) 2013-05-06 2023-11-21 Federal Law Enforcement Development Services, Inc. Network security and variable pulse wave form with continuous communication
US11018774B2 (en) 2013-05-06 2021-05-25 Federal Law Enforcement Development Services, Inc. Network security and variable pulse wave form with continuous communication
US11552712B2 (en) 2013-05-06 2023-01-10 Federal Law Enforcement Development Services, Inc. Network security and variable pulse wave form with continuous communication
US11783345B2 (en) 2014-01-15 2023-10-10 Federal Law Enforcement Development Services, Inc. Cyber life electronic networking and commerce operating exchange
US11200794B2 (en) 2015-08-11 2021-12-14 Federal Law Enforcement Development Services, Inc. Function disabler device and system
US11651680B2 (en) 2015-08-11 2023-05-16 Federal Law Enforcement Development Services, Inc. Function disabler device and system
US10448472B2 (en) 2015-08-11 2019-10-15 Federal Law Enforcement Development Services, Inc. Function disabler device and system
US10932337B2 (en) 2015-08-11 2021-02-23 Federal Law Enforcement Development Services, Inc. Function disabler device and system
WO2022174881A1 (fr) * 2021-02-16 2022-08-25 Huawei Technologies Co., Ltd. Compensation de variation d'éclairement de capteur d'événement

Also Published As

Publication number Publication date
WO2001001675A3 (fr) 2001-07-05
CN1365569A (zh) 2002-08-21
CN1391725A (zh) 2003-01-15
CN2567940Y (zh) 2003-08-20
DE20080319U1 (de) 2002-05-16

Similar Documents

Publication Publication Date Title
US6995794B2 (en) Video camera with major functions implemented in host software
WO2001001675A2 (fr) Camera video a fonctions principales mises en oeuvre dans un logiciel hote
US6184936B1 (en) Multi-function USB capture chip using bufferless data compression
US5835145A (en) Conversion system using programmable tables for compressing transform coefficients
US8428120B2 (en) Method and apparatus of Bayer pattern direct video compression
US5818525A (en) RGB image correction using compressed flat illuminated files and a simple one or two point correction algorithm
JP4907764B2 (ja) ビデオおよびスティール動作用の二重モード・ディジタル・カメラ
US5412427A (en) Electronic camera utilizing image compression feedback for improved color processing
US6961085B2 (en) Digital image processing and data compression circuit
US20080247653A1 (en) Method and apparatus for parallelization of image compression encoders
KR100347508B1 (ko) 촬상 장치
US20060170786A1 (en) Digital camera and method
US20070110155A1 (en) Method and apparatus of high efficiency image and video compression and display
US20100254617A1 (en) Apparatus and method to rotate an image
US6256350B1 (en) Method and apparatus for low cost line-based video compression of digital video stream data
US7190486B2 (en) Image processing apparatus and image processing method
JP5793716B2 (ja) 撮像装置
JP3976353B2 (ja) ディジタル・カメラ
US6580828B1 (en) Fast decoding
US20060001921A1 (en) System and method for high-performance scanner calibration
US7194129B1 (en) Method and system for color space conversion of patterned color images
US20020063899A1 (en) Imaging device connected to processor-based system using high-bandwidth bus
US8462377B2 (en) Method, apparatus, and system for reduction of line processing memory size used in image processing
US20070222891A1 (en) Systems and methods for video data conversion
US20090021645A1 (en) Video signal processing device, video signal processing method and video signal processing program

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): CN DE KR

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
AK Designated states

Kind code of ref document: A3

Designated state(s): CN DE KR

WWE Wipo information: entry into national phase

Ref document number: 1020017016889

Country of ref document: KR

WWE Wipo information: entry into national phase

Ref document number: 009000046

Country of ref document: CN

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642