Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T9/00—Image coding
• • H—ELECTRICITY
  • H01—ELECTRIC ELEMENTS
  • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
  • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
  • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
  • H01L27/144—Devices controlled by radiation
  • H01L27/146—Imager structures
  • H01L27/14601—Structural or functional details thereof
  • H01L27/1462—Coatings
  • H01L27/14623—Optical shielding
• • H—ELECTRICITY
  • H01—ELECTRIC ELEMENTS
  • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
  • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
  • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
  • H01L27/144—Devices controlled by radiation
  • H01L27/146—Imager structures
  • H01L27/14601—Structural or functional details thereof
  • H01L27/14625—Optical elements or arrangements associated with the device
  • H01L27/14627—Microlenses
• • H—ELECTRICITY
  • H01—ELECTRIC ELEMENTS
  • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
  • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
  • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
  • H01L27/144—Devices controlled by radiation
  • H01L27/146—Imager structures
  • H01L27/14643—Photodiode arrays; MOS imagers
  • H01L27/14645—Colour imagers
• • H—ELECTRICITY
  • H01—ELECTRIC ELEMENTS
  • H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
  • H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
  • H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
  • H01L27/144—Devices controlled by radiation
  • H01L27/146—Imager structures
  • H01L27/14665—Imagers using a photoconductor layer
  • H01L27/14667—Colour imagers
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/30—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability
  • H04N19/37—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using hierarchical techniques, e.g. scalability with arrangements for assigning different transmission priorities to video input data or to video coded data
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
  • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation
  • H04N19/423—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation characterised by memory arrangements

Definitions

• the present invention is directed to firmware and processing techniques that enable in-stream, i.e., on-the-fly, compression of digital image sensor data for storage and/or processing of that image data.
• FIG. 1 illustrates a basic conventional block diagram of an digital image recording system, such as a digital camera, in which an image signal recording system 1 includes a lens 2, an image sensor 3, a camera signal processing circuit 4, an image signal compression circuit 5, a recording mode selector circuit 6, a card identifying circuit 7, a system controller 8, a monitor screen 9, a card socket 10, a PC card 11, and a reference information storage circuit 12.
• an image signal recording system 1 includes a lens 2, an image sensor 3, a camera signal processing circuit 4, an image signal compression circuit 5, a recording mode selector circuit 6, a card identifying circuit 7, a system controller 8, a monitor screen 9, a card socket 10, a PC card 11, and a reference information storage circuit 12.
• An image converged through the lens 2 is formed on the image sensor 3, which convert the image into an electrical image signal.
• the image signal is compressed as still picture data by the compression circuit 5.
• the compressed still picture data is stored onto the PC card 11.
• An image being recorded is displayed on the monitor screen 9 so that an operator of the image signal recording 1 can monitor the image.
• Figure 1 is a medium to high-end camera implementation where JPEG or other processes such as auto exposure and white balancing are done on the camera.
• Figure 1 does not reflect typical low-end cameras that do not perform JPEG or other processes such as auto exposure and white balancing on the camera.
• Typical or conventional low-end cameras simply capture the raw image in a
• a non-volatile memory such as a FLASH Memory for image data storage.
• a user acquires an image with an image sensor; e.g., by pressing the shutter button of the digital camera; the image frame acquired by the image sensor must be transferred from the image sensor to the image data storage memory within the time allotted for a single image frame, which is defined by the frame rate of the sensor. This is because in general, an image sensor cannot store acquired image data for longer than a single frame.
• the characteristics of a FLASH memory can be sub-optimal for initial storage of image sensor data. More specifically, for many digital image applications, it has been found that FLASH memory cannot accept data at a speed compatible with typical image sensor frame times. In other words, FLASH memory is often not fast enough to accept a complete image frame within a specified image frame period.
• image compression is desirable for increasing the number of images stored in non-volatile memory and for reducing the time required to download images from camera to host.
• a frame buffer is usually required in cameras because non-volatile memory-write speeds are lower than the desired data rate of image sensors.
• image compression is performed after the transfer from sensor to frame buffer since the buffer is readily accessible for performing complex image compression techniques.
• typical compression techniques such as JPEG, are not always appropriate.
• a first aspect of the present invention is a method for in-stream compression of bytes of digital image sensor data.
• the method captures a scene and converts the captured scene into bytes of digital image sensor data; compresses the bytes of digital image sensor data; stores the compressed bytes of digital image sensor data in a temporary memory; and transfers the bytes of digital image sensor data from the temporary memory to a permanent memory.
• a second aspect of the present invention is a method of modeling, in stream, bytes of digital image sensor data for compression. The method masks a specified number of least significant bits of a byte of digital image sensor data and subtracts alternate bytes of digital image sensor data to produce an entropy-reduced data model.
• a third aspect of the present invention is a method of encoding, in stream, bytes of digital image sensor data for compression.
• the method splits a byte of digital image sensor data into a predetermined number of channels, each channel having a bit width such that the sum of the bit widths of each channel equals a bit width of the byte of digital image sensor data; operates upon each channel of digital image sensor data with a distinct cumulative distribution function; multiplexes the distributed digital image sensor data; and encodes the multiplexed digital image sensor data using arithmetic compression encoding.
• a fourth aspect of the present invention is a method of in-stream compression of bytes of digital image sensor data.
• the method masks a specified number of least significant bits of a byte of digital image sensor data; subtracts alternate bytes of digital image sensor data to produce an entropy-reduced data model; splits a difference byte of digital image sensor data into a predetermined number of channels, each channel having a bit width such that the sum of the bit widths of each channel equals a bit width of the byte of digital image sensor data; operates upon each channel of digital image sensor data with a distinct cumulative distribution function; multiplexes the distributed digital image sensor data; and encodes the multiplexed digital image sensor data using arithmetic compression encoding.
• a fifth aspect of the present invention is a method of division free arithmetic encoding.
• the method fixes a number of elements in a histogram to a number that is a power of 2; determines a number of elements in a bin of a histogram; and performs a bit shifting operation upon the determined number of elements in a bin of a histogram to find a probability of a symbol to be encoded.
• a sixth aspect of the present invention is a method for adaptively fixing a number of elements in a histogram.
• the method produces a new data element to be added to the histogram; adds the new data element to the histogram; tracks an order in which new data elements are added; and removes a data element from the histogram in accordance with the tracked order.
• Another aspect of the present invention is a method for adaptively fixing a number of elements in a histogram.
• the method produces a new data element to be added to the histogram; adds the new data element to the histogram; increments a bin value, the bin value being number of elements in a bin, when a new data element is added to the histogram; determines if elements are to be removed from a present bin in the histogram; decreases a value representing a number of elements to be removed from the present bin in the histogram when it is determined that elements are to be removed from the present bin in the histogram; and removes an element from the histogram when the value representing a number of elements to be removed is decreased.
• Figure 1 is a block diagram showing a conventional digital camera system
• Figure 2 is a block diagram showing a digital camera system according to the concepts of the present invention
• Figure 3 illustrates an example correspondence between a system clock and SRAM timing requirements for data sent to the SRAM on a data bus
• Figure 4 a flowchart showing an example control methodology that enables in- stream image data compression according to the concepts of the present invention
• FIG. 5 is a block diagram showing Bayer Differencing and bit dropping operations according to the concepts of the present invention.
• Figure 6 is a flowchart showing one perspective of a histogram storage and update technique according to the concepts of the present invention
• Figure 7 is a diagram of the decompression operation according to the concepts of the present invention
• Figure 8 illustrates a flowchart showing the weighted round-robin histogram update procedure for division-free arithmetic encoding according to the concepts of the present invention.
• Figure 9 illustrates a block diagram of an encoder according to the concepts of the present invention.
• the present invention positions the compression between the sensor and the frame buffer. This allows the reduction in the size and cost of the frame buffer.
• the present invention provides a hardware efficient compression engine that performs in-stream image compression.
• the data stream from the image sensor contains raw Bayer data where pixels in each row alternate between either red and green or green and blue. With this color scheme, a data set that skips every other pixel is likely to show a stronger correlation than just a simple sequence of adjacent pixels. Since row buffers are not available during an in-stream compression scheme, all modeling must be limited to one dimension. Thus, the present invention provides a first-order predictive model that subtracts every other pixel and encodes the result. To realize the above, the present invention provides a circuit architecture that includes a SRAM. An example of such architecture is illustrated in Figure 2.
• Figure 2 provides a block diagram of the components of an example imager system that enables capture and storage of a digital image, according to the concepts of the present invention.
• the imager system includes an imager 20 that captures image scenes and converters the image into electrical signals or image data.
• a controller 30 e.g., an ASIC or other suitable hardware and/or firmware controller implementation; is included to provide data transfer management between the imager 20 and a memory unit.
• the controller 30 directs the imager 20 to transfer a frame of image data to a compression engine 40.
• the compression engine 40 can be implemented in Nerilog, as a component of the controller 30.
• the controller 30 directs the compressed image data to a
• SRAM 48 for temporary storage. Once a frame of compressed image data is fully stored in SRAM 48, the image data in the SRAM 48 is then directed to FLASH memory 45.
• the in-stream image data compression by the compression engine 40 must operate at the clock rate of the image data transfer from the imager 20. More specifically, there is not available a higher speed system clock that could provide the compression engine 40 with more processing time than corresponds to the image data transfer rate.
• a particularly selected SRAM may not operate at the system clock rate, instead processing data at a rate that is, for example, three or more times slower than the system clock.
• the compressor 40 has available to it three clock cycles of processing time.
• the compression engine 40 is then in effect operating as if it were controlled by a clock that is three times faster than the system clock governing the image data stream transfer. As explained in more detail below, this condition can be exploited to enable highly efficient data transfer between the controller 30, the imager 20, and SRAM 48.
• the data busses are shown as separate busses between the imager 20 and controller 30, between the controller 30 and SRAM 48, and between the SRAM 48 and the FLASH memory 45. In accordance with the present invention, this is not required.
• a single data bus can be employed for transferring data between the various imager system components.
• the concepts of the present invention accommodate for configurations in which only one data bus is available and/or on which all data transfers between the various components must occur. It is further noted that although the present invention, as described in detail below, utilizes a single-bus system implementation, such a single-bus system implementation is not required by the concepts of the present invention.
• a temporary buffer 48 of Figure 2 e.g., an SRAM; that accepts and stores an acquired frame of image data within a specified image acquisition frame time, for later routing to and/or more permanent storage of that data in a FLASH memory.
• an SRAM can pose a data storage problem if the SRAM data storage capability is not sufficient for a selected image sensor. For example, given a digital CMOS image sensor including 1.3 million pixels, with each pixel producing 8 bits of image data, and an SRAM having a 1MB data storage capability, a full frame of image data cannot be held in the SRAM at any given time. Thus optimally, a temporary buffer is needed that is large enough to store the 1.3 million bytes of image data within a single image acquisition frame time, typically, e.g., -10ms.
• the present invention provides a number of embodiments that can be employed to accommodate this data storage requirement.
• both the FLASH 45 and the SRAM 48 can be employed to store the sensor image data during a single frame time.
• the data stream from the image sensor is split between the SRAM and the FLASH memory.
• the data rate of the stream to the SRAM and the FLASH memory sum to equal the data rate of the sensor.
• multiple SRAM buffers are used to accept the image sensor data.
• the use of a 1MB SRAM in conjunction with a 512KB SRAM is sufficient for temporary storage of a full 1.3 million bytes of image data within a single image acquisition frame period.
• the image data is compressed in-stream, i.e., on-the-fly, as the image data is acquired from an image sensor during a single image acquisition frame period, and before the data is written to an SRAM.
• a SRAM having a storage capability that is less than that required for a selected image sensor can accommodate a full frame of image data.
• image data produced by 1.3 million image sensor pixels can be accommodated by a 1MB SRAM.
• the compressed image data can then be sent from the SRAM to a FLASH memory at a subsequent time for more permanent storage.
• the data sent to the FLASH can be in compressed or decompressed form.
• FIG. 4 provides a flowchart of an example, in accordance with the concepts of the present invention, of a control methodology that enables the in-stream image data compression alluded to above.
• the controller first waits, at step S12, for the start of a new image frame.
• the controller resets, at step S16, the compression engine.
• the controller then waits, at step S18, for the start of a new row of the current frame.
• the controller gets, at step S22, an image data byte from the imager and sends, at step S24, the data byte to the compression engine.
• the controller After sending one data byte to the compression engine, the controller checks if the compression engine has completed compression of any data, and therefore, if any compressed data is available, at step S26, for transfer to the SRAM. If there is no compressed data available, the controller then checks if the current row of image data has been processed, at step S28. If the current row has not been completely processed, then the controller gets, at step S22, another data byte from the imager for sending to the compressor. If the current row has been completely processed, then the controller checks if the current frame has been processed, at step S30. If the current frame has not been completely processed, then the controller waits for the start of another row of data from the imager, for sending that image data row to the controller. If the current frame has been completely processed, then the controller awaits the start of a new frame.
• step S26 the controller begins sending, at step S32, that compressed data to the SRAM.
• step S34 the controller checks, at step S34, if the compressor is not full of data being compressed, and if the compressor has not completed processing the entire current image data row. If the compressor is full of data to be compressed, then the controller completes, at step S36, the data writing to the SRAM, and then checks, at step S28, if compression processing of the current row is complete.
• the controller gets, at step S38, a data byte from the imager and sends that data to the compression engine.
• the controller then completes, at step S36, the data byte writing to the SRAM in the current SRAM write cycle.
• This preferred embodiment of the present invention is utilized with an architecture or system in which a single data bus must be or is preferably employed for transfer between the controller and both the imager and the SRAM.
• Figure 3 illustrates an example correspondence between the system clock and SRAM timing requirements for data sent to the SRAM on a data bus.
• the system clock operates at a 20-nanosecond clock cycle.
• the SRAM in this example, requires an 80-nanosecond write cycle; i.e., a new data byte can be accepted by the SRAM every 80 nanoseconds.
• data to be sent to the SRAM must be valid on the data bus for only a part of the write cycle, for example, for 40 nanoseconds, during which a write enable (SRAM WEN) signal is set.
• SRAM WEN write enable
• any such available time during the SRAM write cycle is preferably employed to effectively multiplex imager data with compressed data on a single bus during the SRAM write cycle, thereby more effectively utilizing the bus and increasing the time available to the compression engine for processing image data to be sent to the SRAM.
• the compression engine is effectively operating at a clock rate that is faster than that of the SRAM.
• data compression in general requires two tasks.
• the first is a modeling task, in which the data is modeled to describe any redundancy in the data, thereby to reduce the entropy of the data set.
• the second task is a coding task, in which the data is encoded to produce a compressed version of the data.
• images typically contain a great deal of redundancy that can be exploited to decrease the entropy of the image data.
• a data set can be losslessly compressed to only the entropy of the data itself; i.e., a high entropy data set can be compressed to a lesser extent than a data set of relatively lower entropy.
• any suitable data model can be employed; e.g., a predictive model such as that employed in CALIC (Context Adaptive Lossless Image Compression), or other selected models that enable a decrease in data entropy.
• CALIC Context Adaptive Lossless Image Compression
• row buffers are often required for modeling techniques in which a neighborhood of pixel data values is examined to predict the value of a center-neighborhood pixel under consideration.
• this neighborhood modeling approach cannot be employed.
• hardware requirements be minimized in accordance with the imager system characteristics, and that a corresponding data model be employed; e.g., a first-order predictive modeling technique.
• image data from the image sensor is provided as rows of raw Bayer data; i.e., each pixel in each row alternates between either red and green or green and blue. As a result, data from every other pixel in a row is more correlated than data from adj acent pixels .
• Bayer Differencing can significantly decrease the entropy of an image data set. For example, when the original image data from two different images is entropy encoded, the image data can be compressed by 9% and by 28%, respectively. However, using a Bayer Differencing modeling approach, the same image data from the two different images can be compressed by 25% and 55%, respectively. Bayer Differencing can therefore be employed as a powerful technique for reducing entropy such that increased compression ratios are attainable.
• Bayer Differencing cannot guarantee a selected compression ratio. As explained above, this can be a concern for scenarios in which the storage capacity of an SRAM buffer memory is less than the capacity required to store an entire frame of image data.
• the present invention provides a technique, "bit dropping,” that enables the use of Bayer Differencing while imposing a desired compression ratio.
• bit dropping a technique, “bit dropping,” that enables the use of Bayer Differencing while imposing a desired compression ratio.
• the technique according to the concepts of the present invention, when a frame of image data is acquired by the image sensor, compressed, and then directed to the SRAM memory buffer, it is determined if the full frame of compressed image data can indeed be stored on the SRAM. If the full frame of compressed image data does not fit in the SRAM, then when a second frame of image data is acquired, the controller processes the data such that the least significant bit (LSB) of each pixel data value is dropped before performing Bayer Differencing and compression operations are performed on the data.
• LSB least significant bit
• each dropped bit of image data is found to result in an increase of the compression ratio by about 12.5%. If it is found that even with the LSB of image data dropped, a full frame of compressed image data cannot be accommodated by the SRAM, upon acquisition of a next subsequent image frame, the controller specifies that two LSBs be dropped from each byte in the image data stream for that frame. In accordance with the present invention, LSB dropping can be continually repeated until it is found that a sufficiently high compression ratio is achieved to enable a full frame of image data being stored by the SRAM buffer.
• FIG. 5 there is shown a block diagram of the Bayer Differencing and bit dropping operations described above.
• the image data 50 acquired by the image sensor 20 (not shown) is first directed to a FIFO 52 in which three consecutive pixel data value bytes are held.
• the number of LSBs to be dropped, if any, from the image data bytes are specified by programmable firmware or other suitable technique to produce a corresponding mask 54 of LSBs to be dropped.
• This mask 54 can be imposed on the pixel data values by; e.g., a barrel shifter, or other selected technique.
• the masked pixel data is then directed to; e.g., a two-stage or two-byte; pipeline 56 for carrying out the Bayer Differencing operation.
• three consecutive previous pixel values 58, 60, and 62 are saved such that alternate pixel bytes 58 and 62 can be subtracted by a subtracter 64 to produce an entropy-reduced data model to be encoded for data compression.
• the image data stream is encoded to produce a compressed stream of image data for storage at the SRAM buffer.
• There are numerous suitable methods for encoding the stream that can be employed in accordance with the present invention.
• a first example encoding technique is the LZW technique, which is a dictionary-based run-length encoding technique. This technique is quite hardware intensive and thus may not be suitable for all applications. In addition, the LZW technique requires large adaptive lookup tables, and generally its performance improves as the number of tables is increased. It also requires matching against these lookup tables, an operation that can be difficult to carry out in a single clock cycle.
• a second example encoding technique is Huffman encoding. Huffman also can be employed for many applications, but like the LZW technique is not very hardware efficient due to requirements for storing trees and for variable length encoded table look-up operations.
• Arithmetic encoding can be a preferable compression technique.
• a compression code can be produced in a single clock cycle as the result of a calculation, and the encoding process can reach the entropy of the data set.
• arithmetic encoding can optimally be implemented by storing data values in a cumulative histogram; i.e., a running summation of a histogram.
• Arithmetic encoding is employed by the present invention to encode the data stream because it does not require large hardware lookup tables or trees.
• the encoded symbol can be efficiently obtained in a single clock cycle as the result of a calculation.
• arithmetic encoding can reach the entropy of the data set regardless of the probability distribution. This enables the channel splitting technique described below.
• adaptive arithmetic encoding is essential for high compression performance; however, adaptive arithmetic encoding requires two hardware intensive operations. One is a division to calculate the probability from the adaptive histogram, and the other is a multiplication to rescale the state variables of the encoder. There are some techniques that facilitate multiplication-free arithmetic encoders in order to reduce the hardware requirements of that function. For further reduction in hardware, a division-free adaptive histogram technique is utilized by the present invention. To explain division-free adaptive histogram technique in more detail the following example will be used.
• the present invention uses an adaptive histogram technique that keeps N fixed at a power of two such that the division reduces to a simple bit shift.
• N the present invention removes elements from the histogram as new elements arrive in a weighted round-robin fashion as shown in the flowchart of Figure 8.
• the present invention waits for the next symbol .
• the process at step SI 04, adds symbol y to the histogram and increments m,, by one, wherein m y is the number of elements in bin y.
• step S114 the symbol is removed from the histogram and m x is decremented by one wherein m x is the number of elements in bin x and m x must always be greater than zero.
• a depth of 10 bits in each bin provides a good trade-off between letting the adaptive histogram go stale versus having enough data to yield good statistics.
• Such cannot be provided in an SRAM because all data bins need to be simultaneously accessible for updating, given that the compression operation is tied to the clock rate and thus no extra clocks cycles are available for each byte to be compressed.
• compression encoding of 8-bit data values by an adaptive encoding histogram technique can be challenging for many applications.
• the present invention addresses this challenge by providing a technique in which a stream of 8-bit image data bytes is split into channels, with each separate channel processed as a distinct data stream. As explained in more detail below, this channel splitting technique is found to yield compression ratios that are similar to that achieved when employing a non-split data stream.
• the 8-bit wide image data stream is split it into two 4-bit wide data streams.
• the channel splitting technique of the present invention can be further extended in accordance with the present invention.
• the 8-bit wide image data stream can be divided into three data streams; i.e., three distinct data channels.
• the split can take any suitable configuration; e.g., one 2-bit channel and two 3-bit channels as 3-2-3, or 2-3-3, moving from MSB to LSB.
• This three-way channel splitting enables the histogram bin precision to be dropped to 8 bits while yielding compression results that are similar to that achieved with a full 8-bit wide image data channel.
• This image data channel splitting can be further extended, if appropriate for a given application, and given that adequate clock cycles are available for the number of channels to be employed.
• channel splitter 65 here shown by way of example as 3 channels, with the channels divided as 3-3-2 bits, going from MSB to LSB.
• the channel splitter 65 may be realized by a simple register.
• the three channels are piped to separate, distinct, corresponding histograms; i.e., three distinct cumulative distribution functions 66, 68, and 70.
• Data from the three distribution function histograms 66, 68, and 70 are multiplexed by multiplexer 72 and then encoded by a suitable arithmetic compression encoding implementation circuit 74 as described below.
• the resulting compressed image data which can be of varying bit number from pixel data to pixel data, is then sent to a FIFO 76 and directed by the imager controller 30 to the SRAM 48 for temporary storage.
• histogram storage and update techniques provided by the present invention to be applied to each data channel's cumulative distribution function it has been shown that the application of arithmetic encoding to such histograms can be carried out while eliminating the need for multiplication operations.
• the present invention enables this elimination of multiplication operations, and goes further to simplify the division operations necessary for the adaptive encoding technique where the total number of elements, N, in the histogram may be changing as the encoding progresses.
• each division operation is simplified to a bit shift operation. This is achieved by imposing conditions in which the number of elements, N, in a given histogram is required to remain fixed and to be a power of 2. To enforce this last condition, for each data element added to a histogram, one must be taken away. This could be done using a FIFO approach where one tracks the order in which elements are added and they are removed in order. This, however, requires much more hardware than the technique developed for the present invention.
• the present invention employs two pointers, namely, an addition pointer and a subtraction pointer, which each pointer points to a particular bin in a histogram.
• Figure 9 provides another perspective on the channel splitting concept wherein a complete compression engine with the addition of a small output FIFO to absorb jitter produced by the variable length encoding is illustrated.
• a Bayer Differencing module or circuit 200 that produces an 8-bit dataword receives data.
• the 8-bit dataword is then split into three channels by splitter or mask 210.
• the dataword is split into two 3 -bit datawords and a 2-bit dataword.
• the datawords are fed into a round-robin, division-free, adaptive histogram 220, as described above.
• the round-robin, division-free, adaptive histogram 220 includes three histograms 221, 223, and 225. Data from the three histograms 221, 223, and 225 are multiplexed by multiplexer 230 and then encoded by a suitable arithmetic compression encoding implementation circuit 240 as described below.
• the resulting compressed image data which can be of varying bit number from pixel data to pixel data, is then sent to a FIFO 250.
• the compression engine of Figure 9 functions without the need for any peripheral devices, such as RAM or a processor, and the total number of registers in the complete design total is 269. Synthesized for a 0.35 ⁇ m process, the design takes 0.3 mm of area and approximately 7000 gates.
• Figure 6 provides another perspective on a histogram storage and update technique according to the concepts of the present invention. More specifically, Figure 6 provides a flowchart of another perspective on an implementation provided by the present invention for enabling lossless in-stream image compression.
• the system is initialized, at step S80, by setting the histogram; i.e., cumulative distribution function, to a known value, with an equal number of elements in each histogram bin.
• the subtraction pointer, sub_pointer is set to point to bin 0.
• the subtraction count, sub_count which represents how many data elements have been subtracted out of a given bin, is reset to 1, and the total number of data elements that will be subtracted out of the subtraction pointer bin is set to the value of the bin divided by a selected factor, here for example, 4.
• This division factor is selected based on a condition in which the proportional ratio of elements between bins of the histogram is to remain constant. To achieve this condition, the subtraction pointer is held in a given bin a number of encoding cycles that corresponds to the number of bin elements, divided by some proportional factor, e.g., 4.
• the system waits, at step S82, for the arrival from the channel splitting pipe of a new data element.
• a new data element arrives, at step S84, at a cumulative distribution function, the cumulative distribution function bin at which the sub_pointer is pointing is checked to determine if that bin contains at least one data value element.
• Each cumulative distribution function bin must have at least one element to enable arithmetic encoding in the conventional manner. If the cumulative distribution function does not have more than one element, then the addition pointer is updated, at step S88, to equal the new element value.
• the histogram is updated, at step S90, by adding one to all bins greater than or equal to the bin number at which the addition pointer is currently pointing, and subtracting one from all bins greater than or equal to the bin number at which the subtraction pointer is currently pointing.
• the subtraction count, sub_count equals the subtraction maximum, subjnax. If not, the subtraction count is incremented, at step S94, and the cumulative distribution function awaits, at step S82, the next data element. If the subtraction count does equal the subtraction maximum, then, at step S90, by adding one to all bins greater than or equal to the bin number at which the addition pointer is currently pointing, and subtracting one from all bins greater than or equal to the bin number at which the subtraction pointer is currently pointing.
• the subtraction count, sub_count equals the subtraction maximum, subjnax. If not, the subtraction count is incremented, at step S94, and the cumulative distribution function awaits, at step S82, the next data element
• the subtraction pointer is incremented, indicating that it will be pointing at the next bin; the subtraction maximum is reset as now being equal to the current cumulative distribution function of the current bin; and the subtraction count is to 1.
• the system then awaits, at step S82, the next data element. This completes one cycle of a technique where the number of cumulative distribution function data value items remains constant.
• the relatively small histograms that result from the image data channel splitting technique of the present invention synergistically work with this histogram update technique.
• the subtraction pointer can cycle through the entire histogram relatively quickly, whereby the cumulative distribution function data statistics are preserved from becoming stale, while at the same time providing data of a sufficient quantity to yield good results.
• the channel splitting technique is therefore found to actually enable better performance while at the same time reducing hardware requirements for the system. This is contrary to conventional wisdom, in which it is often suggested to increase channel size to, e.g., 16 or 24 bits, in an effort to improve performance.
• the exact opposite i.e., shrinking of data channel extent; is found to produce improved results.
• the arithmetic encoding technique for compressing the channels of image data can take any suitable implementation, including a multiplication-free technique.
• a conventional signal add-shift step which is really a 2-bit multiply operation
• a 3-bit multiply operation can be more preferably employed; it is found that performance going from 3 to 2 bits is noticeable as a few percentage points but going from 3 bits to full precision is not noticeable in performance.
• a guard register of eight bits can be used to reduce the carry-over effect, and bit stuffing can be employed when the carry-over register happens to fill with eight or more successive l's.
• the encoding operation can be divided into parallel operations to enable further enhanced channel splitting; here it must be recognized that there exists a tradeoff between the resulting enhanced efficiency and a requirement that the arithmetic encoder, containing a relatively complex signal path with a multiplication and complex decision tree, would need to be replicated.
• Figure 7 is a diagram of the decompression operation corresponding to the compression operation above. This decompression can be carried out at any suitable stage of the image data processing; e.g., upon download to a computer or network for further processing and/or storage of the image data.
• compressed data 100 is sent to a decoder 102 for decoding the compressed data.
• the decoder synchronizes its decompression with multiplexer 104, which multiplexes data from the three histograms, CDF1 66, CDF2 68, and CDF3 70, such that the decoded data correctly corresponds with the original uncompressed data.
• the resulting data is fed to de-multiplexer 106 into the corresponding number of split channels that were imposed prior to the image data compression, and then a full 8-bit wide data word is reconstructed by register 108 with the split channel data.
• Such reconstructed image data words are then directed to a two-stage pipeline that can accommodate three data words 112, 114, and 116. Alternating data words are then added together by an adder 118 to compensate for the prior Bayer Differencing operation.
• the decompressed data 120 is fully reconstructed, and can be directed as desired to f rther processing and/or storage operations.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Power Engineering (AREA)
• General Physics & Mathematics (AREA)
• Electromagnetism (AREA)
• Condensed Matter Physics & Semiconductors (AREA)
• Computer Hardware Design (AREA)
• Microelectronics & Electronic Packaging (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Theoretical Computer Science (AREA)
• Compression Or Coding Systems Of Tv Signals (AREA)
• Compression Of Band Width Or Redundancy In Fax (AREA)
• Storing Facsimile Image Data (AREA)

Applications Claiming Priority (3)

Publications (1)

Family

ID=38476987

Family Applications (1)

Country Status (4)

Families Citing this family (4)

Citations (2)

Family Cites Families (7)

• 2003
  • 2003-10-14 EP EP03776298A patent/EP1559072A2/en not_active Withdrawn
  • 2003-10-14 WO PCT/US2003/032220 patent/WO2004034331A2/en active Application Filing
  • 2003-10-14 AU AU2003284067A patent/AU2003284067A1/en not_active Abandoned
  • 2003-10-14 JP JP2004543695A patent/JP2006502673A/ja active Pending

Patent Citations (2)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events