KR100937378B1 - Method, computer program product and device for processing of still images in the compressed domain - Google Patents

Abstract

The present invention relates to the manipulation and especially filtering of image data in the compressed domain. The present invention provides a method and apparatus for manipulating image data in a compressed domain transformed from image data in a spatial domain with a wavelet based transformation algorithm. An advantage of the inventive concept is to speed up the filtering time so that the underlying constraints of the computational power, storage, memory and power consumption in the device may be appropriate for image manipulation / filtering.

Description

Method, computer program product and device for processing of still images in the compressed domain}

The present invention relates to the field of image processing. In particular the invention relates to still image processing in the compressed domain.

An increasing number of portable devices feature multimedia applications that operate with audio and video. These multimedia applications are increasingly attracting customers and they enthusiastically enjoy the use of new multimedia features. In particular, the market for portable telephones and cellular telephones illustrates the rapid acceptance of multimedia feature devices. That is, imaging phones, which are portable phones with image capture capability, represent a class of portable phones, which is the best way to represent a standard in a remarkable short time after the introduction of this product class. Successful development of imaging phones involves the success of comparable digital cameras.

To date, especially of multimedia applications, which are image capture applications, address recording / capture quality, memory requirements, streaming capability (bit rate requirements), and playback / display performance. Developed in the field. Correspondingly, most research on multimedia applications focuses on compression standards, compositing issues, storage representations, and sensor designs. For example, the field of techniques for processing digital images, such as image manipulation including editing, filtering, composing, and special effect implementation (such as brightness adjustment, contrast enhancement, and sharpening) is neglected. It is becoming.

The disregard for the field of image processing is based on complexity and data volume, which is required for image processing, i.e. computational complexity. Typically, portable devices have compression that meets the limitations of such portable devices, in particular computational power and storage capacity, as well as the demands placed on image handling, including wireless reception / transmission, streaming performance, etc. of images. Handle digital images in mode. Because of the complexity, image processing is a typical task currently implemented on workstations and personal computers. These powerful processing devices allow the operation of brute-force algorithms, including the reconstruction of digital images in compression mode, the processing of images in the uncompressed domain, ie manipulation, and finally Allows to compress the manipulated image to regain the manipulated digital image in compression mode. As mentioned earlier, modern compression techniques used for image compression are mainly in compression mode, which accepts a high degree of computational complexity to obtain a deteriorated compression without degrading the digital image or within acceptable limits. It is optimized to obtain the final result size of the digital image.

Nevertheless, the gradual acceptance of multimedia applications can drive the needs of users / customers for image manipulation available on today's powerful home computer equipment.

Accordingly, there is a need to provide efficient techniques and methodologies for efficient manipulation, editing and processing of digital images to processing devices having limited processing power, including limited computing power and limited storage capacity. In particular, the presented techniques and methods incorporate user-desired effects, such as brightness adjustment, contrast enhnacement, filtering and sharpening, into the images when manipulating, editing, and processing the images. Apply.

It is an object of the present invention to overcome the conventional needs of reconstruction before digital image processing operations and recompression after digital image processing operations. It is solved by providing techniques and methods for image processing. The main advantage of the present invention is the significantly reduced computational complexity.

를 제공한다. 리니어 함수

는 피가수(summand)들 및 인자들의 어떤 일차 결합(linear combination )으로 구성된다. 이것은 리니어 함수가

는 0, 하나 또는 그 이상의 리니어 오프셋들

를 포함하고, 0, 하나, 또는 그 이상의 리니어 스케일링 인자들

을 포함함을 의미한다. 예를 들어, 이미지 데이터의 이미지 표현(representation)에서의 함수 특정 필터링 효과(function specific filtering effect)를 얻기 위해, 상기 이미지 데이터에 적용할 수 있는 리니어 함수

가 공간 도메인에 제공된다. 리니어 함수

는 압축 도메인 내부로 변환되고, 그 변환된 후에 변환된 리니어 함수

는 변환된 이미지 데이터의 특정 서브밴드들, 예를 들어 상기 압축된 이미지 데이터의 베이스 밴드(base-band, LL) 또는 압축된 이미지 데이터의 정의된 수의 서브밴드들 상에 적용된다. According to a first aspect of the invention, a method for modifying image data in a compressed domain is provided. Compressed image data, i.e. image data in the compression domain, is obtained from image data in the spatial domain (eg provided by an image capturing sensor), which is a wavelet transform and in particular discrete By an encoder using a subband based transform such as a discrete wavelet transform (DWT). A typical encoder anticipated as a future standard is the JPEG 2000 encoder. The invention is a (original) linear function, which can be applied as a linear filter function.

To provide. Linear function

Is composed of some linear combination of sumands and arguments. This is a linear function

Is zero, one or more linear offsets

Zero, one, or more linear scaling factors

It means to include. For example, a linear function that can be applied to the image data to obtain a function specific filtering effect in the image representation of the image data.

Is provided in the spatial domain. Linear function

Is transformed into the compression domain, and then converted linear function

Is applied on specific subbands of transformed image data, for example a base-band (LL) of the compressed image data or a defined number of subbands of compressed image data.

의 애플리케이션의 결과로 근사치(approximation)가 얻어질 수 있다. 애플리케이션을 위해 선택된 변환된 이미지의 서브밴드들의 수는 결과인 근사치의 퀄러티(quality)를 결정함을 주의해야된다. 더 많은 서브밴드들이 선택될수록, 변환된 리니어 함수

애플리케이션의 결과 생기는 최종 이미지 표현에 더 가깝게 되고, 상기 최종 이미지 표현은 공간 도메인에서 이미지 데이터에 리니어 함수

를 적용함으로써 얻어지는 이미지 표현에 대응한다. 양쪽의 결과 이미지 표현들(즉, 변환된 리니어 함수를 압축 도메인에서 그리고 공간 도메인에서 원본 함수를 각각 적용함으로써 얻어지는) 사이의 차이점들은 보통은 중요하지 않다. 그러므로 상기 압축된 이미지 데이터의 특정 서브밴드들에서만 그리고 특히 베이스 밴드(LL)에 변환된 리니어 함수

를 적용하는 것은 빠른 계산 속도의 이점이 있는 좋은 근사치를 나타낸다. 당업자는 베이스밴드(LL 서브밴드)가 결과인 짧은 계산 처리 시간을 지원하는, 해상도가 감소된 압축해제된 이미지 데이터(공간 영역에서의 이미지 데이터에 대응함)를 표현함을 예상할 수 있다. Transformed linear function

Approximation can be obtained as a result of the application of. Note that the number of subbands in the transformed image selected for the application determines the quality of the resulting approximation. The more subbands are selected, the transformed linear function

Closer to the final image representation resulting from the application, the final image representation being a linear function on the image data in the spatial domain

Corresponds to the image representation obtained by applying. The differences between both resulting image representations (ie, by applying the transformed linear function in the compression domain and in the spatial domain respectively) are usually insignificant. Therefore, a linear function transformed only in certain subbands of the compressed image data and in particular in the baseband LL.

Applying a good approximation has the advantage of fast computational speed. One skilled in the art can expect that the baseband (LL subband) represents a reduced resolution decompressed image data (corresponding to image data in the spatial domain) that supports the resulting short computational processing time.

를 제공한다. 리니어 함수

는 피가수들 및 인자들의 어느 일차 결합으로 구성된다. 이것은, 리니어 함수

가 0, 하나 또는 그 이상의 리니어 오프셋들

및, 0, 하나, 또는 그 이상의 리니어 스케일링 인자들

로 구성됨을 의미한다. 본 발명은 이미지 데이터의 이미지 표현에서의 함수 특정 필터링 효과를 얻기 위해 공간 도메인에서 상기 이미지 데이터에 적용할 수 있는 리니어 함수

를 제공한다. 한 코드 섹션은 변환된 리니어 함수

가 결과로서 생기도록, 리니어 함수

를 압축 도메인 내부로 변환할 수 있게 한다. 추가의 코드 섹션은 상기 변환된 리니어 함수

를 특정한 미리 정해진 수의 압축된 이미지 데이터의 서브밴드들, 특히, 압축된 이미지 데이터의 베이스 밴드(LL) 상에 적용하도록 구성된다. According to a second aspect of the invention, a computer program product for modifying image data in a compressed domain is provided. The computer program product includes a computer readable medium having computer logic recorded on the computer readable medium, which includes code sections composed of instructions executed by a processing device. The present invention (original) linear function

To provide. Linear function

Consists of any first combination of singers and factors. This is a linear function

Is zero, one or more linear offsets

And zero, one, or more linear scaling factors

It means that it consists of. The present invention is a linear function applicable to the image data in the spatial domain to obtain a function specific filtering effect in the image representation of the image data

To provide. One section of code is a converted linear function

Is a linear function

To convert into a compressed domain. An additional section of code is used to convert the linear function

Is applied to a particular predetermined number of subbands of compressed image data, in particular baseband LL of compressed image data.

를 포함한다. 리니어 함수

는 어떤 피가수(summand)들 및 인자들의 일차 결합(linear combination)으로도 구성된다. 이것은 리니어 함수가

는 0, 하나 또는 그 이상의 리니어 오프셋들

를 포함하고, 0, 하나, 또는 그 이상의 리니어 스케일링 인자들

을 포함함을 의미한다. 본 발명은 예를 들어, 이미지 데이터의 이미지 표현(representation)에 있어서 함수 특정 필터링 효과(function specific filtering effect)를 얻기 위해, 공간 도메인에서 상기 이미지 데이터에 적용할 수 있는 리니어 함수

를 제공한다. 이미지 조작 모듈은 상기 변환된 리니어 함수

가 결과로서 생기도록, 리니어 함수

를 압축 도메인 내부로 변환하도록 구성된다. 또한 이미지 조작 모듈은 변환된 리니어 함수

를 압축된 이미지 데이터의 특정 서브밴드들, 예를 들어 압축된 이미지 데이터의 베이스 밴드 상에 적용할 수 있게 한다. According to a third aspect of the invention, consumer electronics (CE) with image capture capability is provided. The apparatus comprises an encoder for image data handling, which uses for example wavelet transforms, in particular discrete wavelet transforms (DWTs), and more particularly subband based transforms such as the JPEG 2000 codec. The encoder allows the image data in the spatial domain to be converted to the image data in the compressed domain, which can save storage volume unlike raw image data storage. The image device is a linear function

It includes. Linear function

Consists of a linear combination of any summands and arguments. This is a linear function

Is zero, one or more linear offsets

Zero, one, or more linear scaling factors

It means to include. The present invention is a linear function applicable to the image data in the spatial domain, for example, to obtain a function specific filtering effect in the image representation of the image data.

To provide. The image manipulation module converts the linear function

Is a linear function

Is converted into the compression domain. The image manipulation module also converts linear functions

Can be applied on specific subbands of compressed image data, for example on a base band of compressed image data.

The remainder and further aspects of the present invention will become apparent from the following description and the accompanying drawings.

1A schematically illustrates a portable CE (Consumer Electronics) device as a first embodiment with image capture capability in accordance with an embodiment of the present invention.

1B schematically illustrates a portable CE device as a second embodiment having image capture capability in accordance with an embodiment of the present invention.

2 illustrates a typical simple state of art image modulation operation.

3 is a diagram schematically illustrating operation modules of a wavelet transform encoder and a wavelet transform decoder for still image compression / reconstruction according to an embodiment of the present invention;

4 schematically illustrates a wavelet transform involving sub-band segmentation and frequency filtering resulting from a three level wavelet transform of a digital image;

5 schematically illustrates an example frequency decomposition for the wavelet transform encoder of FIG. 3.

FIG. 5B schematically illustrates an exemplary frequency synthesis for the wavelet transform encoder of FIG. 3. FIG.

6 schematically illustrates shifts in histogram diagrams of an image affected by brightness adjustments in accordance with an embodiment of the invention.

7 schematically illustrates a three level wavelet transform of a digital image divided into zones and code blocks.

FIG. 8 schematically illustrates an operation sequence of image adjustment by scalar summation / subtraction according to an embodiment of the present invention. FIG.

9 is a schematic illustration of bit plane codings of an LL subband according to an embodiment of the invention.

FIG. 10 schematically illustrates shifts in histogram diagrams of an image affected by contrast enhancements in accordance with an embodiment of the present invention. FIG.

FIG. 11 is a schematic illustration of an exemplary contrast stretching transformation in accordance with an embodiment of the invention. FIG.

 FIG. 12 schematically illustrates an operation sequence of image modulation by scalar multiplication / division according to an embodiment of the present invention. FIG.

13 illustrates an example implementation of an image modulation module according to an embodiment of the invention.

In the following description of the various embodiments, reference is made to the accompanying drawings that form a part hereof, which is shown by way of illustration of various embodiments in which the invention is performed. It is to be understood that other embodiments may be utilized and structural and functional changes may be made without departing from the scope of the present invention. Like reference numerals are used in the drawings and the descriptions that refer to similar or identical parts.

The block diagram of FIG. 1A shows the main structural components of a portable consumer electronic (CE) device 100, which illustratively applies to any type of portable consumer electronic (CE) device used in the present invention. Can be represented. It is to be understood that the invention is not limited to the CE device 100 shown or any other particular type of portable CE device. The illustrated portable CE device 100 is illustratively performed with a camera phone, which typically represents a cellular phone with image capture capability by an image capturing sensor.

In particular, the device 100 includes a cellular radio frequency interface with a central processing unit (CPU) 110, data storage 120, application storage 130, and a radio frequency antenna (not shown). cellular communication means including a frequency interface (I / F) 180 and a subscriber identification module (SIM) 170, audio input / output (I / O) means 140 (typically a microphone and a loudspeaker) ), User interface input / output means, image projection, including keys with a key input controller (Ctrl), a keypad and / or keyboard (150), and a display (160) with a display controller (Ctrl). Image capture sensor 10 including a charge-coupled device (CCD) sensor (not shown) with typical (not shown) optics, and for image handling Being It is embodied as a processor-based or microcontroller-based device that includes an image processing module 20 representing an exemplary implementation of several independent and dependent modules and components.

The operation of the CE device 100 typically uses a central processing unit (CPU) based on an operation system or basic control application that controls the features and functions of the CE device by making the user use the features and functions of the CE device. Is controlled by 100). Display and display controller (Ctrl) 160 are controlled by central processing unit (CPU) 110 and provide information to the user. A keypad and keypad controller (Ctrl) 150 is provided to allow a user to enter information. Information entered via the keypad is provided to the central processing unit (CPU) 110 by the keypad controller Ctrl, which can be commanded and / or controlled according to the input information. The audio input / output (I / O) means 140 comprises at least a speaker for reproducing the audio signal and a microphone for recording the audio signal. The central processing unit (CPU) 110 converts the audio data into audio output signals and the audio input signal into the audio data, for example when the audio data has a proper format for transmission and storage. Can control their conversion. The conversion of digital audio into audio signals and vice versa is traditionally supported by digital / analog and analog / digital circuits.

Additionally, portable CE device 100 according to certain embodiments of the invention shown in FIG. 1A optionally includes a cellular interface (I / F) 180 coupled to a radio frequency antenna, and includes a subscriber identity module (SIM) ( 170) can be operated. The cellular interface (I / F) 180 is installed as a cellular transceiver that receives signals from a cellular antenna, decodes the signals, demodulates the signals, and converts the signals to baseband frequencies. The cellular interface 180 provides an over-the-air interface, which is connected to a subscriber identification module (SIM) to provide a wireless access network (PLMN) of public land mobile networks (PLMNs). Serve cellular communication with a corresponding base station (BS) in a radio access network (RAN). The output of the cellular interface (I / F) 180 thus consists of a data stream that may require further processing by the central processing unit (CPU) 110. The cellular interface (I / F) 180, installed as a cellular transceiver, is also configured to receive data from a central processing unit (CPU), which is a base station of a radio access network (RAN) via an over-the-air interface. BS). Accordingly, the cellular interface (I / F) 180 encodes, modulates, and upconverts the data to embody a radio frequency signal. The cellular antenna then transmits the resulting radio frequency to a base station (BS) of a random access network (RAN) of a public land mobile network (PLMN).

Image capture sensor 10 is typically implemented with charge-coupled devices (CCDs) and optics. Charge coupled devices (CCDs) comprising grids of pixels are used for digital image capture as light-sensing devices in digital cameras, digital optical scanners and digital video cameras. The image is projected by optics (a single lens or an arrangement of one or more lenses) on a capacitor array (CCD), allowing each capacitor to accumulate charge in proportion to the light intensity at that location. Two-dimensional arrays used in digital video and digital still cameras capture the entire image or rectangular portions of the entire image. After the array appears in the image, the control circuitry causes each capacitor to transfer its content to a neighboring capacitor. The last capacitor in the array dumps its charge to an amplifier that converts it to voltage. By repeating this process, the control circuit converts the entire contents of the array into various voltages, samples the raw image data for further handling by the image processing module 20, digitizes it, to provide. The image processing module 20 allows the user of the CE device 100 to take still digital images and consequently to take video sequences as well. Typically, original image data is compressed by the image processing module 20 and stored in data storage. The image processing module 20 implements the coding and encoding modules required for codecs, i.e. still digital image processing (and video processing), among others, and the implemented components of the image processing module 20 are preferably software Application components, the operation of which is supported by a specific hardware implementation and is advantageous for improving the processing performance and functionality of the image processing module 20.

Recently CMOS image sensors have become popular and generally replace CCDs in the case of image capture. CMOS image sensors can be produced using a CMOS fabrication process, which is the dominant technology in all chip manufacturing, and CMOS image sensors can be inexpensive to manufacture and can be included in devices with the same signal conditioning circuitry. It should be understood that the present invention is not limited to any particular image capture sensor type. Typical alternative portable CE devices may include personal digital assistants, handheld computers, and the like, which may have image capture sensors and have image processing capability. Since the implementation of such a typical microcontroller in accordance with CE devices is well known in the art, the detailed implementations are outside the implementation scope of the present invention.

The block diagram of FIG. 1B shows the main structural components of the portable consumer electronics (CE) device 101, and may illustratively represent any kind of portable consumer electronics (CE) device available in the present invention.

The illustrated portable CE device 101 is illustratively performed as a digital camera that allows a user to capture digital images. In particular, the device 101 is connected to the microcontroller (μC) 111 via the microcontroller (μC), the image data storage 121, the key input controller (Ctrl) 151 and the display controller (Ctrl). User interface input / output means including a display with 161, an image comprising a typical charge-coupled device (CCD) or CMOS image sensor (not shown) with optics (not shown) for image projection Micro-controller comprising a capture sensor 10 and an image processing module 20 representing the implementation of some exemplary dependent and independent modules and components required for image handling ( is embodied in μC) based devices. The operation of the CE device 101 is controlled by the microcontroller (μC) 111, which controls the operation of the implemented components. The user can interact with the CE device 101 by key input, which affects the functions of the CE device 101. Traditionally, these CE devices 101 have their operations fixed, unlike processing-based CE devices. The implementation of digital cameras is well known in the art, and the details of the implementation are outside the scope of the implementation of the present invention.

Nevertheless, the implementation of the image capture sensor 10 based on hard-wired or software applications may provide the user with the same functionality. Further description will focus on the concepts of the present invention, which is generally directed to image processing implemented within CE devices 100, 101 (see FIGS. 1A, 1B implemented within image processing module 20). Related. In particular, the inventive concept relates to image manipulation, editing, filtering, etc., which may operate in accordance with user instructions, or may also operate automatically and / or independently. . Referring to Fig. 2, a typical process of image manipulation, editing, filtering and the like will be shown. Given an image, assume that the user wants to adjust the image for contrast enhancement. Typical simple image enhancement and / or processing operations, such as brightness adjustment, contrast enhancement or color filtering, may be used for full decoding of the compressed image representation within the CE devices 100, the spatial domain ( For the application of these operations in the spatial domain, and for the compressed image representation, full recompression of the edited / modified image is required. In the case of portable CE devices, they are limited to computational power and memory, and such image processing operations can be costly in computation, especially if the image has a high resolution. The area of computational power and memory is dependent on the weight of portable CE devices, as well as the electrical power consumption due to computation time, compute load, module empowerment (in the case of selective activation / deactivation of the module). In terms of size, it determines the overall design of such portable CE devices. As such, it should be noted that digital camera manufacturers and camera mobile phone manufacturers, which produce the typical type of portable CD devices to which the present invention relates, compete to increase the resolution of the images they have.

Currently, there are up to 2 megapixel sensors for cell phones and up to 5 megapixel sensors for typical digital cameras. These figures are expected to increase further. In edited / processed images, users generally apply other filtering operations to adjust contrast, improve contrast, and improve the visual quality of the images. As mentioned above, these operations are performed by reconstructing and recompressing the adjusted digital images, which can be of significant time and power. It is assumed that compressing an extremely small sized color digital image can take about 1 unit of time (depending on the computational complexity provided by the portable CD device and the complexity of the computation of the compression algorithm used). Increasing the color image size, for example, to VGA (640x480 pixels), results in about four times the corresponding compression time. These observations, which are just linear extensions, indicate that the encoding time per color digital image of one mega pixel can be increased by more than 13 times (compared to very small color digital images). In fact, increasing by 1 megapixel in image resolution further increases encoding time, depending on input and output resolution factors.

In the following, the JPEG 2000 compression algorithm will become the standard compression algorithm in the future due to better image compression than JPEG and new important features for image applications such as scaling and scaling in resolution and quality. 3 schematically illustrates JPEG 2000 compression (which may also be referred to as encoding) as well as JPEG 2000 reconstruction (which may also be referred to as decoding).

An operation flow diagram of a typical JPEG 2000 encoder 200 can be obtained from FIG. 3. The first module (not shown) of the encoder 200 before the compression sequence is a separate module, the function of which is to crop the image to the manageable size customarily required for significant size digital images. The digital images are separated into spatially nonoverlapping tiles of the same size. Each component is then processed separately. Below, a decorrelation step module 210 is given, whose function is to de-correlate color components of the digital color image. Within certain color images, in the case of multicomponent images, component transform is performed to uncorrelate the components. For example, a color image with RGB (red, green and blue) components can be transformed into Luminance, Chrominance red and Chrominance blue (YCrCb) or reversible component transform (RCT) component space. Image data (each of the digital image and components of the digital image) is provided to a conversion module 220, which is configured to convert the image data into a wavelet domain. After the transformation, image data in the wavelet domain is quantized by quantization module 330. The quantized coefficients that result from the quantization operation are then regrouped by entropy encoding module 240 to facilitate localized spatial and resolution access.

Each subband of the quantized coefficients is divided into rectangular blocks that are not overlapped. Three spatially located rectangles (one of each subband at a given resolution level) form a packet partition. Each packet partition is further divided into code blocks, each code block being compressed into an embedded bit stream with a rate-distortion curve. The embedded bit stream of the code blocks is assembled into packets by the bit stream assembler module 250, where each packet represents a quality increment of one resolution level at one spatial location. . A layer is formed by collecting packets from full packet partitions of full tiles and full resolution levels of full components so that a layer of one quality increase in the full image is at maximum resolution. Note that the final JPEG 2000 bit stream may consist of multiple layers.

Similarly, an operational flow diagram of a typical JPEG 2000 decoder 300 can also be obtained from FIG. 3. Decoding and reconstruction of the JPEG 2000 compressed image representation provides modules that perform inverse functions with respect to each of the module functions described above with reference to the JPEG 2000 encoder 200. A compressed image representation is provided to the bit stream parsing module 310, which is included in the required components and tiles and finally packets assembled therein. Separate and / or extract code blocks. Entropy decoding module 320 is configured to reverse the regrouping obtained by entropy encoding module 310.

Inverse quantization module 330 reconstructs image data in the wavelet domain from the quantized coefficients and the image data in the wavelet domain is provided to inverse transform module 340, and inverse transform module 340 is inverse wavelet transform Is performed to generate image data in the spatial domain. The inverse decorrelation step module 350 then reconverts the uncorrelated components. This means, for example, that a color image in Luminance, Chrominance red and Chrominance blue (YCrCb) or reversible component transform (RCT) component space is converted to a color image with RGB (red, green and blue) components. In the case of separating into non-overlapping tiles of spatially the same size because of the image size, the final module (not shown) of the decoder 300 following the decorrelation step module 350 is responsible for the components that are processed separately during decoding. Merge.

Note that JPEG 2000 is a standardized image compression algorithm, developed by the Joint Photographic Experts Group (JPEG) and published by the International Standards Organization (ISO) as ISO / IEC 15444-1: 2000, which is incorporated by reference as a background art. do. Encoding / compression according to the JPEG 2000 standard takes about 1.2 times compared to decoding / decompression according to the JPEG 2000 standard. For encoding, the numbers below are approximate percentages of how long each operation takes (the list is not exhaustive, i.e. it only mentions the main contributions).

Image data reading: ~ 5%

-Unnecessary color (if necessary): ~ 5%

Wavelet Transformation: ~ 10%

   Bits can change depending on the number of decompositions to be performed

Entropy Coding: 70%

   Can vary depending on the bits per pixel being targeted.

   The JPEG 2000 standard is a bit-plane based encoder.

  Image quality and computational complexity increase with increasing number of bit planes to encode

Allocation rate and bit stream assembly: ~ 10%

In the case of reconstruction, operations require symmetric computational power in nature (the list is not exhaustive, ie only mentions the main contributions).

Bitstream parsing: ~ 5%

Reverse color disassociation (if needed): ~ 5%

Inverse Wavelet Conversion: ~ 10%

   Bits can change depending on the number of decompositions to be performed

Reverse entropy decoding: ~ 70%

   Can change depending on the bits per pixel being targeted.

   The JPEG 2000 standard is a bit-plane based coder.

  Image quality and computational complexity increase with increasing number of bit planes to encode and decode, respectively

-Write image data: ~ 10%.

In order to increase the performance of processing operations, it is necessary to avoid re-encoding the image after changing the image content. This basically means avoiding the main factors in the encoding process, ie the entropy coding and decoding processes. As can be seen from the enumeration described above, as well as in FIG. 2, performing operations to change the original image to produce a modified image, while gaining significant gains in computation time, requires avoiding the entropy coding / decoding portion as much as possible. . In terms of computational complexity, entropy encoding / decoding is an obstacle. Significant gains in computational cost can be obtained by avoiding the same.

The most common high performance image encoders in an application are transform-based coders. The transform de-correlates the pixels and compresses the energy of the pixels into a small number of coefficients for efficient coding of the transform coefficients. Since most of the energy is compressed into slightly larger transform coefficients, an entropy coding scheme can be used to encode them efficiently.

The wavelet transform acts on the entire image (or part of the image, one tile of a component in the case of a large color image), providing better energy compression than in the case of discrete cosine transforms, There are no blocking artifacts. In addition, the wavelet transform decomposes the image into L level dyadic wavelet pyramids, as shown in FIG. 4. The resulting wavelet coefficients can be easily scaled in terms of resolution: the image can be reconstructed 2 ^M times smaller than the original image, without using wavelet coefficients at the finest M levels. The multiresolution capability of the wavelet transform makes it ideally suited for scalable image coding.

A wavelet is a signal representation in which low pass coefficients represent a slow change in a long data segment and high pass coefficients represent a localized change in a short data segment. The framework of the one-dimensional wavelet transform is shown in Figures 5a and 5b. The input row can be resolved through low and high pass analysis finite impulse response (FIR) filters, and subsamples the filtered result by factor 2. ). To reconstruct the original input stream, the low pass and high pass coefficients are upsampled to factor 2 and passed through another low and high pass synthesized FIR filter pair.

로 명시되고, 결과인 조작된(처리된, 편집된 등) 이미지는

로 명시된다. The sections below will introduce the mathematical notation of the manipulation operations performed on digital images. For the purpose of marking, the original image

The resulting manipulated (processed, edited, etc.) image is

It is specified as.

Brightness Adjustment-scalar addition / subtraction

및

사이의 관계는 아래와 같이 주어진다:In this image manipulation operation,

And

The relationship between is given by:

, 한 세트의 정수들이다. 예를 들어, 전형적인 디지털 칼라 이미지들은 8비트의 휘도(luminance) 표현을 가질 수 있고, 휘도는 256개의 상이한 레벨들을 가질 수 있고; 따라서 D = [- 255, 255]이다. 만약

인 방정식(1)이 D의 범위를 초과한다면, D에서 가장 가까운 값으로 잘려진다. At this time,

, A set of integers. For example, typical digital color images may have an 8-bit luminance representation, and the luminance may have 256 different levels; Thus D = [-255, 255]. if

If Eq. (1) exceeds the range of D, it is truncated to the nearest value of D.

조건은 명도의 증가와 동일하고, 따라서 이미지를 더 밝게 하며,

은 명도의 감소와 동일하고, 따라서 이미지를 더 어둡게 만든다. 디지털 이 미지 상에서 명도 조정의 효과는 명도 오퍼레이션(operation) 전 및 후에 이미지의 히스토그램(histogram)을 봄으로서 알 수 있다(도 6). 아래의 섹션들은 압축 도메인 내에 디지털 이미지들 상에서 명도 오퍼레이션을 적용하는 방법을 논의한다. 공간 도메인에서 이미지의 이미지 데이터를 또 다른 추가의(more) 에너지 콤팩트(compact) 도메인 속으로 변환하기 위해, JPEG 2000 표준은 웨이브릿 변환을 이용한다. 표준의 베이스 라인에서, 허용되는 필터들은 5-3 또는 9-7 웨이브릿 필터들 중의 하나이다; 분해 유형은 다이어딕이다(dyadic)(또한, Mallat으로 알려짐)

The condition is equal to the increase in brightness, thus making the image brighter,

Is equivalent to a decrease in brightness, thus darkening the image. The effect of brightness adjustment on the digital image can be seen by looking at the histogram of the image before and after the brightness operation (FIG. 6). The sections below discuss how to apply lightness operations on digital images within the compression domain. In order to convert image data of an image in the spatial domain into another more energy compact domain, the JPEG 2000 standard uses wavelet transform. In the standard baseline, the allowed filters are one of 5-3 or 9-7 wavelet filters; The decomposition type is dyadic (also known as Mallat)

In order to determine the operations required to translate brightness and contrast operations (described below) from the spatial domain to the wavelet domain (i.e., like the designation compression domain), we first define briefly below It is necessary to understand the transform operations known as discrete wavelet transforms (DWTs).

는 아래의 전개식에 의해 표현될 수 있고, 스케일링(scaling) 함수 및 그 함수의 웨이브릿 도함수(derivative)들의 식으로 정의될 수 있다. :Which continuous function

Can be expressed by the following expression, and can be defined by the expression of the scaling function and its wavelet derivatives. :

및

는 각각 스케일링 및 웨이브릿 계수들이고,

및

는 각각 스케일링 및 웨이브릿 함수들이다. 이 전개식에서, 제1 서메이션(summation)은

스케일에서

의 낮은 해상도 또는 정밀하지 않은 근사값을 나타내는 함수를 준다. 제2 서메이션에서 각 증가하는

에 대해, 더 높은 또는 더 정밀한 해상도 함수가 추가되고, 이것은 디테일(detail)을 증가시킨다. 위의 웨이브릿 전개식에서 계수들

및

의 세트는 함수

의 이산 웨이브릿 변환(discrete wavelet transform, DWT)이라고 지칭된다. At this time,

And

Are the scaling and wavelet coefficients, respectively,

And

Are the scaling and wavelet functions, respectively. In this development, the first summation is

On the scale

Gives a function that represents a low resolution or inexact approximation of. Increasing in the second summation

For, a higher or more precise resolution function is added, which increases the detail. Coefficients in Wavelet Expansion

And

Set of functions

Is referred to as a discrete wavelet transform (DWT).

스케일에서 스케일링 및 웨이브릿 계수들은

스케일에서의 스케일링 계수들에 관련된다 :By the following two relations,

In scaling, the scaling and wavelet coefficients

Relates to scaling factors at scale:

및

과 함께, 하위 스케일들에서의 웨이브릿 및 스케일링 계수들 또는 이산 웨이브릿 변환 계수들을 계산하는데 사용될 수 있다. The above equations indicate that the scaling factors at higher scale are scaling and wavelet filters, respectively.

And

Together, it can be used to calculate wavelet and scaling coefficients or discrete wavelet transform coefficients at lower scales.

가 원본 해상도에서 스케일이라면, In fact, it is assumed that the discrete signal at the original resolution corresponds to the scaling factors. therefore,

If is scale at original resolution,

Discrete wavelet transform coefficients at lower scales are obtained using the equation below.

By formulating the DWT of the signal, brightness adjustment is applicable in the wavelet domain. As mentioned above, the brightness operation in the spatial domain corresponds to:

는 이미지(원본 공간 해상도)의

레벨에서 적용되는 명도 오프셋을 의미한다. 명도 오퍼레이션을 거친 신호를 의미하는 프라임

기호를 사용하여, 상이한 스케일들

에서 웨이브릿 도메인 내의 대응하는 명도 오퍼레이션이 스케일링 및 웨이브릿 계수들(이산 웨이브릿 변환 (DWT) 계수들)의 항으로 아래와 같이 주어진다. Where constant

Of the image (original space resolution)

The brightness offset applied at the level. Prime means the signal after the brightness operation

Using scales, different scales

The corresponding brightness operation in the wavelet domain is given in terms of scaling and wavelet coefficients (discrete wavelet transform (DWT) coefficients) as follows.

스케일에서:

On the scale:

Using equations (5) and (8),

스케일에서:

On the scale:

At this time,

스케일에서:

On the scale:

At this time,

인 일반적인 스케일의 경우, 스케일 계수들은 아래와 같이 주어진다: Going into this format,

For a typical scale with, the scale factors are given by:

At this time,

을 스케일링 계수들인

에 더함으로써 수행될 수 있다. 항

은 스케일링 필터를 의미한다. In other words, the brightness operation in the wavelet domain is term

Scaling factors

Can be performed by adding to. term

Means a scaling filter.

의 값을 구하기 위해 유사한 분석을 할 수 있다. 간결하게 하기 위해 그리고 반복을 회피하기 위해(분석이 매우 유사하므로), 분석의 모든 단계들을 열거하지 않고 최종 결과만 제공할 것이다.

인 일반적인 스케일의 경우, 웨이브릿 계수들은 아래와 같이 주어진다:Wavelet coefficients

Similar analysis can be done to find the value of. For brevity and to avoid repetition (since the analysis is very similar), we will not list all the steps of the analysis and only provide the final result.

For a general scale of, the wavelet coefficients are given by:

At this time,

을 웨이브릿 계수들

에 합함으로써 수행될 수 있다. 항

은 웨이브릿 필터를 의미한다. In other words, the brightness operation in the wavelet domain is term

Wavelet coefficients

Can be performed by term

Means wavelet filter.

은 웨이브릿 도메인 내에 이미지에서 DC 시프트를 나타낸다. 웨이브릿 변환된 계수들은 모든 서브밴드들에서 원본 웨이브릿 변환 이미지로부터 변경된다. 그러나, 대부분의 에너지가 변환된 이미지의 LL 서브밴드에 집중되기 때문에, 공간 도메인에서 명도 오퍼레이션의 제1의 알맞은(good) 근사치를 얻기 위해 오직 LL 서브밴드에서만 이런 변경을 적용하는 것으로 충분하다. 따라서, 공간 도메인에서 DC 인자의 덧셈(/뺄셈)은 웨이브릿 도메인의 베이스 밴드(LL 서브밴드)에서 대응하는 DC 인자의 덧셈(/뺄셈)과 대략 동일하다. term

Denotes a DC shift in the image within the wavelet domain. The wavelet transformed coefficients are changed from the original wavelet transformed image in all subbands. However, since most of the energy is concentrated in the LL subbands of the transformed image, it is sufficient to apply this change only in the LL subbands to obtain the first good approximation of the brightness operation in the spatial domain. Thus, the addition (/ subtraction) of the DC factor in the spatial domain is approximately equal to the addition (/ subtraction) of the corresponding DC factor in the baseband (LL subband) of the wavelet domain.

Thus, as a first step, instead of applying brightness adjustment in the spatial domain, brightness adjustment in the wavelet domain is applicable. Here, the gain in this performance is approximately 10-15% (since the discrete wavelet transform is not as large as shown above). If decoding of other bands is not performed, then the gain is more than 10 times, as described below. The approximate change here only occurs in the LL subband. Therefore, there is no need to re-encode in other subbands. The JPEG 2000 standard and other encoders based on subband encoding and in particular wavelet transform (eg discrete wavelet transform), which do not use any inter-band correlation, are the same for other bands. To maintain code streams.

Thus, performance can be significantly improved by changing only the LL subbands in the image data in the compression domain.

Note that after quantization, each subband (from the discrete wavelet transform) is divided into rectangular blocks, that is, rectangles that do not overlap. Three spatially consecutive rectangles (one subband from each subband at each resolution level) contain a packet partition location or precinct; See FIG. 7. Each packet partition location is divided into non-overlapping rectangles called "code blocks", which form an input to the entropy coder described with reference to FIG. 2 above. The coded data of each codeblock is distributed over one or more layers in the code stream. Each layer improves image quality continuously and monotonically so that the decoder can decode the code block factors contained in each layer in turn. Data representing a particular tile, layer, component, resolution, and region appears in a code stream within a continuous segment called a packet. Packet data is aligned on 8 bits (1 byte) boundary. The data in the packet is ordered so that the factors from the LL, HL, LH, and HH subbands appear in that order. In each subband, code block factors appear in raster order and are limited within the range formed by the associated zone. Only those code blocks that contain samples from related subbands, limited to a region, are represented in a packet.

Assume a digital image with a resolution of 1280x960 pixels, equal to 1.2 mega pixels. At five levels, this image can be broken down into subbands of 640x480, 320x240, 160x120, 80x60, and 40x30 pixel dimensions. If the code blocks used to compress this image are 32x32 pixels (used in the most common block size), the number of code blocks in this image may be 1200 blocks. There are only four blocks in the LL subband. If only these four blocks are changed and the rest are unchanged, then only 0.33% (= 4/1200) of the entropy coded block data should be changed. When omitting image read operations, performing 0.33% of color transform operations, wavelet transform operations, and entropy coding operations and parts of bitstream forming operations reduce the overall time to less than 10% of the time to re-encode the entire image. Let's do it.

For high resolution images, the same analysis still applies. In fact, since the LL subband can be a smaller fraction of the entire image, significant gains in processing time can be obtained. In addition, bit stream forming operations may require less computational power. The changes required in the bit stream forming operation as a result of this operation are described below with reference to FIG. 8, which schematically illustrates an operation sequence for LL subband change according to an embodiment of the invention.

Referring to FIG. 8, in operation S90, the LL subband of the digital image to be changed is provided.

All code blocks in the LL subband must be re-encoded (ie, the first sub-operation S100). There are two possibilities for re-encoding the LL subband. The first possibility is shown in the first branch of the depicted operation sequence comprising operations S101, S102, S103, and the second possibility is shown in the depicted operation sequence comprising operations S104, S105, S106. It is shown at the second fork.

Referring to the first split, the entire LL subband is re-encoded. The LL subband does not require any inverse wavelet transform.

은 양의 또는 음의 값일 수 있음을 유의하라. The reconstructed data can be decoded and placed in the subband after applying the scalar sum (or difference), according to equations (12) to (15). Indicating a DC shift value

Note that can be a positive or negative value.

은 도 9에서 도시된 바와 같이, 비트 플레인

로부터 시작할 수 있다. 명도를 조정할 때, 합해지거나 빼질 DC 시프트 값은 대부분 원본 샘플 값의 일부이다. 따라서,

개의 최하위 비트 프레인들만이 방정식 (12) 내지 (15)에 따라 변경될 수 있다. 모든 제1

(

는 최상위비트(most significant bit)를 나타냄)은 동일하게 남아있을 수 있다. 결과적으로, 즉

최하위 비트 플레인들(last significant bit-planes)인, 변경에 고려되는 이런 비트 플레인들만이 디코딩, 조작, 재인코딩될 수 있고, 이것은 부분적 디코딩을 의미한다. 그렇지 않으면, 즉 DC 시프트 인자가 원본 샘플 값의 일부가 아니라면, 계산은 위에서 설명된 제1 갈림에 도시된 것과 같이 방정식 (12) 내지 (15)에 따라 수행되어야 한다. JPEG 2000 표준 또는 어느 다른 서브밴드 기반의 인코더의 인코딩 스피드를 추가로 증가시키기 위해, 압축 성능에서 약간의 하락을 희생하여 선택적 연산 인코딩이 생략될 수 있고 대신에

비트 플레인들이 미가공(raw) 비트들에 송신될 수 있다. Referring to the second fork, the LL subbands are partially re-encoded by decoding portions of the blocks and re-encoding them. Represents a DC shift value to sum or subtract

Is a bit plane, as shown in FIG.

You can start from When adjusting the brightness, the DC shift values to be summed or subtracted are mostly part of the original sample value. therefore,

Only the least significant bit planes can be changed according to equations (12) through (15). All first

(

Denotes the most significant bit) may remain the same. As a result, i.e.

Only those bit planes that are considered for modification, which are the last significant bit-planes, can be decoded, manipulated, and re-encoded, which means partial decoding. Otherwise, ie if the DC shift factor is not part of the original sample value, the calculation must be performed according to equations (12) to (15) as shown in the first branch described above. In order to further increase the encoding speed of the JPEG 2000 standard or any other subband based encoder, the optional computational encoding may be omitted at the expense of a slight drop in compression performance, instead.

Bit planes may be sent in raw bits.

All packets including the changed blocks in the second suboperation S110 need to be changed. Compressed data pertaining to these blocks is included and packet headers are changed in operations S111 and S112, for example.

In operation S124, the header information must be updated due to data inclusion. The operations below show more detail relating to the necessary packet header changes.

"Input / non-input" bits in the packet header: i.e. if a changed block is in a packet in the LL subband, the changed block in the original bit stream is not significant (or meaningless), whereas in some bit planes it is significant (or meaningless) The calculation bit shall be changed.

Code block calculation information (coded in tag trees) must be changed.

The number of empty bit planes in modified code blocks needs to be updated.

The code block length will need to be changed because brightness adjustments change the compressed data.

Changes in packet headers are very low in complexity. The JPEG 2000 standard provides a random access feature that allows an application to introduce the effect of accessing only those packets that need to be changed in the bit stream. The application is. Nevertheless, if the application still wants to perform new rate distortion optimization, it can do so with low complexity. These changes are expected to equal approximately 5% of the encoding time for the worst case scenario. Approximately acceleration by factor 20 can thus be proposed.

In the case of grayscale images, the brightness adjustment is applied to the single component image itself. For color images, the brightness adjustment only applies to the luminance component of the image. Chrominance components do not need to be changed.

의 적용을 확장함으로써(extend) 향상될 수 있음을 유의해야 한다. The approximation represents the DC shift value to additional subbands as defined in equations (12) to (15).

It should be noted that this can be improved by extending the application of.

Contrast Adjustment-scalar multiplication / division

In contrast manipulation, the operation is based on stretching / contracting the histogram of the image. An exemplary contrast manipulation is shown in FIG. 10. The histogram of the image represents the frequency of occurrence of gray levels in the image. Modeling the histogram on any property allows you to achieve the desired effects.

가 이미지

의 히스토그램을 나타낸다고 하자. 그러면,

는 밀도( intensity)

를 갖는

의 픽셀들의 수를 나타낸다. 히스토그램 변조의 관점에서, 원하는 히스토그램

를 갖는 이미지

가 발생될 것이다. 이것은 이미지

로부터

를 얻기 위해

를

로 맵핑할(map) 수 있는 함수/ 변환을 찾음으로써 달성될 수 있다.

Autumn image

Let's assume that we represent a histogram. then,

Is intensity

Having

Represents the number of pixels. In terms of histogram modulation, the desired histogram

Image with

Will be generated. This is an image

from

To get

To

This can be accomplished by finding a function / transformation that can be mapped to.

A popular linear way to apply contrast adjustment is to apply the following transformation to the luminance density of an image. (cf. Fig. 11)

이다.

및

파라미터들은 이미지의 어두운 도메인들, 중간 도메인들, 및 밝은 도메인들에 각각 대응하고, 이미지에서 상대적인 콘트라스트 조정을 결정한다. 콘트라스트 확장의 경우,

이고, 콘트라스트 축소의 경우,

이다. 따라서, 콘트라스트 조정은 적절한 인자(들)을 적용하여 픽셀들의 휘도 밀도를 단순히 스케일링함으로써 적용될 수 있음을 알 수 있다. Typically at this time

to be.

And

The parameters correspond to dark domains, intermediate domains, and bright domains of the image, respectively, and determine a relative contrast adjustment in the image. For contrast expansion,

, For contrast reduction,

to be. Thus, it can be seen that contrast adjustment can be applied by simply scaling the luminance density of the pixels by applying the appropriate factor (s).

를 사용하는 것이다.

가 이미지 픽셀들의 휘도 밀도들의 평균이라면, 이때 다음 식이 성립한다:A simpler way to adjust contrast is to use a single contrast adjustment factor for the entire image.

Is to use

If is the average of the luminance densities of the image pixels, then the following equation holds:

An analysis similar to that done above for the lightness operation will be performed. The contrast adjustment equation in the spatial domain is given by

At this time,

는 이미지(원본 공간 해상도)의

레벨에 적용된다. Where constant

Of the image (original space resolution)

Applies to the level.

심볼을 콘트라스트 오퍼레이션을 거친 신호를 나타내기 위해 사용함으로써, 상이한 스케일들

에서 웨이브릿 도메인 내의 대응하는 콘트라스트 오퍼레이션은, 스케일링 및 웨이브릿 계수들(이산 웨이브릿 변환)의 항으로 아래와 같이 주어진다. Prime

Different scales by using a symbol to represent a signal that has undergone contrast operation

The corresponding contrast operation in the wavelet domain in is given below in terms of scaling and wavelet coefficients (discrete wavelet transform).

스케일에서:

On the scale:

Using equations (5) and (18),

스케일에서:

On the scale:

At this time,

스케일에서:

On the scale:

At this time,

인 일반적인 스케일의 경우, 스케일 계수들은 아래와 같이 주어진다:Going into this format,

For a typical scale with, the scale factors are given by:

At this time,

로 스케일링 계수들

을 스케일링하고 그것에

를 합함으로써 수행될 수 있다. 항

은 스케일링 필터를 의미한다. In other words, the contrast operation in the wavelet domain is a contrast scaling factor.

Low scaling factors

And scale it to

Can be performed by summation. term

Means a scaling filter.

의 값을 구하기 위해 유사한 분석이 행해질 수 있다. 간결하게 하기 위해 그리고 반복을 회피하기 위해(분석이 매우 유사하므로), 분석의 모든 단계들을 열거하지 않고 최종 결과만 제공할 것이다.

인 일반적인 스케일의 경우, 웨이브릿 계수들은 아래와 같이 주어진다. Wavelet coefficients

Similar analysis can be done to find the value of. For brevity and to avoid repetition (since the analysis is very similar), we will not list all the steps of the analysis and only provide the final result.

For a typical scale, the wavelet coefficients are given by

At this time,

로 웨이브릿 계수들

를 스케일링하고 그것에

항을 합함으로써 수행될 수 있다. 항

웨이브릿 필터를 나타낸다. In other words, the contrast operation in the wavelet domain is a contrast scaling factor.

Low wavelet coefficients

And scale it to

Can be performed by combining terms. term

Represents a wavelet filter.

의 곱은 사실상 웨이블릿 도메인의 베이스밴드(LL 서브밴드)에서 그것의 변경과 대략 동일하다. 따라서, 공간 도메인에서 콘트라스트 조정을 적용하는 대신에, 웨이블릿 도메인에서의 콘트라스트 조정을 적용할 수 있다. 이런 수행에서 이득은 전에 설명한 바와 같이 대략 10에서 15%이다. As can be seen from the above equations, the wavelet transformed coefficients are changed from the original wavelet image in all subbands. However, since most of the energy is concentrated in the LL subbands of the transformed image, it is sufficient to apply this change only to the LL subbands to obtain a first reasonable approximation of the contrast operation in the spatial domain. Thus, contrast factor in the spatial domain

The product of is substantially equal to its change in the baseband (LL subband) of the wavelet domain. Thus, instead of applying contrast adjustment in the spatial domain, it is possible to apply contrast adjustment in the wavelet domain. The gain in this performance is approximately 10 to 15% as described previously.

The approximate change here only occurs in the LL subband. Re-encoding other subbands is therefore not necessary. The JPEG 2000 standard and possibly other subband coders and especially wavelet transform encoders, which do not use any correlation between bands, can keep the same code streams for different bands. Thus, a significant improvement in operation performance can be obtained by simply changing the compressed data of the LL subbands. As long as implementations and complexity issues are involved, similar discussions reapply here as in the brightness adjustment approach. Thus, the performance of contrast adjustment operations can be significantly improved by simply changing the compressed data of the LL subbands.

Complexity analysis and implementation aspects of contrast adjustment follow an approach very similar to the brightness adjustment approach, and thus will not be repeated here for brevity. 12 schematically illustrates an operation sequence for LL subband change according to an embodiment of the present invention.

12, in operation S90, the LL subband of the digital image to be changed is provided.

)). LL 서브밴드를 재인코딩하는 2개의 가능성들이 존재한다. 제1 가능성은 오퍼레이션들

, 및

을 포함하는 도시된 오퍼레이션 시퀀스의 제1 갈림 내에 도시되고, 제2 가능성은 오퍼레이션들

, 및

을 포함하는 도시된 오퍼레이션 시퀀스의 제2 갈림 내에 도시된다. All code blocks in the LL subband must be re-encoded (that is, the first sub-operation (

)). There are two possibilities for re-encoding the LL subband. The first possibility is operations

, And

Shown within a first fork of the illustrated operation sequence, the second possibility being operations

, And

It is shown within a second branch of the depicted operation sequence that includes.

인자는 1보다 크거나 또는 작을 수 있고 (각각 곱셈 오퍼레이션 및 나눗셈 오퍼레이션과 동일함) 시프트 값을 나타내는

는 양의 또는 음의 값일 수 있다는 것을 유의하라.Referring to the operations of the first fork, the entire LL subband is re-encoded. The LL subband does not require any inverse wavelet transform. The reconstructed data can be decoded and placed in the subband after applying the scalar product (or division) and sum (or difference) according to equations (22) to (25).

The argument can be greater than or less than 1 (equivalent to the multiplication and division operations, respectively) and represent a shift value.

Note that can be a positive or negative value.

및 합함수(summand)

에 따라 , 오퍼레이션 S105에 관해 위에서 주어진 논의에 유사한 변경이 비트플레인

로부터 시작할 수 있다. 따라서,

최하위 비트플레인만이 방정식들(22) 내지 (25)에 따라 변경될 수 있다. 모든 제1

(

는 최상위 비트를 나타냄)은 동일하게 남을 수 있다. 결론적으로, 즉

최하위 비트 플레인들인, 변경에 고려되는 이런 비트 플레인들만이 디코딩될 수 있고, 조작될 수 있고, 재인코딩될 수 있고, 이것은 또한 부분적 디코딩을 의미한다. 그렇지 않으면, 계산은 위에서 설명된 제1 갈림에 도시된 것과 같이 방정식 (22) 내지 (25)에 따라 수행되어야 한다. 어느 웨이브릿 변환 베이스 인코더 뿐만 아니라 JPEG 2000 표준의 인코딩 스피드를 추가로 증가시 키기 위해, 압축 성능에서 약간의 하락을 희생하여 선택적 연산 인코딩이 생략될 수 있고 대신에

비트 플레인들이 미가공 비트들에 송신될 수 있다.With reference to the operations of the second fork, the LL subband can be partially re-encoded by decoding portions of the blocks and re-encoding them. Under certain circumstances, namely

And summand

In accordance with this, a similar change is made to the discussion given above with respect to operation S105.

You can start from therefore,

Only the least significant bitplane can be changed according to equations (22) to (25). All first

(

Denotes the most significant bit) may remain the same. In conclusion, i.e.

Only those bit planes that are considered for modification, which are the least significant bit planes, can be decoded, manipulated and re-encoded, which also means partial decoding. Otherwise, the calculation must be performed according to equations (22) to (25) as shown in the first branch described above. In order to further increase the encoding speed of the JPEG 2000 standard as well as any wavelet transform base encoder, the optional computational encoding may be omitted at the expense of a slight drop in compression performance.

Bit planes may be transmitted in raw bits.

All packets including the changed blocks in the second suboperation S110 need to be changed. Compressed data pertaining to these blocks is included and packet headers are changed in operations S111 and S112, for example.

In another operation, the header information must be updated due to data ingestion. The operation below relates to the required packet header change in more detail.

"Input / non-input" bits in the packet header: i.e. if the changed block is in a packet in the LL subband, the changed block in the original bit stream is not significant (or meaningless), whereas in some bit planes it is significant (or meaningless) In order to do that, the bit of the bit should be changed.

Code block calculation information (coded in tag trees) must be changed.

The number of empty bit planes in modified code blocks needs to be updated.

The code block length will change because brightness adjustments change the compressed data.

의 애플리케이션을 확장함으로써 근사가 향상될 수 있음을 유의해야 한다. Indicating a linear factor for additional subbands as defined in equations (22) to (25)

It should be noted that the approximation can be improved by extending the application of.

Generalized Linear Filtering

를 주는 입력 이미지

상에 일반화 N-tap 리니어 필터링은 아래와 같이 쓰여질 수 있다:Filtered image

Input image

The generalized N- tap linear filtering on the phase can be written as:

은 적절한(appropriete) 변환

에 의한 인덱스 쌍

에 관련된다. In this case, index

Is the proper (appropriete) conversion

Index pairs by

Is related.

스케일링 파라미터들

는

tap의 필터의 필터 계수들이다. 방정식(26)은 스틸 이미지들의 리니어 필터링의 일반화된 형식이고 매우 다양한 리니어 이미지 필터링 오퍼레이션들을 설명하기 위해 사용될 수 있다. 이런 것들 중 가장 일반적인 것은 스무딩(smoothening)(저역 통과 필터링),샤프닝(sharpening)(고역 통과 필터링), 에버리징(averaging)(산술 평균 필터) 등이다.

Scaling parameters

Is

Filter coefficients of the filter of tap. Equation 26 is a generalized form of linear filtering of still images and can be used to describe a wide variety of linear image filtering operations. The most common of these are smoothing (low pass filtering), sharpening (high pass filtering), averaging (arithmetic mean filter), and so on.

From the discussion of brightness and contrast adjustments, it can be seen that both sum (or difference) and multiplication (or scaling) of pixel coefficients of a still JPEG 2000 encoded image are possible in the compression domain. From equation 26 it can be seen that generalized linear filtering is a simple extension of these two basic operations, i.e. it can be linearly decomposed into a finite number of sum and multiplication operations to obtain the desired effects. Thus, by the superposition principle of the results on brightness and contrast adjustment and the superposition of the linear operations, any linear image processing operation can be performed in the compression domain using the inventive concept. In particular, two important filtering operations will be described below:

Sharpening operation

로 곱함으로써 계수가 향상될 수 있다. Sharpening is an image processing operation in which edges are enhanced within an image so that the image can look sharper than before. Sharpening an image is similar to enhancing high frequency information (such as edges) of an image. Wavelet coefficients in the high frequency subbands of the wavelet transformed image include information related to the edges. Thus, the coefficients in the high frequency subbands need to be improved. Assume an image where the coefficients in the high frequency subbands have three levels of wavelet decomposition (FIG. 4). The coefficients at level 1 of the transformed image represent the sharpest high frequency content in the image, the coefficients at the next level (level 2) represent the next clearest high frequency content, and so on. Therefore, in order to improve the high frequency information, it is inevitable that the coefficients at level 1 should be improved most. Constant argument

The coefficient can be improved by multiplying by.

Depending on the strength of the desired sharpening operation, the coefficients in the next level of subbands can also be improved. If light shifting operations are required, only the coefficients of level 1 subbands can be improved. If a stronger sharpening operation is required, then the coefficients may be improved in the subbands of the next level (level 2) until reaching the highest level of the wavelet pyramid that can improve the high frequency subbands of that level, and so on. . However, since it contains only low frequency content, the LL subband of the converted image is not changed.

Subbands with changed coefficients need to be entropy encoded as previously described in the previous sections. Thus, the gain in acceleration depends on the number of subbands that must be changed and in turn on the strength of the desired sharpening operation.

Smoothing operation

Image smoothing is similar to suppressing the high frequency content of an image. Wavelet coefficients in the high frequency subbands of the wavelet transformed image include information related to the edges and similar high frequency information. By suppressing this high frequency information, the edges can be smoothed. Thus, the smoothing operation in the JPEG 2000 standard can be achieved by suppressing the coefficients in the high frequency subbands. Assume an image with three levels of wavelet decomposition (cf. FIG. 4). The coefficients at level 1 of the transformed image represent the sharpest high frequency content in the image, the coefficients at the next level (level 2) represent the next clearest high frequency content, and so on. Thus, in order to suppress high frequency information, it is inevitable that the coefficients at level 1 should be suppressed most. Thus, the first step is to suppress the coefficients of level 1 to be equal to zero.

Depending on the strength of the desired smoothing operation, the coefficients in the next level subbands may also be suppressed. If a light smoothing operation is required, only the coefficients of level 1 subbands can be suppressed. If a stronger smoothing operation is required, the coefficients may also be suppressed in subbands of the next level (level 2) until the highest level of the wavelet pyramid can suppress the high frequency subbands of that level, and so on. to be. However, since the LL subband of the converted image contains only low frequency content, it is not changed.

Subbands with changed coefficients need to be entropy coded as previously described in the previous sections. Thus, the gain in acceleration depends on the number of subbands that must be changed, and in turn on the strength of the desired smoothing operation.

Implementation embodiment

)에 제공되고, 그것은 압축 도메인에서 이미지 데이터의 LL 서브밴드의 공급을 가능하게 하는 도 8 및 도 12의 오퍼레이션들은 S90을 참조하여 정의된 것과 같이 특정 요구된 기능성(fuctionality)을 갖는 도 3의 비트 스트림 파싱 모듈(310)일 수 있다. 이때, LL 서브밴드의 이미지 데이터는 엔트로피 디코더(

)에 제공되고, 그것은 도 8 및 도 12의 오퍼레이션들(

,

)을 참조하여 정의된 것과 같은, 특정 요구된 기능을 갖는 도 3의 엔트로피 디코더(320)일 수 있 다. 이미지 조작은 방정식(12) 내지 방정식(15) 및/ 또는 방정식(22) 내지 방정식(25)을 제공한다(각각 도 8 및 도 12의 각각의 오퍼레이션들(

,

) 뿐만 아니라 오퍼레이션들(102, 105)) 참조). 조작(500) 이후에, 압축 도메인에서 변경된 이미지 데이터는 엔트로피 인코더(

)에 제공되고, 그것은 도 8 및 도 12의 오퍼레이션들(

,

)을 참조하여 정의된 것과 같이 특정 요구된 기능성을 갖는 도 3의 비트 스트림 파싱 모듈(240)일 수 있다. 마지막으로, 변경된 이미지 데이터는 변경된 이미지의 압축된 이미지 표현을 형성하기 위해 이미지 데이터로 재산입된다(re-included). 비트 스트림 어셈블리 모듈(

)은 재어셈블리(re-assembly)를 제공하고, 비트 스트림 어셈블리 모듈(

)은 도 8 및 도 12의 오퍼레이션(S110)을 참조하여 정의된 것과 같이 특정 요구된 기능을 갖는 도 3의 어셈블리 모듈(

)일 수 있다. Referring to FIG. 13, an exemplary implementation of the methodology described in detail above is shown. The implementation may be part of the implementation of the JPEG 2000 encoder / decoder shown in FIG. 3, described in detail above with reference to it. According to the inventive concept, image data (i.e., encoded domain or compressed image representation) in the compressed domain is converted into a bitstream parsing module (i.e.

8 and 12, which enable the supply of LL subbands of image data in the compression domain, the bits of FIG. 3 with specific required functionality as defined with reference to S90. Stream parsing module 310. At this time, the image data of the LL subband is entropy decoder (

), Which is the operations of FIGS. 8 and 12 (

,

May be the entropy decoder 320 of FIG. 3 with certain desired functionality, such as defined with reference to FIG. Image manipulation provides equations (12) to (15) and / or equations (22) to (25) (respective operations of FIGS. 8 and 12, respectively)

,

) As well as operations 102, 105). After operation 500, the image data changed in the compression domain is entropy encoder (

), Which is the operations of FIGS. 8 and 12 (

,

Bit stream parsing module 240 of FIG. 3 with specific desired functionality as defined with reference to FIG. Finally, the altered image data is re-included with the image data to form a compressed image representation of the altered image. Bit stream assembly module (

) Provides re-assembly, and the bit stream assembly module (

) Is an assembly module of FIG. 3 having a particular requested function as defined with reference to operations S110 of FIGS. 8 and 12.

May be).

Like one described with reference to FIGS. 1A and 1B, the modules shown in FIG. 13 are preferably implemented in an image processing module. One skilled in the art has described the invention in detail with reference to the JPEG 2000 standard operating in accordance with Discrete Wavelet Transform (DWT), but the inventive concept is applicable to any subband based encoding methodology, in particular with wavelet transform encoding. You can expect it to be possible. In conclusion, it should not be understood that the present invention is limited to the JPEG 2000 standard and discrete wavelet transform encoders / decoders, respectively.

The modules, components, and functionalities described in connection with embodiments of the present invention and to be included in a CE device having image capture capability according to an embodiment of the present invention may be included in a portable CE device. It should be noted that, such as, may be configured as a data processing unit. In addition, the modules, components, and functionalities may include one or more data processing units (microprocessors), wherein the one or more data processing units include instructions for executing essential processing operations to perform functions and operations. Or microcontrollers or sections of code for execution on application specific integrated circuits (ASICs) Alternatively, or in addition, modules, components, and functionalities or portions thereof. The functions may be implemented based on one or more hardware modules, such as application specific integrated circuits (ASIC) The implementation of the inventive concept is not limited to any particular implementation, including software and / or hardware implementation.

Some features and aspects of the invention have been shown and described with reference to specific embodiments only by way of example and not by way of limitation. Those skilled in the art are aware that alternative implementations and various changes to the disclosed embodiments are within the scope and contemplation of the invention. Accordingly, it is intended that the invention be limited only by the scope of the appended claims.

Claims (33)

In a method of modifying image data in the compressed domain, the compressed image data can be obtained from the image data in the spatial domain by a subband encoder, in particular based on wavelet transform, At least one linear offset

And at least one linear scaling factor

Linear function composed of at least one of the groups containing

To provide The linear function

Is applicable to the image data in the spatial domain; The linear function

Converting into a compressed domain; And Linear function on the image data in the spatial domain

The transformed linear function such that the approximated values of the modified subbands are obtained as a result.

Applying to the defined number of subbands of the compressed image data, Method for changing image data in the compressed domain. The linear function of claim 1, wherein the linear function

Converting into a compressed domain: At least one converted offset value

To produce this result, the at least one linear offset

Converting the data into a compressed domain; Method for changing image data in the compressed domain. The method of claim 2,

Defined spatial resolution level

Representing a linear offset which is a constant to be applied to the image data in the spatial domain, Method for changing image data in the compressed domain. The method of claim 2, wherein the code blocks of the modified subbands are Entropy decode the defined number of subbands, The transformed linear function in the code blocks of the defined number of subbands such that modified subbands result as a result.

, Where The at least one converted offset value

Added to each value of this code block; The at least one linear scaling factor

Is multiplied by each value of the code blocks; Entropy re-encoding the modified subbands comprising the codeblocks,  Are all re-encoded,  Method for changing image data in the compressed domain. The method of claim 2, wherein the code blocks of the modified subbands, A defined number of said code blocks

Entropy decodes the bitplanes of The converted offset value

Where the converted offset value represents a portion of the original sample value.

Is partially re-encoded by applying a to the bit planes, Method for changing image data in the compressed domain. The method of claim 1, wherein the function

Can define a linear filter function to be applied to the image data, the linear filter function including in particular either brightness enhancement, or contrast enhancement, or both, Method for changing image data in the compressed domain. The method of claim 1, Providing a generalization function applicable to the image data in the spatial domain, and The at least one linear offset of the generalization function

And at least one linear scaling factor

The linear function being at least one of a group comprising a

Linearly decomposing with Method for changing image data in the compressed domain. The method of claim 7, wherein the linearly disintegrating comprises: Generating a linearly approximated function based on said generalization function, Method for changing image data in the compressed domain. 2. The linear function of claim 1, wherein the transformed linear function

Adapting packet headers of packets containing code blocks subject to the above of Method for changing image data in the compressed domain. 2. The linear function of claim 1, wherein the transformed linear function

Is applied to at least a first level of a baseband including a subband LL representing high frequency information of the image data, Method for changing image data in the compressed domain. 8. The method of claim 7, wherein the generalization function enables at least sharpening filtering and smoothing filtering. A computer program product comprising a computer readable medium having program code recorded on a computer readable medium, the computer program product comprising: The program code makes it possible, in particular by means of a subband encoder based on a wavelet transform, to change the image data in the compressed domain obtained from the image data in the spatial domain, the program code being: At least one linear offset

And at least one linear scaling factor

Linear function composed of at least one of the groups containing

Provides a linear function

A code section applicable to the image data in the spatial domain; The linear function

A code section for transforming into a compressed domain; And A linear function on the image data in the spatial domain

The transformed linear function such that the approximated values of the modified subbands are obtained as a result.

A code section for applying a to a defined number of subbands of the compressed image data, Computer program product. 13. The linear function of claim 12, wherein the linear function

Converting into a compressed domain: At least one converted offset value

To produce this result, the at least one linear offset

Containing a section of code for converting into a compressed domain, Computer program product. The method of claim 13,

Defined spatial resolution level

Representing a linear offset which is a constant to be applied to the image data in the spatial domain, Computer program product. The method of claim 13, wherein the code blocks of the modified subbands are A code section for entropy decoding the defined number of subbands, The transformed linear function in the code blocks of the defined number of subbands such that the modified subbands result as a result.

, Where The at least one converted offset value

Is added to each value of said code blocks; The at least one linear scaling factor

A code section multiplied by each value of the code blocks; And By means of a code section for entropy re-encoding the modified subbands comprising the codeblocks, Computer program product, all re-encoded. The method of claim 13, wherein the code blocks of the modified subbands, A defined number of said code blocks

A code section for entroping decoding bitplanes; And The converted offset value

Where the converted offset value represents a portion of the original sample value.

A computer program product partially re-encoded by a section of code for applying the bit planes to the bit planes. 13. The apparatus of claim 12, wherein the transformed linear function

And a code section for modifying packet headers of packets containing code blocks subject to the above. 13. The method of claim 12, wherein the function

Can define a linear filter function to be applied to the image data, the linear filter function in particular comprising either or both of brightness enhancement or contrast enhancement. The method of claim 12,  A section of code for providing a generalization function applicable to the image data in the spatial domain; And The at least one linear offset of the generalization function

And at least one linear scaling factor

The linear function being at least one of a group comprising a

A computer program product comprising a section of code for linearly decomposed into a. 20. The method of claim 19, wherein the linear decomposition is: Generating a linearly approximated function based on said generalization function. 13. The apparatus of claim 12, wherein the transformed linear function

Is applied to at least a first level of a baseband comprising a subband (LL) representing high frequency information of the image data. 20. The computer program product of claim 19, wherein the generalization function enables at least sharpening filtering and smoothing filtering. An image captureable consumer electronic device comprising: a subband encoder, in particular based on a wavelet transform, for converting image data in a spatial domain into a data image in a compressed domain; The imaging device is: At least one linear offset

And at least one linear scaling factor

A linear function comprised of at least one of a group comprising a and applicable to the image data in the spatial domain

; The linear function

Convert to a compressed domain, Linear function on the image data in the spatial domain

The transformed linear function such that the approximated values of the modified subbands are obtained as a result.

And an image manipulation module for applying the to the defined number of subbands of the compressed image data. The method of claim 23, wherein The image manipulation module comprises at least one transformed offset value

At least one linear offset to occur as a result of this

Image-capable consumer electronic device configured to convert the data into a compressed domain. The method of claim 24,

Defined spatial resolution level

An image captureable consumer electronic device representing a linear offset that is a constant to be applied to the image data in the spatial domain. 25. The apparatus of claim 24, wherein the image manipulation module is configured to operate on code blocks of the modified subbands and to re-encode all the code blocks, The entropy decoder is configured to decode the defined number of subbands, The image manipulation module performs the transformed linear filter function on the code blocks of the defined number of subbands such that modified subbands result as a result.

Is configured to apply, wherein The at least one converted offset value

Added to each value of this code block; The at least one linear scaling factor

Is multiplied by each value of the code blocks;  An entropy encoder is configured to re-encode the modified subbands containing the codeblocks, Consumer electronic device capable of capturing images. The method of claim 24, The image manipulation module is configured to operate on code blocks of the modified subbands, and to partially re-encode the code blocks, An entropy decoder is a defined number of said code blocks

To decode bitplanes of; The image manipulation module is adapted to convert the offset value.

Is configured to apply to the bit planes, the offset value

An image capture capable consumer electronic device representing part of the original sample value. The method of claim 24, The transformed linear function

And an image manipulation module configured to update packet headers of packets containing code blocks subject to the application of the image capture module. The method of claim 24, wherein the function

Can define a linear filter function to be applied to the image data, the linear filter function comprising in particular either or both of brightness enhancement or contrast enhancement, Consumer electronic device capable of capturing images. The method of claim 24, A generalization function applicable to the image data in the spatial domain; And The at least one linear offset of the generalization function

And at least one linear scaling factor

The linear function being at least one of a group comprising a

An image manipulation module configured to linearly disassemble into Consumer electronic device capable of capturing images. The method of claim 30, wherein the linear decomposition is, An image captureable consumer electronic device comprising generating a linearly approximated function based on the generalization function. 25. The apparatus of claim 24, wherein the transformed linear filter function

Is applied to at least a first level of a baseband comprising a subband (LL) representing high frequency information of the image data. 33. The system of claim 32, wherein the linear filter function

Is an image captureable consumer electronic device that enables at least sharpening filtering and smoothing filtering.

Images