KR20220164562A - Method and apparatus for alternative multi-learning rates in neural image compression - Google Patents

Abstract

Neural network based alternative end-to-end (E2E) image compression (NIC) performed by at least one processor includes: receiving an input image for an E2E NIC framework; determining a replacement image based on the training model of the E2E NIC framework; encoding the replacement image to generate a bitstream; and mapping the replacement image to the bitstream to generate a compressed representation of the input image. Also, the input can be partitioned into blocks, in which case an alternate representation is determined for each block and each block is encoded instead of the entire alternate image.

Description

Method and apparatus for alternative multi-learning rates in neural image compression

Traditional hybrid video codecs are difficult to optimize overall. An improvement of a single module may not produce a coding gain of the overall performance. Recently, standards groups and companies are actively exploring potential needs for standardization of future video coding technology. These standards groups and companies have established the JPEG-AI group focused on AI-based end-to-end neural image compression using Deep Neural Networks (DNN). The Chinese AVS standard has also formed an AVS-AI special group to study neural image and video compression techniques. The success of recent approaches has led to increasing industry interest in advanced neural image and video compression methodologies.

According to example embodiments, substitutional end-to-end (E2E) neural image compression using a neural network performed by at least one processor. ) (NIC) method, comprising: receiving an input image for an E2E NIC framework; Based on the training model of the E2E NIC framework, determining a substitute image; encoding the replacement image to generate a bitstream; and mapping the replacement image to the bitstream to generate a compressed representation.

According to exemplary embodiments, an apparatus for alternative end-to-end (E2E) neural image compression (NIC) using a neural network, comprising: at least one memory configured to store program code; and at least one processor configured to read the program code and operate as directed by the program code. The program code may include receiving code configured to cause at least one processor to receive an input image for the E2E NIC framework; a first determining code configured to cause at least one processor to determine a replacement image based on a training model of an E2E NIC framework; a first encoding code configured to cause at least one processor to encode the replacement image to generate a bitstream; and mapping code configured to cause at least one processor to map the surrogate image to the bitstream to generate a compressed representation.

According to example embodiments, a non-transitory computer-readable medium storing instructions, which, when executed by at least one processor for alternative end-to-end (E2E) neural image compression (NIC), at least Causes a processor to receive an input image to the E2E NIC framework, determine a replacement image based on a training model of the E2E NIC framework, encode the replacement image to generate a bitstream, and A non-transitory computer-readable medium is provided that allows mapping an image to a bitstream to generate a compressed representation.

1 is a diagram of an environment in which the methods, apparatuses, and systems described herein, in accordance with embodiments, may be implemented.
FIG. 2 is a block diagram of example components of one or more devices of FIG. 1 .
3 is an exemplary diagram illustrating a replacement learning-based image coding preprocessing model.
4 illustrates an example of block-wise image coding.
5 is a flow diagram of an alternative end-to-end (E2E) neural image compression (NIC) method according to embodiments.
6 is a block diagram of an apparatus for alternative end-to-end (E2E) neural image compression (NIC), according to embodiments.

Embodiments receive a picture and tune elements of the substitutional representation of a picture to optimize the rate-distortion performance of coding the substitutional representation of the picture based on an end-to-end (E2E) optimized framework. determining an alternative representation of the picture by performing an optimization process that The E2E optimized framework may be a pre-trained artificial neural network (ANN) based image or video coding framework. An alternate representation of a picture may be encoded to generate a bitstream. In an artificial neural network-based video coding framework, different modules are jointly optimized from input to output to improve an end goal (eg, rate-distortion performance) by performing a machine learning process, resulting in an end-to-end End (E2E) Optimized Neural Image Compression (NIC) can be generated.

1 is a diagram of an environment 100 in which the methods, apparatuses, and systems described herein may be implemented, in accordance with embodiments.

As shown in FIG. 1 , environment 100 may include user device 110 , platform 120 , and network 130 . Devices in environment 100 may be interconnected through wired connections, wireless connections, or a combination of wired and wireless connections.

User device 110 includes one or more devices capable of receiving, generating, storing, processing, and/or providing information associated with platform 120 . For example, user device 110 may include a computing device (eg, a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (eg, a smart phone, a wireless telephone ( radiotelephone, etc.), a wearable device (eg, a pair of smart glasses or a smart watch), or a similar device. In some implementations, user device 110 can receive information from platform 120 and/or transmit information to platform 120 .

Platform 120 includes one or more devices as described elsewhere herein. In some implementations, platform 120 can include a cloud server or group of cloud servers. In some implementations, platform 120 can be designed to be modular in which software components can be swapped in or swapped out. As such, platform 120 can be easily and/or quickly reconfigured for different uses.

In some implementations, as shown, platform 120 can be hosted in cloud computing environment 122 . In particular, while the implementations described herein describe platform 120 as being hosted in cloud computing environment 122, in some implementations, platform 120 may not be cloud-based (i.e., a cloud computing environment may be implemented outside of the cloud), or may be partially cloud-based.

Cloud computing environment 122 includes an environment hosting platform 120 . The cloud computing environment 122 does not require end-user (eg, user device 110) knowledge about the physical location and configuration of the system(s) and/or device(s) hosting the platform 120. It can provide services such as computation, software, data access, storage, etc. As shown, a cloud computing environment 122 is a group of computing resources 124 (collectively referred to as "computing resources 124" and individually referred to as "computing resources 124"). can include

Computing resource 124 includes one or more personal computers, workstation computers, server devices, or other types of computing and/or communication devices. In some implementations, computing resource 124 can host platform 120 . Cloud resources may include compute instances running on computing resource 124 , storage devices provided by computing resource 124 , data transfer devices provided by computing resource 124 , and the like. In some implementations, computing resource 124 can communicate with other computing resources 124 via wired connections, wireless connections, or a combination of wired and wireless connections.

As further shown in FIG. 1 , computing resource 124 includes one or more applications ("APP") 124-1, one or more virtual machines ("VM") 124-2, virtualized storage ("VM") 124-2, VS") 124-3, one or more hypervisors ("HYP") 124-4, and the like.

Applications 124 - 1 include one or more software applications that may be provided to or accessed by user device 110 and/or platform 120 . Application 124 - 1 may eliminate the need to install and run software applications on user device 110 . For example, application 124 - 1 may include software associated with platform 120 and/or any other software that may be provided via cloud computing environment 122 . In some implementations, one application 124-1 can send/receive information to/from one or more other applications 124-1 via virtual machine 124-2.

The virtual machine 124-2 includes a software implementation of a machine (eg, computer) that executes programs like a physical machine. The virtual machine 124-2 may be either a system virtual machine or a process virtual machine, depending on the purpose and degree of correspondence by the virtual machine 124-2 to any real machine. A system virtual machine may provide a complete system platform supporting the execution of a complete operating system ("OS"). A process virtual machine can run a single program and can support a single process. In some implementations, virtual machine 124 - 2 can execute on behalf of a user (eg, user device 110 ) and perform data management, synchronization, or long-duration data transfers. It is possible to manage the infrastructure of the cloud computing environment 122, such as.

Virtualized storage 124 - 3 includes one or more storage systems and/or one or more devices that use virtualization techniques within the storage systems or devices of computing resource 124 . In some implementations, within the context of a storage system, types of virtualizations can include block virtualization and file virtualization. Block virtualization can refer to abstracting (or separating) logical storage from physical storage so that the storage system can be accessed regardless of the physical storage or heterogeneous structure. Separation can allow administrators of storage systems flexibility in how they manage storage for end users. File virtualization can remove dependencies between data accessed at the file level and where files are physically stored. This may enable optimization of storage usage, server consolidation, and/or performance of non-disruptive file migrations.

Hypervisor 124 - 4 may provide hardware virtualization techniques that allow multiple operating systems (eg, “guest operating systems”) to run concurrently on a host computer such as computing resource 124 . Hypervisor 124-4 may present a virtual operating platform to guest operating systems and may manage the execution of guest operating systems. Multiple instances of various operating systems may share virtualized hardware resources.

Network 130 includes one or more wired and/or wireless networks. For example, the network 130 may be a cellular network (eg, a fifth generation (5G) network, a long-term evolution (LTE) network, a third generation (3G) network, a CDMA (code division multiple access networks, etc.), public land mobile networks (PLMNs), local area networks (LANs), wide area networks (WANs), metropolitan area networks (MANs), telephone networks (e.g., Public Switched Telephone Network (PSTN) )), private networks, ad hoc networks, intranets, the Internet, fiber-based networks, etc., and/or combinations of these or other types of networks.

The number and arrangement of devices and networks shown in FIG. 1 is provided as an example. Indeed, additional devices and/or networks, fewer devices and/or networks, different devices and/or networks, or devices and/or network arranged differently than those shown in FIG. there may be Also, two or more devices shown in FIG. 1 may be implemented within a single device, or a single device shown in FIG. 1 may be implemented as multiple distributed devices. Additionally or alternatively, a set of devices (eg, one or more devices) of environment 100 may perform one or more functions described as being performed by another set of devices of environment 100 .

FIG. 2 is a block diagram of example components of one or more devices of FIG. 1 .

Device 200 may correspond to user device 110 and/or platform 120 . As shown in FIG. 2, the device 200 includes a bus 210, a processor 220, a memory 230, a storage component 240, an input component 250, an output component 260, and a communication interface ( 270) may be included.

Bus 210 includes components that allow communication between components of device 200 . Processor 220 is implemented in hardware, firmware, or a combination of hardware and software. The processor 220 may include a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an ASIC ( application-specific integrated circuit), or other type of processing component. In some implementations, processor 220 includes one or more processors that can be programmed to perform a function. Memory 230 may include random access memory (RAM), read only memory (ROM), and/or other types of dynamic or static storage devices (e.g., random access memory) that store information and/or instructions for use by processor 220. eg, flash memory, magnetic memory, and/or optical memory).

Storage component 240 stores information and/or software related to operation and use of device 200 . For example, the storage component 240 may include a hard disk (eg, a magnetic disk, an optical disk, a magneto-optic disk, and/or a solid state disk), a compact disc (CD), ), digital versatile disc (DVD), floppy disk, cartridge, magnetic tape, and/or other types of non-transitory computer readable media, along with a corresponding drive.

Input component 250 allows device 200 to receive information via, for example, user input (eg, a touch screen display, keyboard, keypad, mouse, buttons, switches, and/or microphone). contains components. Additionally or alternatively, the input component 250 may include sensors (eg, global positioning system (GPS) components, accelerometers, gyroscopes, and/or actuators) for sensing information. Output components 260 include components that provide output information from device 200 (eg, a display, a speaker, and/or one or more light-emitting diodes (LEDs)).

Communications interface 270 may be a transceiver-like component (e.g., a transceiver) that enables device 200 to communicate with other devices via, for example, a wired connection, a wireless connection, or a combination of wired and wireless connections. and/or separate receiver and transmitter). Communications interface 270 may allow device 200 to receive information from and/or provide information to other devices. For example, the communication interface 270 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, and the like. there is.

Device 200 may perform one or more processes described herein. Device 200 may perform these processes in response to processor 220 executing software instructions stored by a non-transitory computer readable medium, such as memory 230 and/or storage component 240 . A computer readable medium is defined herein as a non-transitory memory device. A memory device includes memory space within a single physical storage device or memory space distributed across multiple physical storage devices.

Software instructions may be read into memory 230 and/or storage component 240 from another computer readable medium or from another device via communication interface 270 . When executed, software instructions stored in memory 230 and/or storage component 240 may cause processor 220 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Accordingly, implementations described herein are not limited to any particular combination of hardware circuitry and software.

The number and arrangement of components shown in FIG. 2 is provided as an example. In practice, device 200 may include additional components, fewer components, different components, or components arranged differently than those shown in FIG. 2 . Additionally or alternatively, a set of components (eg, one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200 .

를 컴퓨트하는 것이다. 그 다음, 압축된 표현

를 DNN 디코더에 대한 입력으로서 사용하여 이미지

를 재구성한다. 일부 NIC 방법들은 VAE(variational autoencoder) 구조를 취할 수 있으며, 여기서, DNN 인코더들은 전체 이미지 x를 그것의 입력으로서 직접 사용하고, 이는 출력 표현(즉, 압축된 표현

)을 컴퓨트하기 위해 블랙 박스(black box)처럼 작동하는 네트워크 계층들의 세트를 통과한다. 이에 대응하여, DNN 디코더들은 전체 압축된 표현

를 그것의 입력으로서 취하며, 이는 재구성된 이미지

를 컴퓨트하기 위해 다른 블랙 박스처럼 작동하는 다른 네트워크 계층들의 세트를 통과한다. 레이트-왜곡(Rate-Distortion)(R-D) 손실은 다음 타깃 손실 함수(loss function)

을 사용하여 트레이드-오프 하이퍼파라미터

를 갖고 재구성된 이미지

의 왜곡 손실

와 압축된 표현

의 비트 소비(bit consumption) R 사이의 트레이드-오프를 달성하도록 최적화된다.Given an input image x, the NIC's target uses image x as input to the DNN encoder to obtain a compact compressed representation for storage and transmission purposes.

is to compute Then, the compressed expression

image using as input to the DNN decoder

reconstitute Some NIC methods may take a variational autoencoder (VAE) structure, where DNN encoders directly use the whole image x as its input, which is an output representation (i.e., a compressed representation).

) through a set of network layers that act like a black box to compute . Correspondingly, DNN decoders use the full compressed representation

takes as its input, which is the reconstructed image

It passes through another set of network layers that act like other black boxes to compute . The Rate-Distortion (RD) loss is the target loss function

Trade-off hyperparameters using

image reconstructed with

distortion loss of

and compressed expression

is optimized to achieve a trade-off between the bit consumption R of

Embodiments relating to preprocessing suggest that for each input image to be compressed, online training can be used to find the best replacement and compress this replacement instead of the original image. By using this replacement, the encoder can achieve better compression performance. This method is used as a preprocessing step to boost the compression performance of any E2E NIC compression method. It does not require any training or fine-tuning of the pretrained compression model itself or any training data. Detailed methods and apparatus for preprocessing models, in accordance with one or more embodiments, will now be described.

3 is an exemplary diagram illustrating a replacement learning-based image coding preprocessing model.

는 길이

을 갖는 비트-스트림에 매핑되고(인코딩 매핑(300)), 이는 그 다음에 왜곡 손실

를 갖고

에서 원본 공간에 다시 매핑된다(디코딩 매핑(310)).Learning-based image compression can be viewed as a two-step mapping process. As shown in FIG. 3, the original image in a high-dimensional space

is the length

(encoding mapping 300), which is followed by distortion loss

have

is mapped back to the original space in (decoding mapping 310).

가 길이

을 갖는 비트-스트림에 매핑되도록 존재하는 경우, 이는 그 다음에 왜곡 손실

를 갖고 원본 이미지

에 더 가까운 공간

에 매핑된다. 거리 측정 또는 손실 함수가 주어지면, 대체 이미지를 사용하여 더 나은 압축이 달성될 수 있다. 최상의 압축 성능은 수학식 (1)에 따른 타깃 손실 함수의 전역 최소값(global minimum)에서 달성된다. 다른 예시적인 실시예에서는, 디코딩된 이미지와 원본 이미지

사이의 차이들을 감소시키기 위해, ANN의 임의의 중간 단계들에서 대체들이 발견될 수 있다.In an exemplary embodiment, as shown in FIG. 3 , the alternate image

the length

If present to be mapped to a bit-stream with

have the original image

space closer to

is mapped to Given a distance measure or loss function, better compression can be achieved using alternative images. The best compression performance is achieved at the global minimum of the target loss function according to Equation (1). In another exemplary embodiment, the decoded image and the original image

Substitutions can be found at any intermediate stages of the ANN, in order to reduce the differences between

Unlike the model training phase, where gradients are used to update the model's parameters, in preprocessing models, the model's parameters are fixed, and the gradient can be used to update the input image itself. By replacing non-differentiable parts with differentiable parts (e.g., by replacing quantization with noise injection), the entire model becomes differentiable (thus, the gradients are backpropagated ( can be backpropagated). Thus, the optimization can be iteratively solved by gradient descent.

과 함께, 하이퍼파라미터들이 학습 프로세스를 위해 사용된다. 예를 들어, 스텝 사이즈는 학습 프로세스에서 수행되는 그래디언트 하강법 알고리즘 또는 역전파 계산에서 사용될 수 있다. 반복들의 수는 학습 프로세스가 종료될 수 있는 시기를 제어하기 위한 최대 반복들의 수의 임계값으로서 사용될 수 있다.There are two main hyperparameters in this preprocessing model: step size and number of steps. The step size represents the 'learning rate' of online training. Images with different types of content can correspond to different step sizes to achieve the best optimization results. The number of steps represents the number of updates operated. target loss function

With , hyperparameters are used for the learning process. For example, the step size can be used in gradient descent algorithms or backpropagation calculations performed in the learning process. The number of iterations can be used as a threshold for the maximum number of iterations to control when the learning process can end.

4 illustrates an example of block-wise image coding.

In an exemplary embodiment, image 400 may first be segmented into blocks (illustrated by dashed lines in FIG. 4 ), and the segmented blocks may be compressed instead of image 400 itself. Compressed blocks are shaded in FIG. 4 and blocks to be compressed are not shaded. The divided blocks may have the same size or may not have the same size. The step size for each block may be different. To this end, different step sizes may be assigned to image 400 to achieve better compression results. Block 410 is an example of one of the divided blocks having height h and width w.

In an exemplary embodiment, the image may be compressed without being split into blocks, and the entire image is the input to the E2E NIC model. Different images may have different step sizes to achieve optimized compression results.

In another exemplary embodiment, the step size may be selected based on characteristics of the image (or block), eg, the RGB variance of the image. In embodiments, RGB may refer to a red-green-blue color model. Also, in another exemplary embodiment, the step size may be selected based on the RD performance of the image (or block). Thus, according to the embodiments, multiple replacements are generated based on multiple step sizes, and the one with better compression performance is selected.

5 is a flow diagram of an alternative end-to-end (E2E) neural image compression (NIC) method 500 according to embodiments.

In some implementations, one or more process blocks in FIG. 5 may be performed by platform 120 . In some implementations, one or more process blocks in FIG. 11 can be performed by another device or group of devices that is separate from or includes platform 120 , such as user device 110 .

As shown in FIG. 5 , at operation 510 , method 500 includes receiving an input image for an E2E NIC framework.

At operation 520 , the method 500 includes determining a replacement image based on the training model of the E2E NIC framework. The replacement image may be determined by the optimization process of the training model of the E2E NIC framework. This is done by adjusting elements of the input image to generate alternative representations, selecting the elements with the least distortion loss between the input image and the alternative representations and using them as the replacement image. In addition, the training model of the E2E NIC framework may be trained based on the learning rate of the input image, the number of updates to the input image, and the distortion loss. One or more replacement images may be determined based on the different learning rates of the input image. Learning rates are selected based on the characteristics of the input image. The characteristics can be the RGB variance of the input image or the RD performance of the input image.

When the input image is divided into blocks, the replacement block generates replacement block representations by adjusting elements of the divided blocks, and selects elements with the smallest distortion loss between the divided blocks and replacement block representations, By using it as a replacement block, it can be determined by the optimization process of the E2E NIC framework's training model. In addition, the training model of the E2E NIC framework may be trained based on the learning rate of the divided blocks, the number of updates for the divided blocks, and the distortion loss of each of the divided blocks. One or more replacement blocks may be determined based on the different learning rates of the partitioned blocks. Learning rates are selected based on the characteristics of the partitioned blocks. The characteristics can be the RGB variance of the partitioned blocks or the RD performance of the partitioned blocks.

At operation 530, the method 500 includes encoding the replacement image to generate a bitstream.

At operation 540, the method 500 includes mapping the replacement image to the bitstream to generate a compressed representation. In embodiments, one or more of the bitstream or compressed representation may be transmitted to, for example, a decoder and/or a receiving device.

Method 500 may also include segmenting the input image into one or more blocks. In this case, operations 520-540 are performed on each of the blocks instead of the entire input image. That is, the method 500 includes determining a replacement block for each of the partitioned blocks based on a training model of the E2E NIC framework, encoding the replacement block to generate a block bitstream, and converting the replacement block into a block It will further include mapping to the bitstream to generate the compressed block. The partitioned blocks can have the same size or different sizes, and each block has a different learning rate.

Method 500 may use an artificial neural network based on a pretrained image coding model, where the parameters of the artificial neural network are fixed and a gradient is used to update the input image.

5 shows example blocks of a method, in some implementations, the method may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those shown in FIG. 5 . can Additionally or alternatively, two or more of the blocks of the method may be performed in parallel.

6 is a block diagram of an alternative end-to-end (E2E) neural image compression (NIC) device 600, according to embodiments.

As shown in FIG. 6 , the device 600 includes a receive code 610 , a decision code 620 , an encoding code 630 , and a mapping code 640 .

Receive code 610 is configured to cause at least one processor to receive an input image for the E2E NIC framework.

The decision code 620 is configured to cause the at least one processor to determine a replacement image based on a training model of the E2E NIC framework.

Encoding code 630 is configured to cause at least one processor to encode the replacement image to generate a bitstream.

Mapping code 640 is configured to cause at least one processor to map the surrogate image to a bitstream to generate a compressed representation.

Apparatus 600 may further include segmentation code configured to cause at least one processor to segment an input image into one or more blocks. In this case, the decision code 620, the encoding code 630, and the mapping code 640 are performed using each of the divided blocks instead of the entire input image.

The replacement image determined by the decision code 620 may be determined by an optimization process of the training model of the E2E NIC framework. This includes adjustment code configured to cause at least one processor to adjust elements of the input image to generate alternate representations, and to cause the at least one processor to cause the element with the least distortion loss between the input image and the alternate representations. This is done by selection code that is configured to allow selection of images as replacement images. In addition, the training model of the E2E NIC framework may be trained based on the learning rate of the input image, the number of updates to the input image, and the distortion loss. One or more replacement images may be determined based on the different learning rates of the input image. Learning rates are selected based on the characteristics of the input image. The characteristics can be the RGB variance of the input image or the RD performance of the input image.

Additionally, apparatus 600 may use an artificial neural network based on a pretrained image coding model, where the parameters of the artificial neural network are fixed and a gradient is used to update the input image.

6 shows example blocks of an apparatus, in some implementations the apparatus may include additional blocks, fewer blocks, different blocks, or blocks arranged differently than those shown in FIG. 6 . can Additionally or alternatively, two or more of the blocks of the device may be combined.

Embodiments herein describe E2E image compression methods. The methods utilize alternative mechanisms to improve NIC coding efficiency by using a flexible and general framework that accommodates various types of quality metrics.

E2E image compression methods according to one or more embodiments may be used individually or combined in any order. Further, each of the methods (or embodiments), encoder and decoder may be implemented by processing circuitry (eg, one or more processors or one or more integrated circuits). In one example, one or more processors execute programs stored on non-transitory computer readable media.

The foregoing disclosure provides examples and descriptions, but is not intended to be exhaustive or to limit implementations to the precise forms disclosed. Modifications and variations are possible in light of the above disclosure or may be obtained from practice of the implementations.

As used herein, the term component is intended to be interpreted broadly as hardware, firmware, or a combination of hardware and software.

It will be apparent that the systems and/or methods described herein may be implemented in different forms of hardware, firmware, or a combination of hardware and software. The actual specialized control hardware or software code used to implement these systems and/or methods is not limiting the implementations. Thus, the operation and behavior of systems and/or methods have been described herein without reference to specific software code, and software and hardware may be designed to implement systems and/or methods based on the description herein. should understand

Although combinations of features may be recited in the claims and/or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. Indeed, many of these features may be combined in ways not specifically recited in the claims and/or disclosed in the specification. Each dependent claim listed below may directly depend on only one claim, but the disclosure of possible implementations includes each dependent claim in combination with every other claim in the set of claims.

No element, action, or command used herein should be construed as such unless expressly described as important or essential. Also, as used herein, the articles “a” and “an” are intended to include one or more items, and may be used interchangeably with “one or more”. Also, as used herein, the term "set" is intended to include one or more items (eg, related items, unrelated items, combinations of related and unrelated items, etc.) and can be used interchangeably with "one or more". Where only one item is intended, the term "one" or similar language is used. Also, as used herein, the terms “has,” “have,” “having” and the like are intended to be open-ended terms. Further, the phrase “based on” is intended to mean “based, at least in part, on” unless expressly stated otherwise.

Claims (20)

A substitutional end-to-end (E2E) neural image compression (NIC) method using a neural network performed by at least one processor, comprising:
Receiving an input image for the E2E NIC framework;
determining a substitute image based on a training model of the E2E NIC framework;
encoding the replacement image to generate a bitstream; and
mapping the surrogate image to the bitstream to generate a compressed representation;
Including, method. According to claim 1,
splitting the input image into one or more blocks;
determining a replacement block for each of the one or more blocks based on a training model of the E2E NIC framework;
encoding the replacement block to generate a block bitstream; and
mapping the replacement block to the block bitstream to generate a compressed block;
In addition,
wherein the one or more blocks have the same size, and each block of the one or more blocks has a different learning rate. The method of claim 1, wherein the replacement image is determined by performing an optimization process of a training model of the E2E NIC framework, the optimization process comprising:
adjusting elements of the input image to generate substitute representations; and
Selecting elements having the smallest distortion loss between the input image and the replacement representations and using them as the replacement image
Including, method. 2. The method of claim 1, wherein a training model of the E2E NIC framework is trained based on a learning rate of the input image, a number of updates to the input image, and a distortion loss. 5. The method of claim 4, wherein a plurality of replacement images are determined based on learning rates;
wherein the learning rates are selected based on characteristics of the input image. 6. The method of claim 5, wherein the characteristics of the input image are one of RGB variance of the input image and RD performance of the input image. The method of claim 1, wherein the training model of the E2E NIC framework is an artificial neural network based on pre-trained image coding,
wherein the parameters of the artificial neural network are fixed and a gradient is used to update the input image. Apparatus for alternative end-to-end (E2E) neural image compression (NIC) using neural networks, comprising:
at least one memory configured to store program code; and
at least one processor configured to read the program code and operate as directed by the program code
Including, the program code,
receiving code configured to cause at least one processor to receive an input image for an E2E NIC framework;
a first determining code configured to cause at least one processor to determine a replacement image based on a training model of the E2E NIC framework;
a first encoding code configured to cause at least one processor to encode the replacement image to generate a bitstream; and
a first mapping code configured to cause at least one processor to map the surrogate image to the bitstream to generate a compressed representation;
Including, device. According to claim 8,
segmentation code configured to cause at least one processor to segment the input image into one or more blocks;
second decision code configured to cause at least one processor to determine a replacement block for each of the one or more blocks based on a training model of the E2E NIC framework;
a second encoding code configured to cause at least one processor to encode the replacement block to generate a block bitstream; and
a second mapping code configured to cause at least one processor to map the replacement block to the block bitstream to generate a compressed block;
In addition,
wherein the one or more blocks have the same size, and each block of the one or more blocks has a different learning rate. The method of claim 8, wherein the replacement image is determined by performing an optimization process of a training model of the E2E NIC framework, the optimization process comprising:
adjustment code configured to cause at least one processor to adjust elements of the input image to generate alternative representations; and
selection code configured to cause at least one processor to select elements with the smallest distortion loss between the input image and the alternate representations to use as the alternate image;
Including, device. 9. The apparatus of claim 8, wherein a training model of the E2E NIC framework is trained based on a learning rate of the input image, a number of updates to the input image, and a distortion loss. 12. The method of claim 11, wherein a plurality of alternate images are determined based on learning rates;
wherein the learning rates are selected based on characteristics of the input image. 13. The apparatus of claim 12, wherein the characteristics of the input image are one of RGB dispersion of the input image and RD performance of the input image. The method of claim 8, wherein the training model of the E2E NIC framework is an artificial neural network based on pre-trained image coding,
wherein the parameters of the artificial neural network are fixed and a gradient is used to update the input image. A non-transitory computer readable medium storing instructions,
The instructions, when executed by at least one processor for alternative end-to-end (E2E) neural image compression (NIC), cause the at least one processor to:
Receive an input image to the E2E NIC framework;
Determine a replacement image based on the training model of the E2E NIC framework;
encoding the replacement image to generate a bitstream;
A non-transitory computer-readable medium that causes mapping of the replacement image to the bitstream to generate a compressed representation. 16. The method of claim 15, wherein the instructions, when executed by at least one processor, cause the at least one processor to further:
divide the input image into one or more blocks;
determine a replacement block for each of the one or more blocks based on a training model of the E2E NIC framework;
encoding the replacement block to generate a block bitstream;
mapping the replacement block to the block bitstream to generate a compressed block;
The one or more blocks have the same size, and each block of the one or more blocks has a different learning rate. 16. The method of claim 15, wherein the instructions, when executed by at least one processor, cause the at least one processor to further perform an optimization process of a training model of the E2E NIC framework, the optimization process comprising: ,
adjusting elements of the input image to generate alternative representations; and
Selecting elements with the smallest distortion loss between the input image and the replacement representations and using them as the replacement image.
A non-transitory computer readable medium comprising a. 16. The non-transitory computer-readable medium of claim 15, wherein a training model of the E2E NIC framework is trained based on a learning rate of the input image, a number of updates to the input image, and distortion loss. 19. The method of claim 18, wherein a plurality of alternate images are determined based on learning rates;
the learning rates are selected based on characteristics of the input image;
wherein the characteristics of the input image are one of RGB variance of the input image and RD performance of the input image. The method of claim 15, wherein the training model of the E2E NIC framework is an artificial neural network based on pre-trained image coding,
wherein the parameters of the artificial neural network are fixed and a gradient is used to update the input image.

Images