WO2023010750A1 - 一种图像颜色映射方法、装置、终端设备及存储介质 - Google Patents

一种图像颜色映射方法、装置、终端设备及存储介质 Download PDF

Info

Publication number
WO2023010750A1
WO2023010750A1 PCT/CN2021/138080 CN2021138080W WO2023010750A1 WO 2023010750 A1 WO2023010750 A1 WO 2023010750A1 CN 2021138080 W CN2021138080 W CN 2021138080W WO 2023010750 A1 WO2023010750 A1 WO 2023010750A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
color
processed
layer
adjustment parameters
Prior art date
Application number
PCT/CN2021/138080
Other languages
English (en)
French (fr)
Inventor
陈翔宇
章政文
董超
乔宇
Original Assignee
中国科学院深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 中国科学院深圳先进技术研究院 filed Critical 中国科学院深圳先进技术研究院
Publication of WO2023010750A1 publication Critical patent/WO2023010750A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/92—Dynamic range modification of images or parts thereof based on global image properties
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/90—Determination of colour characteristics
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the present application relates to the technical field of image processing, and in particular to an image color mapping method, device, terminal equipment and storage medium.
  • the local feature information of the image is generally extracted by the neural network to realize the conversion between the original image and the optimized image.
  • the amount of information that can be represented by local feature information is limited, and the method based on local feature information will cause more artificial artifacts in the optimized image, and there are deviations in color, resulting in poor quality of the optimized image.
  • the present application provides an image color mapping method, device, terminal equipment, and storage medium, which can improve the problem of poor quality of optimized images in current image color modification tasks.
  • an embodiment of the present application provides an image color mapping method, the method includes: acquiring an image to be processed, inputting the image to be processed into a trained color mapping model for processing, and outputting an optimized image, the color mapping model includes main network and color conditional network;
  • the color condition network includes at least one color condition module and a feature conversion module connected in sequence, at least one color condition module is used to extract global color feature information from the low-resolution image of the image to be processed, and the feature conversion module is used to convert the global color feature information converted into N sets of adjustment parameters, and the N sets of adjustment parameters are respectively used to adjust the N intermediate features extracted by the main network during the process of converting the image to be processed into an optimized image, and N is an integer greater than or equal to 1.
  • the image color mapping method provided by this application uses at least one color condition module in the color condition network to extract and compress the global color features of the low-resolution input image to be processed, which can avoid Introduces artifacts to the optimized image. Convert the global feature information into adjustment parameters through the feature conversion module to represent the color prior information of the image to be processed, and use the adjustment parameters to adjust the intermediate features extracted in the main network to adapt to the color prior information of different images to be processed The corresponding optimized image is generated, thereby improving the quality of the optimized image.
  • the color condition module includes a convolutional layer, a pooling layer, a first activation function, and an IN layer connected in sequence.
  • the feature conversion module includes a Dropout layer, a convolutional layer, a pooling layer, and N fully connected layers; the Dropout layer, the convolutional layer, and the pooling layer are connected in sequence to process the global color feature information to obtain the condition vector; N fully-connected layers are respectively used to perform feature conversion on the conditional vector to obtain N sets of adjustment parameters.
  • the main network includes N GFM layers, N sets of adjustment parameters are respectively input to the N GFM layers, and the GFM layers are used to adjust the intermediate features input to the GFM layer according to the adjustment parameters.
  • the main network also includes N convolutional layers and N-1 second activation functions, and the N GFM layers are respectively connected to the output terminals of the N convolutional layers, and the convolution kernel size of the convolutional layer is 1 ⁇ 1 .
  • setting the size of the convolution kernel in the network to 1 ⁇ 1 can effectively reduce the number of network parameters, thereby reducing the computational complexity of the network.
  • the image to be processed is a video frame obtained from the SDR video, and each frame of the video frame in the SDR video is optimized by a color mapping model to output a corresponding optimized image, and the frame corresponding to the SDR video is obtained after combining the frames. HDR video.
  • an image color mapping device which includes:
  • an acquisition unit configured to acquire an image to be processed
  • the processing unit is used to input the image to be processed into the trained color mapping model for optimization processing, and output the optimized image.
  • the color mapping model includes a main network and a color condition network, and the color condition network includes a plurality of color condition modules connected in sequence and Feature conversion module, a plurality of color condition modules are used to extract global color feature information from the low-resolution image of the image to be processed, and the feature conversion module is used to convert the global color feature information into N groups of adjustment parameters; N groups of adjustment parameters are used respectively To adjust the N intermediate features extracted by the main network during the process of converting the image to be processed into an optimized image, where N is an integer greater than or equal to 1.
  • the color condition module includes a convolutional layer, a pooling layer, a first activation function, and an IN layer connected in sequence.
  • the embodiment of the present application provides a terminal device, including a memory, a processor, and a computer program stored in the memory and operable on the processor.
  • a terminal device including a memory, a processor, and a computer program stored in the memory and operable on the processor.
  • the processor executes the computer program, any of the above-mentioned first aspect one method.
  • an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of the above-mentioned first aspects is implemented.
  • an embodiment of the present application provides a computer program product, which, when the computer program product is run on a terminal device, causes the terminal device to execute the method in any one of the foregoing first aspects.
  • Fig. 1 is a network structure diagram of a color mapping model provided by an embodiment of the present application
  • Fig. 2 is a schematic diagram of HDR and SDR color gamut representation ranges provided by an embodiment of the present application
  • Fig. 3 is a schematic flow chart of converting HDR video to SDR video according to an embodiment of the present application
  • Fig. 4 is a schematic structural diagram of an image color mapping device provided by an embodiment of the present application.
  • Fig. 5 is a schematic structural diagram of a terminal device provided by an embodiment of the present application.
  • embodiments of the present application provide an image color mapping method, device, terminal device, and storage medium.
  • the image to be processed is optimized through the color mapping model provided by this application, and an optimized image with higher contrast and rich colors is output.
  • the color mapping model includes the main network and the color conditional network.
  • the color conditional network is used to extract adjustment parameters from the low-resolution image of the image to be processed, and use the adjustment parameters to adjust the main network in the process of converting the image to be processed into an optimized image.
  • the generated intermediate features can adaptively adjust the color mapping between the image to be processed and the optimized image according to the characteristics of different images to be processed, so as to avoid artifacts in the optimized image and improve the quality of the optimized image.
  • the color mapping model can be deployed in an image processing device.
  • the image processing device may be a mobile terminal such as a smart phone, a tablet computer, or a camera, or a device capable of processing image data such as a desktop computer, a robot, or a server.
  • the color mapping model provided in this application includes a main network and a color conditional network.
  • the color condition network includes at least one color condition block (Color Condition Block, CCB) and a feature conversion module connected in sequence.
  • At least one color condition module is used to extract global color characteristic information from the low-resolution image of the image to be processed.
  • the feature conversion module is used to convert the global color feature information into N sets of adjustment parameters.
  • N sets of adjustment parameters are respectively used to adjust N intermediate features extracted by the main network during the process of converting the image to be processed into an optimized image, and N is an integer greater than or equal to 1.
  • the image to be processed may be down-sampled by a certain multiple (for example, down-sampled by 4 times) to obtain a corresponding low-resolution image.
  • the low-resolution image is obtained by downsampling the image to be processed by 4 times.
  • the size of the low-resolution image is the same as that of the image to be processed, but the number of pixels per unit area of the image to be processed is equal to the number of pixels per unit area of the low-resolution image 4 times the amount.
  • the color mapping model provided by this application extracts and compresses the global color features of the low-resolution image of the input image to be processed through at least one color condition module. Compared with the method based on local feature extraction, it can avoid introducing artificial artifacts into the optimized image. film.
  • the global feature information is converted into adjustment parameters to represent the color prior information of the image to be processed, and the adjustment parameters are used to adjust the intermediate features of the image to be processed extracted in the main network, so that the color of the image to be processed can be adjusted according to the color prior information of the image to be processed.
  • the corresponding optimized image is adaptively generated based on the empirical information, thereby improving the quality of the optimized image.
  • the color condition module includes a convolutional layer, a pooling layer, a first activation function and an IN (Instance Normalization) layer connected in sequence.
  • the color condition module can perform global feature extraction on the input low-resolution image. Compared with the method based on image local feature extraction, it can effectively represent the global feature information of the image to be processed, thereby avoiding the introduction of artificial artifacts in the optimized image. film.
  • the feature conversion module includes dropout layer, convolutional layer, pooling layer and N fully connected layers.
  • the dropout layer, the convolution layer and the pooling layer are connected in sequence to process the global color feature information extracted by at least one color condition module to obtain a condition vector.
  • N fully connected layers are used to perform feature conversion on the conditional vectors to obtain N sets of adjustment parameters. It should be noted that each fully connected layer processes the condition vector to obtain a set of adjustment parameters, and finally the number of fully connected layers can be the same as the number of sets of adjustment parameters.
  • the color mapping model shown in FIG. 1 includes four color condition modules connected in sequence.
  • the size of the convolution kernel in the convolution layer is 1 ⁇ 1, and the pooling layer adopts average pooling.
  • the first activation function is a non-linear activation function LeakyReLU.
  • the main network includes N global feature modulation (Global Feature Modulation, GFM) layers, and N sets of adjustment parameters are input to the N GFM layers.
  • the GFM layer can adjust the intermediate features input to the GFM layer according to the adjustment parameters.
  • the main network can be any neural network model that can realize the task of color optimization or color conversion.
  • the color condition module provided by this application can be connected to the main network to obtain the color mapping model provided by this application.
  • the main network can be a fully convolutional network. That is, the main network includes N convolutional layers and N-1 second activation functions, and the N GFM layers are respectively connected to the output terminals of the N convolutional layers.
  • the main network is used to convert the image to be processed into an optimized image, and during the conversion process, N convolutional layers can be used to extract N intermediate features.
  • the size of the convolution kernel in each convolution layer is 1 ⁇ 1.
  • the second activation function may be a nonlinear activation function ReLU.
  • the convolution kernel size of the convolution layer is 1 ⁇ 1, and the parameters of the network model are less, which can effectively reduce the complexity of calculation, improve the efficiency of operation, and further improve the performance of the algorithm. real-time.
  • the number of fully connected layers in the color conditional network and the number of groups of correspondingly generated adjustment parameters should be designed based on the number of convolutional layers in the main network. For example, if the main network includes N convolutional layers, it means that the N intermediate features generated by the N convolutional layers need to be adjusted. Therefore, the color conditional network needs to output N sets of adjustment parameters corresponding to the N intermediate features, and the main network needs to have N GFM layers to adjust the N intermediate features according to the N sets of adjustment parameters.
  • the main network includes 3 convolution (Conv) layers, 3 GFM layers and 2 second activation function (ReLU) layers.
  • the main network sequentially includes a convolutional layer, a GFM layer, a ReLU layer, a convolutional layer, a GFM layer, a ReLU layer, a convolutional layer, and a GFM layer from input to output.
  • the color conditional module includes 4 CCB layers connected in sequence;
  • the feature conversion module can include a Dropout layer, a convolution (Conv) layer, an average pooling (Avgpool) layer, and respectively connected with 3 fully connected (FC) layers connected by the condition vector (Condition Vector) output by the average pooling layer.
  • Each fully connected layer can convert the condition vector into a corresponding set of adjustment parameters ( ⁇ , ⁇ ), and the color conditional network outputs a total of 3 sets of adjustment parameters (ie, adjustment parameter 1, adjustment parameter 2, and adjustment parameter 3).
  • Each GFM layer in the main network adjusts the intermediate features input to the GFM layer according to the corresponding adjustment parameters, which can be expressed as formula (1):
  • xi represents the i-th intermediate feature input to the GFM layer
  • GFM(xi ) represents the adjustment result of the GFM layer on the input intermediate feature xi according to the adjustment parameters ( ⁇ , ⁇ ).
  • the color mapping model uses the color conditional network to extract the color feature information of the image to be processed as prior information, which is used to adjust the intermediate features in the main network, so that the color mapping model can be based on the color prior features of different images to be processed
  • the information adaptively outputs an optimized image corresponding to the image to be processed, avoiding artificial artifacts in the optimized image, thereby improving the quality of the optimized image.
  • the color mapping model provided in this application is versatile and can be applied to any task that requires color optimization or color conversion of the image to be processed. Such as image editing, image retouching and toning, image coloring, SDR (Standard Dynamic Range) video conversion to HDR (High Dynamic Range) video, etc.
  • image editing image retouching and toning
  • image coloring image coloring
  • SDR Standard Dynamic Range
  • HDR High Dynamic Range
  • Fig. 2 is a schematic diagram showing ranges of HDR and SDR color gamuts.
  • BT.709 and BT.2020 are TV parameter standards issued by ITU (International Telecommunication Union)
  • DCI-P3 is a color gamut standard formulated by the American film industry for digital cinema. It can be seen from Figure 2 that BT.2020 has the largest color gamut among DCI-P3, BT.709 and BT.2020, followed by DCI-P3, and BT.709 has the smallest color gamut. .
  • HDR video uses the BT.709 color gamut
  • HDR video uses the wider BT.2020 color gamut or DCI-P3 color gamut.
  • the HDR video can show higher contrast and richer colors than the SDR video.
  • the initial color mapping model can be trained by designing corresponding training sets and loss functions, so as to obtain color mapping models suitable for different tasks.
  • the training process and application of the color mapping model provided by this application will be exemplarily described below.
  • Step 1 obtain the training set.
  • the training set can include a plurality of SDR video frame samples and HDR video frame samples corresponding to a plurality of SDR video frame samples one-to-one.
  • an SDR video sample and its corresponding HDR video sample are acquired first.
  • SDR video samples and corresponding HDR video samples can be obtained from public video websites. It is also possible to perform SDR and HDR processing on videos in the same RAW data format, respectively, to obtain SDR video samples and corresponding HDR video samples. It is also possible to use the SDR camera and the HDR camera respectively to shoot corresponding SDR video samples and HDR video samples in the same scene. After the SDR video samples and their corresponding HDR video samples are acquired, the SDR video samples and their corresponding HDR video samples are respectively frame-drawn to obtain multiple SDR video frame samples and one-to-one correspondence with multiple SDR video frame samples HDR video frame samples.
  • Step 2 using the training set and the preset loss function to train the initial color mapping model to obtain the trained color mapping model.
  • the SDR video frame samples are input into the main network of the initial color mapping model.
  • Multiple SDR video frame samples are respectively down-sampled to obtain multiple low-resolution images, and the low-resolution images are input into the color condition network of the initial color mapping model to obtain adjustment parameters to adjust the initial color mapping model Predicted HDR video frames.
  • the preset loss function f is used to describe the HDR video frame predicted by the initial color mapping model
  • the L2 loss between HDR video frame sample H can be expressed as formula (2):
  • the initial color mapping model can be iteratively trained by the gradient descent method until the model converges, and the trained color mapping model can be obtained.
  • FIG. 3 is a schematic flowchart of a method for converting an HDR video to an SDR video provided in an embodiment of the present application. From Figure 3, we can know that the SDR video can be converted into HDR video with high contrast and more colors based on the trained color mapping model. Firstly, frame extraction is performed on the obtained SDR video to be processed, and the video frame obtained from the SDR video is the image to be processed input to the color mapping model shown in FIG. 1 .
  • the video frame For each video frame in the SDR video, the video frame is fed into the main network of the trained color mapping model.
  • the video frame is subjected to 4 times downsampling processing to obtain a low-resolution image, and the low-resolution image is input into the color condition network of the trained color mapping model to obtain multiple adjustment parameters.
  • Multiple GFM layers in the main network adjust the intermediate features input to the GFM layer according to the corresponding adjustment parameters, and finally output the optimized image corresponding to the video frame.
  • the HDR video corresponding to the SDR video is obtained.
  • the color mapping model provided by this application can be directly added to the post-processing process of terminal devices such as cameras, so as to improve the quality of images or videos captured by terminal devices such as cameras from the perspective of software.
  • the color mapping model provided in this application can also be used as an image/video post-stage color enhancement method to optimize the color of existing SDR or other image data.
  • the embodiment of the present application also provides an image color mapping device.
  • the embodiment of the device corresponds to the embodiment of the aforementioned image color mapping method.
  • this embodiment of the device does not repeat the details of the aforementioned method embodiments one by one. , but it should be clear that the device in this embodiment can correspondingly implement all the content in the foregoing method embodiments.
  • FIG. 4 is a schematic structural diagram of an image color mapping device provided in an embodiment of the present application.
  • the image color mapping device 100 provided in this embodiment includes an acquisition unit 101 and a processing unit 102 .
  • the acquiring unit 101 is configured to acquire an image to be processed.
  • the processing unit 102 is configured to input the image to be processed into the trained color mapping model for optimization processing, and output the optimized image.
  • the color mapping model includes a main network and a color conditional network.
  • the color conditional network includes multiple color conditional modules and feature conversion modules connected in sequence. Multiple color conditional modules are used to extract global color feature information from low-resolution images of images to be processed.
  • the feature conversion module is used to convert the global color feature information into N sets of adjustment parameters. N sets of adjustment parameters are respectively used to adjust N intermediate features extracted by the main network during the process of converting the image to be processed into an optimized image, and N is an integer greater than or equal to 1.
  • the color condition module includes a convolutional layer, a pooling layer, a first activation function, and an IN layer connected in sequence.
  • the feature conversion module includes a dropout layer, a convolutional layer, a pooling layer and N fully connected layers.
  • the dropout layer, the convolutional layer and the pooling layer are connected in sequence to process the global color feature information and obtain the conditional vector.
  • N fully-connected layers are respectively used to perform feature conversion on the condition vector to obtain N sets of adjustment parameters.
  • the main network includes N GFM layers, N sets of adjustment parameters are respectively input to the N GFM layers, and the GFM layers are used to adjust the intermediate features input to the GFM layer according to the adjustment parameters.
  • the main network also includes N convolutional layers and N-1 second activation functions, and the N GFM layers are respectively connected to the output terminals of the N convolutional layers, and the convolution kernel size of the convolutional layer is 1 ⁇ 1 .
  • the image to be processed is a video frame obtained from the SDR video, and each frame of the video frame in the SDR video is optimized by a color mapping model to output a corresponding optimized image, and the frame corresponding to the SDR video is obtained after combining the frames. HDR video.
  • a terminal device 200 in this embodiment includes: a processor 201 , a memory 202 , and a computer program 204 stored in the memory 202 and operable on the processor 201 .
  • the computer program 204 can be run by the processor 201 to generate instructions 203 , and the processor 201 can implement the steps in the above embodiments of the image color mapping method according to the instructions 203 .
  • the processor 201 executes the computer program 204, the functions of the modules/units in the above-mentioned device embodiments are realized, for example, the functions of the unit 101 and the unit 102 shown in FIG. 4 .
  • the computer program 204 can be divided into one or more modules/units, and one or more modules/units are stored in the memory 202 and executed by the processor 201 to complete the present application.
  • One or more modules/units may be a series of computer program instruction segments capable of accomplishing specific functions, and the instruction segments are used to describe the execution process of the computer program 204 in the terminal device 200 .
  • FIG. 5 is only an example of the terminal device 200, and does not constitute a limitation to the terminal device 200. It may include more or less components than those shown in the figure, or combine certain components, or different components. , for example, the terminal device 200 may also include an input and output device, a network access device, a bus, and the like.
  • the processor 201 can be a central processing unit (Central Processing Unit, CPU), and can also be other general-purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), on-site Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
  • a general-purpose processor may be a microprocessor, or the processor may be any conventional processor, or the like.
  • the storage 202 may be an internal storage unit of the terminal device 200 , such as a hard disk or memory of the terminal device 200 .
  • the memory 202 can also be an external storage device of the terminal device 200, such as a plug-in hard disk equipped on the terminal device 200, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash memory card (Flash Card) and so on.
  • the memory 202 may also include both an internal storage unit of the terminal device 200 and an external storage device.
  • the memory 202 is used to store computer programs and other programs and data required by the terminal device 200 .
  • the memory 202 can also be used to temporarily store data that has been output or will be output.
  • the terminal device provided in this embodiment can execute the foregoing method embodiment, and its implementation principle and technical effect are similar, and details are not repeated here.
  • the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in the foregoing method embodiment is implemented.
  • the embodiment of the present application further provides a computer program product, which, when the computer program product runs on a terminal device, enables the terminal device to implement the method described in the foregoing method embodiments when executed.
  • the above integrated units are realized in the form of software function units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the procedures in the methods of the above embodiments in the present application can be completed by instructing related hardware through computer programs, and the computer programs can be stored in a computer-readable storage medium.
  • the computer program When executed by a processor, the steps in the above-mentioned various method embodiments can be realized.
  • the computer program includes computer program code, and the computer program code may be in the form of source code, object code, executable file or some intermediate form.
  • the computer-readable storage medium may at least include: any entity or device capable of carrying computer program codes to a photographing device/terminal device, a recording medium, a computer memory, a read-only memory (Read-Only Memory, ROM), a random access Memory (Random Access Memory, RAM), electrical carrier signal, telecommunication signal and software distribution medium.
  • a photographing device/terminal device a recording medium
  • a computer memory a read-only memory (Read-Only Memory, ROM), a random access Memory (Random Access Memory, RAM), electrical carrier signal, telecommunication signal and software distribution medium.
  • ROM read-only memory
  • RAM random access Memory
  • electrical carrier signal telecommunication signal and software distribution medium.
  • U disk mobile hard disk
  • magnetic disk or optical disk etc.
  • computer readable media may not be electrical carrier signals and telecommunication signals under legislation and patent practice.
  • references to "one embodiment” or “some embodiments” or the like in this application means that a particular feature, structure, or characteristic described in connection with the embodiment is included in one or more embodiments of the present application.
  • appearances of the phrases “in one embodiment,” “in some embodiments,” “in other embodiments,” “in other embodiments,” etc. in various places in this specification are not necessarily All refer to the same embodiment, but mean “one or more but not all embodiments” unless specifically stated otherwise.
  • the terms “including”, “comprising”, “having” and variations thereof mean “including but not limited to”, unless specifically stated otherwise.
  • first and second are used for description purposes only, and cannot be interpreted as indicating or implying relative importance or implicitly indicating the quantity of indicated technical features.
  • the features defined as “first” and “second” may explicitly or implicitly include at least one of these features.
  • connection and “connected” should be understood in a broad sense, for example, it can be mechanical connection or electrical connection; it can be direct connection or through An intermediate medium is indirectly connected, which can be the internal communication of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, those of ordinary skill in the art can understand the above terms in this application according to the specific situation. specific meaning.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

本申请提供一种图像颜色映射方法、装置、终端设备及存储介质，涉及图像处理技术领域。该方法包括：获取待处理图像，将待处理图像输入到已训练的颜色映射模型中进行处理，输出优化图像，颜色映射模型包括主网络和颜色条件网络；颜色条件网络包括依次连接的至少一个颜色条件模块和特征转换模块，至少一个颜色条件模块用于从待处理图像的低分辨率图像中提取全局颜色特征信息，特征转换模块用于将全局颜色特征信息转换为N组调节参数，N组调节参数分别用于调节主网络在将待处理图像转换为优化图像的过程中提取到的N个中间特征，N为大于或者等于1的整数。该方法可以改善目前图像颜色修饰任务中优化图像的质量较差的问题。

Description

一种图像颜色映射方法、装置、终端设备及存储介质 技术领域
本申请涉及图像处理技术领域，尤其涉及一种图像颜色映射方法、装置、终端设备及存储介质。
背景技术
在处理图像编辑、图像修饰与调色、图像上色、SDR视频转HDR视频等图像颜色修饰任务时，都需要对原始图像或者原始视频的颜色进行转换或者优化，优化后的图像或视频具备更高的对比度以及更丰富的色彩，也可以更好的反应真实环境中的视觉信息。
目前，基于神经网络的图像处理技术已得到广泛应用。在利用传统的神经网络实现上述任务的过程中，一般是利用神经网络提取图像的局部特征信息以实现原始图像与优化图像之间的转换。但是局部特征信息所能表示的信息量有限，基于局部特征信息的方法会使优化后的图像中产生较多的人工伪影，且色彩存在偏差，导致优化后的图像质量较差。
发明内容
本申请提供一种图像颜色映射方法、装置、终端设备及存储介质，可以改善目前图像颜色修饰任务中优化图像的质量较差的问题。
第一方面，本申请实施例提供了一种图像颜色映射方法，该方法包括：获取待处理图像，将待处理图像输入到已训练的颜色映射模型中进行处理，输出优化图像，颜色映射模型包括主网络和颜色条件网络；
颜色条件网络包括依次连接的至少一个颜色条件模块和特征转换模块，至少一个颜色条件模块用于从待处理图像的低分辨率图像中提取全局颜色特征信息，特征转换模块用于将全局颜色特征信息转换为N组调节参数，N组调节参数分别用于调节主网络在将待处理图像转换为优化图像的过程中提取到的N个中间特征，N为大于或者等于1的整数。
本申请提供的图像颜色映射方法，利用颜色条件网络中的至少一个颜色条件模块对输入的待处理图像的低分辨率图像进行全局颜色特征提取和压缩，与基于局部特征提取的方法相比可以避免给优化图像中引入人工伪影。通过特征转换模块将全局特征信息转换成调节参数以表示待处理图像的颜色先验信息，并利用调节参数调节主网络中提取到的中间特征，以根据不同待处理图像的颜色先验信息自适应的生成对应的优化图像，进而提高了优化图像的质量。
可选地，颜色条件模块包括依次连接的卷积层、池化层、第一激活函数和IN层。
可选地，特征转换模块包括Dropout层、卷积层、池化层和N个全连接层；Dropout层、卷积层和池化层依次连接，用于对全局颜色特征信息进行处理，得到条件向量；N个全连接层分别用于对条件向量进行特征转换，得到N组所述调节参数。
可选地，主网络包括N个GFM层，N组调节参数分别输入至N个GFM层，GFM层用于根据调节参数对输入至GFM层的中间特征进行调节。
可选地，主网络还包括N个卷积层和N-1个第二激活函数，N个GFM层分别连接N个卷积层的输出端，卷积层的卷积核大小为1×1。
基于上述可选的方式，将网络中卷积核的大小设置为1×1，可以有效的减少网络参数的数量，进而降低网络的运算复杂度。
可选地，待处理图像为从SDR视频中获取到的视频帧，SDR视频中的每一帧视频帧经过颜色映射模型优化处理后输出对应的优化图像，经过合帧后得到与SDR视频对应的HDR视频。
第二方面，本申请实施例提供了一种图像颜色映射装置，该装置包括：
获取单元，用于获取待处理图像；
处理单元，用于将待处理图像输入到已训练的颜色映射模型中进行优化处理，输出优化图像，颜色映射模型包括主网络和颜色条件网络，颜色条件网络包括依次连接的多个颜色条件模块和特征转换模块，多个颜色条件模块用于从待处理图像的低分辨率图像中提取全局颜色特征信息，特征转换模块用于将全局颜色特征信息转换为N组调节参数；N组调节参数分别用于调节主网络在将待处理图像转换为优化图像的过程中提取到的N个 中间特征，N为大于或者等于1的整数。
可选地，颜色条件模块包括依次连接的卷积层、池化层、第一激活函数和IN层。
第三方面，本申请实施例提供了一种终端设备，包括存储器、处理器以及存储在存储器中并可在处理器上运行的计算机程序，处理器执行计算机程序时实现如上述第一方面中任一项的方法。
第四方面，本申请实施例提供了一种计算机可读存储介质，计算机可读存储介质存储有计算机程序，计算机程序被处理器执行时实现如上述第一方面中任一项的方法。
第五方面，本申请实施例提供了一种计算机程序产品，当计算机程序产品在终端设备上运行时，使得终端设备执行上述第一方面中任一项的方法。
可以理解的是，上述第二方面至第五方面的有益效果可以参见上述第一方面和第一方面的各可能的实施方式所带来的有益效果的相关描述，在此不再赘述。
附图说明
为了更清楚地说明本申请实施例中的技术方案，下面将对实施例或现有技术描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本申请的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是本申请一实施例提供的一种颜色映射模型的网络结构图；
图2是本申请一实施例提供的HDR和SDR色域表示范围的示意图；
图3是本申请一实施例提供的一种HDR视频转SDR视频的流程示意图；
图4是本申请一实施例提供的一种图像颜色映射装置的结构示意图；
图5是本申请一实施例提供的一种终端设备的结构示意图。
具体实施方式
以下描述中，为了说明而不是为了限定，提出了诸如特定系统结构、 技术之类的具体细节，以便透彻理解本申请实施例。然而，本领域的技术人员应当清楚，在没有这些具体细节的其它实施例中也可以实现本申请。在其它情况中，省略对众所周知的系统、装置、电路以及方法的详细说明，以免不必要的细节妨碍本申请的描述。
为了提高图像颜色修饰任务中优化图像的质量，本申请实施例提供了一种图像颜色映射方法、装置、终端设备和存储介质。通过本申请提供的颜色映射模型对待处理图像进行优化处理，输出具有更高对比度和丰富色彩的优化图像。其中颜色映射模型包括主网络和颜色条件网络，颜色条件网络用于从待处理图像的低分辨率图像中提取调节参数，并利用调节参数调节主网络在将待处理图像转换成优化图像的过程中产生的中间特征，以根据不同的待处理图像的特性，自适应的调节待处理图像与优化图像之间的颜色映射，从而避免优化图像中产生伪影，提高优化图像的质量。
下面结合附图，对本申请的技术方案进行详细描述。下面通过参考附图描述的实施例是示例性的，旨在用于解释本申请，而不能理解为对本申请的限制。
结合图1对本申请提供的一种颜色映射模型对本申请提供的图像颜色映射方法进行示例性的介绍。该颜色映射模型可以部署在图像处理设备中。图像处理设备可以是智能手机、平板电脑、摄像机等移动终端，还可以是台式电脑、机器人、服务器等能够处理图像数据的设备。
在一种可能的实现方式中，本申请提供的颜色映射模型包括主网络和颜色条件网络。其中，颜色条件网络包括依次连接的至少一个颜色条件模块(Color Condition Block，CCB)和特征转换模块。至少一个颜色条件模块用于从待处理图像的低分辨率图像中提取全局颜色特征信息。特征转换模块用于将全局颜色特征信息转换为N组调节参数。N组调节参数分别用于调节主网络在将待处理图像转换为优化图像的过程中提取到的N个中间特征，N为大于或者等于1的整数。
示例性的，可以将待处理图像下采样一定的倍数(例如下采样4倍)后得到相应的低分辨率图像。假设将待处理图像下采样4倍后得到低分辨率图像，低分辨率图像与待处理图像的大小相同，只是待处理图像在单位面积内的像素数量为低分辨率图像在单位面积内的像素数量的4倍。
本申请提供的颜色映射模型通过至少一个颜色条件模块对输入的待处理图像的低分辨率图像进行全局颜色特征提取和压缩，与基于局部特征提取的方法相比可以避免给优化图像中引入人工伪影。通过特征转换模块将全局特征信息转换成调节参数以表示待处理图像的颜色先验信息，并利用调节参数调节主网络中提取到的待处理图像的中间特征，以根据不同待处理图像的颜色先验信息自适应的生成对应的优化图像，进而提高了优化图像的质量。
在一个实施例中，如图1所示，颜色条件模块包括依次连接的卷积层、池化层、第一激活函数和IN(Instance Normalization)层。该颜色条件模块可以对输入的低分辨率图像进行全局特征提取，与基于图像局部特征提取的方法相比，可以有效的表征待处理图像的全局特征信息，进而可以避免在优化图像中引入人工伪影。
特征转换模块包括Dropout层、卷积层、池化层和N个全连接层。其中，Dropout层、卷积层和池化层依次连接，用于对至少一个颜色条件模块提取到的全局颜色特征信息进行处理，得到条件向量。N个全连接层分别用于对条件向量进行特征转换，得到N组调节参数。需要说明的是，每个全连接层都分别对条件向量进行处理得到一组调节参数，最终全连接层的个数可与调节参数的组数相同。
示例性的，如图1所示的颜色映射模型中包括依次连接的4个颜色条件模块。在颜色条件模块与特征转换模块中，卷积层中卷积核的大小均为1×1，且池化层均采用平均池化。第一激活函数为非线性激活函数LeakyReLU。
在本申请实施例中，主网络包括N个全局特征调制(Global Feature Modulation，GFM)层，将N组调节参数输入至N个GFM层。GFM层可以根据调节参数对输入至GFM层的中间特征进行调节。
其中，主网络可以是任意能够实现颜色优化或者颜色转换的任务的神经网络模型。通过在主网络中插入N个GFM层，可以将本申请提供的颜色条件模块连接到主网络中，得到本申请提供的颜色映射模型。
在一个示例中，主网络可以是全卷积网络。即主网络包括N个卷积层及N-1个第二激活函数，N个GFM层分别连接N个卷积层的输出端。主 网络用于将待处理图像转换成优化图像，且在转换的过程中，N个卷积层可用于提取N个中间特征。每个卷积层中卷积核大小均为1×1。第二激活函数可以为非线性激活函数ReLU。
在本申请实施例提供的颜色映射模型中，卷积层的卷积核大小均为1×1，网络模型的参数较少，可以有效地降低计算的复杂度，提高运算效率，进而提高算法的实时性。
需要说明的是，颜色条件网络中全连接层的个数以及对应生成的调节参数的组数应基于主网络中卷积层的个数进行设计。例如，主网络中包括N个卷积层，则说明需要对N个卷积层生成的N个中间特征进行调节。因此，颜色条件网络中需要输出与N个中间特征对应的N组调节参数，主网络中需要有N个GFM层根据这N组调节参数对N个中间特征进行调节。
示例性的，如图1所示，假设N＝3，则主网络包括3个卷积(Conv)层、3个GFM层及2个第二激活函数(ReLU)层。具体的，主网络从输入到输出依次包括卷积层、GFM层、ReLU层、卷积层、GFM层、ReLU层、卷积层和GFM层。相应的，在颜色条件网络中，颜色条件模块包括依次连接的4个CCB层；特征转换模块可以包括依次连接的Dropout层、卷积(Conv)层、平均池化(Avgpool)层，以及分别与平均池化层输出的条件向量(Condition Vector)连接的3个全连接(FC)层。每个全连接层可以将该条件向量转换成相应的一组调节参数(γ,β)，颜色条件网络共输出3组调节参数(即调节参数1、调节参数2和调节参数3)。主网络中的每个GFM层根据对应的调节参数对输入到该GFM层的中间特征进行调节，可以表示为公式(1)：
GFM(x i)＝γ*x i+β        (1)
在公式(1)中，x i表示输入到GFM层的第i个中间特征；GFM(x i)表示GFM层根据调节参数(γ,β)对输入的中间特征x i的调节结果。
可以理解的是，包含不同场景的待处理图像与优化图像之间存在不同的颜色映射关系。本申请提供的颜色映射模型，通过颜色条件网络提取待处理图像的颜色特征信息作为先验信息，用于调节主网络中的中间特征，使得颜色映射模型可以基于不同待处理图像的颜色先验特征信息自适应地输出与待处理图像对应的优化图像，避免优化图像中出现人工伪影，从而 改善优化图像的质量。
本申请提供的颜色映射模型具备泛用性，可以应用于任何需要对待处理图像进行颜色优化或者颜色转换的任务。例如图像编辑、图像修饰与调色、图像上色、SDR(Standard Dynamic Range)视频转HDR(High Dynamic Range)视频等。
以SDR视频转HDR视频为例，由于受到拍摄设备的限制，现有的HDR视频资源较少，需要将已有的大量的SDR视频转换成HDR视频以满足用户的需求。图2为HDR和SDR色域表示范围的示意图。其中，BT.709和BT.2020都是ITU(国际电信联盟)发布的电视参数标准，DCI-P3是美国电影工业为数字电影院所制定的色域标准。从图2中可以看出，DCI-P3、BT.709和BT.2020中色域范围最大的是BT.2020，DCI-P3的色域范围次之，BT.709所表示的色域范围最小。目前，SDR视频采用的是BT.709色域，而HDR视频采用的是色域范围更为宽广的BT.2020色域或DCI-P3色域。就同一视频而言，无论HDR视频采用BT.2020色域还是DCI-P3色域，HDR视频可以比SDR视频展现出更高的对比度以及更加丰富的色彩。
现有技术中，常见的将SDR视频转为HDR视频的方法大多数是利用图像编码技术将SDR数据转换成HDR数据，使得HDR数据可以在HDR终端设备上播放。此外还需要通过超分辨率转换方法，将低分辨率的SDR视频内容转换成符合HDR视频标准的高分辨率HDR视频内容。现有的视频转换方法的计算成本较高，且转换后的HDR视频会出现人工伪影和色彩偏差，从而影响视频的质量。与现有方法相比，本申请提供的颜色映射模型可以根据不同场景下的SDR与HDR之间的颜色映射关系，自适应地将不同的SDR视频转换成对应的HDR视频，可以改善转换后的HDR视频中存在的人工伪影以及色彩偏差。
可以理解的是，针对不同的任务，可以通过设计对应的训练集和损失函数来训练初始的颜色映射模型，从而得到适用于不同任务的颜色映射模型。下面以SDR视频转HDR视频任务为例，对本申请提供的颜色映射模型的训练过程及应用进行示例性的说明。
步骤一，获取训练集。
针对SDR视频转HDR视频任务，训练集可以包括多个SDR视频帧样 本及与多个SDR视频帧样本一一对应的HDR视频帧样本。
具体的，首先获取SDR视频样本及其对应的HDR视频样本。示例性的，可以从公开的视频网站中获取SDR视频样本及对应的HDR视频样本。也可以对同一RAW数据格式的视频分别进行SDR和HDR处理，得到SDR视频样本及其对应的HDR视频样本。还可以分别利用SDR相机和HDR相机在同一场景下，分别拍摄对应的SDR视频样本和HDR视频样本。在获取到SDR视频样本及其对应的HDR视频样本之后，分别对SDR视频样本及其对应的HDR视频样本进行抽帧处理，得到多个SDR视频帧样本及与多个SDR视频帧样本一一对应的HDR视频帧样本。
步骤二，利用训练集和预设的损失函数对初始的颜色映射模型进行训练，得到已训练的颜色映射模型。
在搭建好初始的颜色映射模型后，将SDR视频帧样本输入到初始的颜色映射模型的主网络中。分别对多个SDR视频帧样本进行下采样处理，得到多个低分辨率图像，并将低分辨率图像输入到初始的颜色映射模型的颜色条件网络中得到调节参数，以调节初始的颜色映射模型预测的HDR视频帧。
预设的损失函数f用于描述初始的颜色映射模型预测的HDR视频帧
Figure PCTCN2021138080-appb-000001
与HDR视频帧样本H之间的L2损失，可以表示为公式(2)：
Figure PCTCN2021138080-appb-000002
基于训练集和上述预设的损失函数，可以通过梯度下降法对初始的颜色映射模型进行迭代训练，直到模型收敛，即可得到已训练的颜色映射模型。
图3为本申请实施例提供的一种HDR视频转SDR视频的方法流程示意图。从图3中可知，可以基于已训练的颜色映射模型将SDR视频转换成具有高对比度和更多色彩的HDR视频。首先对获取到的待处理的SDR视频进行抽帧处理，从SDR视频中获取到的视频帧即为图1中所示的输入到颜色映射模型的待处理图像。
针对SDR视频中的每一帧视频帧，将视频帧输入到已训练的颜色映射模型的主网络中。对视频帧进行4倍下采样处理，得到低分辨率图像，并将低分辨率图像输入到已训练的颜色映射模型的颜色条件网络中，得到多 个调节参数。主网络中的多个GFM层根据对应的调节参数对输入到GFM层的中间特征进行调节，最终输出与视频帧对应的优化图像。将SDR视频中的每一帧视频帧对应的优化图像进行合帧处理后，得到与SDR视频对应的HDR视频。
需要说明的是，本申请提供的颜色映射模型可以直接加入相机等终端设备的后期处理过程，以从软件的角度提升相机等终端设备拍摄的图像或视频的质量。本申请提供的颜色映射模型也可以作为一种图像/视频后期色彩增强手段，对已有的SDR或其他图像数据进行颜色优化。
本申请实施例还提供了一种图像颜色映射装置，该装置实施例与前述图像颜色映射方法实施例对应，为便于阅读，本装置实施例不再对前述方法实施例中的细节内容进行逐一赘述，但应当明确，本实施例中的装置能够对应实现前述方法实施例中的全部内容。
图4为本申请实施例提供的图像颜色映射装置的结构示意图，如图4所示，本实施例提供的图像颜色映射装置100包括获取单元101和处理单元102。
具体的，获取单元101用于获取待处理图像。处理单元102用于将待处理图像输入到已训练的颜色映射模型中进行优化处理，输出优化图像。颜色映射模型包括主网络和颜色条件网络，颜色条件网络包括依次连接的多个颜色条件模块和特征转换模块，多个颜色条件模块用于从待处理图像的低分辨率图像中提取全局颜色特征信息，特征转换模块用于将全局颜色特征信息转换为N组调节参数。N组调节参数分别用于调节主网络在将待处理图像转换为优化图像的过程中提取到的N个中间特征，N为大于或者等于1的整数。
可选地，颜色条件模块包括依次连接的卷积层、池化层、第一激活函数和IN层。
可选地，特征转换模块包括Dropout层、卷积层、池化层和N个全连接层。Dropout层、卷积层和池化层依次连接，用于对全局颜色特征信息进行处理，得到条件向量。N个全连接层分别用于对条件向量进行特征转换，得到N组所述调节参数。
可选地，主网络包括N个GFM层，N组调节参数分别输入至N个GFM 层，GFM层用于根据调节参数对输入至GFM层的中间特征进行调节。
可选地，主网络还包括N个卷积层和N-1个第二激活函数，N个GFM层分别连接N个卷积层的输出端，卷积层的卷积核大小为1×1。
可选地，待处理图像为从SDR视频中获取到的视频帧，SDR视频中的每一帧视频帧经过颜色映射模型优化处理后输出对应的优化图像，经过合帧后得到与SDR视频对应的HDR视频。
所属领域的技术人员可以清楚地了解到，为了描述的方便和简洁，仅以上述各功能单元、模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能单元、模块完成，即将所述装置的内部结构划分成不同的功能单元或模块，以完成以上描述的全部或者部分功能。实施例中的各功能单元、模块可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中，上述集成的单元既可以采用硬件的形式实现，也可以采用软件功能单元的形式实现。另外，各功能单元、模块的具体名称也只是为了便于相互区分，并不用于限制本申请的保护范围。上述系统中单元、模块的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
基于同一发明构思，本申请实施例还提供了一种终端设备。如图5所示，该实施例的终端设备200包括：处理器201、存储器202以及存储在存储器202中并可在处理器201上运行的计算机程序204。计算机程序204可被处理器201运行，生成指令203，处理器201可根据指令203实现上述各个图像颜色映射方法实施例中的步骤。或者，处理器201执行计算机程序204时实现上述各装置实施例中各模块/单元的功能，例如图4所示的单元101和单元102的功能。
示例性的，计算机程序204可以被分割成一个或多个模块/单元，一个或者多个模块/单元被存储在存储器202中，并由处理器201执行，以完成本申请。一个或多个模块/单元可以是能够完成特定功能的一系列计算机程序指令段，该指令段用于描述计算机程序204在终端设备200中的执行过程。
本领域技术人员可以理解，图5仅仅是终端设备200的示例，并不构成对终端设备200的限定，可以包括比图示更多或更少的部件，或者组合 某些部件，或者不同的部件，例如终端设备200还可以包括输入输出设备、网络接入设备、总线等。
处理器201可以是中央处理单元(Central Processing Unit，CPU)，还可以是其它通用处理器、数字信号处理器(Digital Signal Processor，DSP)、专用集成电路(Application Specific Integrated Circuit，ASIC)、现场可编程门阵列(Field-Programmable Gate Array，FPGA)或者其它可编程逻辑器件、分立门或者晶体管逻辑器件、分立硬件组件等。通用处理器可以是微处理器或者该处理器也可以是任何常规的处理器等。
存储器202可以是终端设备200的内部存储单元，例如终端设备200的硬盘或内存。存储器202也可以是终端设备200的外部存储设备，例如终端设备200上配备的插接式硬盘，智能存储卡(Smart Media Card，SMC)，安全数字(Secure Digital，SD)卡，闪存卡(Flash Card)等。进一步地，存储器202还可以既包括终端设备200的内部存储单元也包括外部存储设备。存储器202用于存储计算机程序以及终端设备200所需的其它程序和数据。存储器202还可以用于暂时地存储已经输出或者将要输出的数据。
本实施例提供的终端设备可以执行上述方法实施例，其实现原理与技术效果类似，此处不再赘述。
本申请实施例还提供一种计算机可读存储介质，其上存储有计算机程序，计算机程序被处理器执行时实现上述方法实施例所述的方法。
本申请实施例还提供一种计算机程序产品，当计算机程序产品在终端设备上运行时，使得终端设备执行时实现上述方法实施例所述的方法。
上述集成的单元如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本申请实现上述实施例方法中的全部或部分流程，可以通过计算机程序来指令相关的硬件来完成，所述的计算机程序可存储于一计算机可读存储介质中，该计算机程序在被处理器执行时，可实现上述各个方法实施例的步骤。其中，所述计算机程序包括计算机程序代码，所述计算机程序代码可以为源代码形式、对象代码形式、可执行文件或某些中间形式等。所述计算机可读存储介质至少可以包括：能够将计算机程序代码携带到拍照装置/终端设备的任何实体或装置、记录介质、计算机存储器、只读存储器 (Read-Only Memory，ROM)、随机存取存储器(Random Access Memory，RAM)、电载波信号、电信信号以及软件分发介质。例如U盘、移动硬盘、磁碟或者光盘等。在某些司法管辖区，根据立法和专利实践，计算机可读介质不可以是电载波信号和电信信号。
在本申请中描述的参考“一个实施例”或“一些实施例”等意味着在本申请的一个或多个实施例中包括结合该实施例描述的特定特征、结构或特点。由此，在本说明书中的不同之处出现的语句“在一个实施例中”、“在一些实施例中”、“在其他一些实施例中”、“在另外一些实施例中”等不是必然都参考相同的实施例，而是意味着“一个或多个但不是所有的实施例”，除非是以其他方式另外特别强调。术语“包括”、“包含”、“具有”及它们的变形都意味着“包括但不限于”，除非是以其他方式另外特别强调。
在本申请的描述中，需要理解的是，术语“第一”、“第二”仅用于描述目的，而不能理解为指示或暗示相对重要性或者隐含指明所指示的技术特征的数量。由此，限定有“第一”、“第二”的特征可以明示或者隐含地包括至少一个该特征。
此外，在本申请中，除非另有明确的规定和限定，术语“连接”、“相连”等应做广义理解，例如可以是机械连接，也可以是电连接；可以是直接连接，也可以通过中间媒介间接相连，可以是两个元件内部的连通或两个元件的相互作用关系，除非另有明确的限定、对于本领域的普通技术人员而言，可以根据具体情况理解上述术语在本申请中的具体含义。
以上各实施例仅用以说明本申请的技术方案，而非对其限制；尽管参照前述各实施例对本申请进行了详细的说明，本领域的普通技术人员应当理解：其依然可以对前述各实施例所记载的技术方案进行修改，或者对其中部分或者全部技术特征进行等同替换；而这些修改或者替换，并不使相应技术方案的本质脱离本申请各实施例技术方案的范围。

Claims (10)

  1. 一种图像颜色映射方法，其特征在于，包括：
    获取待处理图像，将所述待处理图像输入到已训练的颜色映射模型中进行优化处理，输出优化图像，所述颜色映射模型包括主网络和颜色条件网络；
    所述颜色条件网络包括依次连接的至少一个颜色条件模块和特征转换模块，所述至少一个颜色条件模块用于从所述待处理图像的低分辨率图像中提取全局颜色特征信息，所述特征转换模块用于将所述全局颜色特征信息转换为N组调节参数，N组所述调节参数分别用于调节所述主网络在将所述待处理图像转换为所述优化图像的过程中提取到的N个中间特征，所述N为大于或者等于1的整数。
  2. 根据权利要求1所述的方法，其特征在于，所述颜色条件模块包括依次连接的卷积层、池化层、第一激活函数和IN层。
  3. 根据权利要求1所述的方法，其特征在于，所述特征转换模块包括Dropout层、卷积层、池化层和N个全连接层；
    所述Dropout层、所述卷积层和所述池化层依次连接，用于对所述全局颜色特征信息进行处理，得到条件向量；
    所述N个全连接层分别用于对所述条件向量进行特征转换，得到N组所述调节参数。
  4. 根据权利要求1所述的方法，其特征在于，所述主网络包括N个GFM层，N组所述调节参数分别输入至N个所述GFM层，所述GFM层用于根据所述调节参数对输入至所述GFM层的所述中间特征进行调节。
  5. 根据权利要求4所述的方法，其特征在于，所述主网络还包括N个卷积层和N-1个第二激活函数，N个所述GFM层分别连接N个所述卷积层的输出端，所述卷积层的卷积核大小为1×1。
  6. 根据权利要求1至5任一项所述的方法，其特征在于，所述待处理图像为从SDR视频中获取到的视频帧，所述SDR视频中的每一帧视频帧经过所述颜色映射模型优化处理后输出对应的所述优化图像，经过合帧后得到与所述SDR视频对应的HDR视频。
  7. 一种图像颜色映射装置，其特征在于，包括：
    获取单元，用于获取待处理图像；
    处理单元，用于将所述待处理图像输入到已训练的颜色映射模型中进行优化处理，输出优化图像，所述颜色映射模型包括主网络和颜色条件网络，所述颜色条件网络包括依次连接的多个颜色条件模块和特征转换模块，所述多个颜色条件模块用于从所述待处理图像的低分辨率图像中提取全局颜色特征信息，所述特征转换模块用于将所述全局颜色特征信息转换为N组调节参数；N组所述调节参数分别用于调节所述主网络在将所述待处理图像转换为所述优化图像的过程中提取到的N个中间特征，所述N为大于或者等于1的整数。
  8. 根据权利要求7所述的装置，其特征在于，所述颜色条件模块包括依次连接的卷积层、池化层、第一激活函数和IN层。
  9. 一种终端设备，包括存储器、处理器以及存储在所述存储器中并可在所述处理器上运行的计算机程序，其特征在于，所述处理器执行所述计算机程序时实现如权利要求1至6任一项所述的方法。
  10. 一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，其特征在于，所述计算机程序被处理器执行时实现如权利要求1至6任一项所述的方法。
PCT/CN2021/138080 2021-08-02 2021-12-14 一种图像颜色映射方法、装置、终端设备及存储介质 WO2023010750A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110882146.3A CN113781318A (zh) 2021-08-02 2021-08-02 一种图像颜色映射方法、装置、终端设备及存储介质
CN202110882146.3 2021-08-02

Publications (1)

Publication Number Publication Date
WO2023010750A1 true WO2023010750A1 (zh) 2023-02-09

Family

ID=78836455

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/138080 WO2023010750A1 (zh) 2021-08-02 2021-12-14 一种图像颜色映射方法、装置、终端设备及存储介质

Country Status (2)

Country Link
CN (1) CN113781318A (zh)
WO (1) WO2023010750A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113781318A (zh) * 2021-08-02 2021-12-10 中国科学院深圳先进技术研究院 一种图像颜色映射方法、装置、终端设备及存储介质
CN113781322A (zh) * 2021-08-02 2021-12-10 中国科学院深圳先进技术研究院 一种色域映射方法、装置、终端设备及存储介质
CN116797446A (zh) * 2022-03-17 2023-09-22 中国移动通信有限公司研究院 一种数据处理方法、装置及设备

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110149507A (zh) * 2018-12-11 2019-08-20 腾讯科技（深圳）有限公司 视频处理方法、数据处理设备及存储介质
CN111598799A (zh) * 2020-04-30 2020-08-28 中国科学院深圳先进技术研究院 图像调色增强方法和图像调色增强神经网络训练方法
CN111861940A (zh) * 2020-07-31 2020-10-30 中国科学院深圳先进技术研究院 一种基于条件连续调节的图像调色增强方法
US20210166360A1 (en) * 2017-12-06 2021-06-03 Korea Advanced Institute Of Science And Technology Method and apparatus for inverse tone mapping
CN113096021A (zh) * 2019-12-23 2021-07-09 中国移动通信有限公司研究院 一种图像处理方法、装置、设备及存储介质
CN113781318A (zh) * 2021-08-02 2021-12-10 中国科学院深圳先进技术研究院 一种图像颜色映射方法、装置、终端设备及存储介质

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040130546A1 (en) * 2003-01-06 2004-07-08 Porikli Fatih M. Region growing with adaptive thresholds and distance function parameters
CN107154059B (zh) * 2017-06-26 2020-08-04 杭州当虹科技股份有限公司 一种高动态范围视频处理方法
EP3776474A1 (en) * 2018-04-09 2021-02-17 Dolby Laboratories Licensing Corporation Hdr image representations using neural network mappings
CN111274971A (zh) * 2020-01-21 2020-06-12 南京航空航天大学 一种基于颜色空间融合网络及空间变换网络的交通识别方法
CN111626954B (zh) * 2020-05-22 2022-05-06 兰州理工大学 壁画图像色彩还原方法、装置、存储介质及计算机设备
CN112991209B (zh) * 2021-03-12 2024-01-12 北京百度网讯科技有限公司 图像处理方法、装置、电子设备及存储介质

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210166360A1 (en) * 2017-12-06 2021-06-03 Korea Advanced Institute Of Science And Technology Method and apparatus for inverse tone mapping
CN110149507A (zh) * 2018-12-11 2019-08-20 腾讯科技（深圳）有限公司 视频处理方法、数据处理设备及存储介质
CN113096021A (zh) * 2019-12-23 2021-07-09 中国移动通信有限公司研究院 一种图像处理方法、装置、设备及存储介质
CN111598799A (zh) * 2020-04-30 2020-08-28 中国科学院深圳先进技术研究院 图像调色增强方法和图像调色增强神经网络训练方法
CN111861940A (zh) * 2020-07-31 2020-10-30 中国科学院深圳先进技术研究院 一种基于条件连续调节的图像调色增强方法
CN113781318A (zh) * 2021-08-02 2021-12-10 中国科学院深圳先进技术研究院 一种图像颜色映射方法、装置、终端设备及存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
CHEN, XIANGYU ET AL.: "A New Journey from SDRTV to HDRTV", 2021 IEEE/CVF INTERNATIONAL CONFERENCE ON COMPUTER VISION (ICCV), 17 October 2021 (2021-10-17), pages 4480 - 4489, XP034092233, DOI: 10.1109/ICCV48922.2021.00446 *

Also Published As

Publication number Publication date
CN113781318A (zh) 2021-12-10

Similar Documents

Publication Publication Date Title
WO2023010754A1 (zh) 一种图像处理方法、装置、终端设备及存储介质
WO2023010750A1 (zh) 一种图像颜色映射方法、装置、终端设备及存储介质
US11373275B2 (en) Method for generating high-resolution picture, computer device, and storage medium
US10861133B1 (en) Super-resolution video reconstruction method, device, apparatus and computer-readable storage medium
Chen et al. A new journey from SDRTV to HDRTV
WO2023010749A1 (zh) 一种hdr视频转换方法、装置、设备及计算机存储介质
JP7359521B2 (ja) 画像処理方法および装置
CN107302657B (zh) 适用于物联网的图像采集系统
US20120224788A1 (en) Merging Multiple Exposed Images in Transform Domain
CN108600783B (zh) 一种帧率调节方法、装置及终端设备
WO2020215180A1 (zh) 图像处理方法、装置和电子设备
KR20170115503A (ko) 화상을 역-톤 매핑하기 위한 장치 및 방법
WO2023010751A1 (zh) 图像高亮区域的信息补偿方法、装置、设备及存储介质
WO2021213336A1 (zh) 一种画质增强装置及相关方法
CN108665415A (zh) 基于深度学习的图像质量提升方法及其装置
WO2023010755A1 (zh) 一种hdr视频转换方法、装置、设备及计算机存储介质
CN110807735A (zh) 图像处理方法、装置、终端设备及计算机可读存储介质
CN114998122A (zh) 一种低照度图像增强方法
CN110717864A (zh) 一种图像增强方法、装置、终端设备及计算机可读介质
CN109102463B (zh) 一种超分辨率图像重建方法及装置
CN113066020A (zh) 图像处理方法及装置、计算机可读介质和电子设备
CN111953888B (zh) 暗光成像方法、装置、计算机可读存储介质及终端设备
WO2023010753A1 (zh) 一种色域映射方法、装置、终端设备及存储介质
WO2022052820A1 (zh) 一种数据处理方法、系统及装置
US20220303557A1 (en) Processing of Chroma-Subsampled Video Using Convolutional Neural Networks

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21952613

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE