CN115297254B - Portable high dynamic imaging fusion system under high radiation condition - Google Patents

Portable high dynamic imaging fusion system under high radiation condition Download PDF

Info

Publication number
CN115297254B
CN115297254B CN202210781478.7A CN202210781478A CN115297254B CN 115297254 B CN115297254 B CN 115297254B CN 202210781478 A CN202210781478 A CN 202210781478A CN 115297254 B CN115297254 B CN 115297254B
Authority
CN
China
Prior art keywords
image
network
camera
dynamic
photographing
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202210781478.7A
Other languages
Chinese (zh)
Other versions
CN115297254A (en
Inventor
罗晓燕
蔡开泉
司马翔怡
魏靖峰
张宇晨
鲁泽
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beihang University
Original Assignee
Beihang University
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beihang University filed Critical Beihang University
Priority to CN202210781478.7A priority Critical patent/CN115297254B/en
Publication of CN115297254A publication Critical patent/CN115297254A/en
Application granted granted Critical
Publication of CN115297254B publication Critical patent/CN115297254B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04MTELEPHONIC COMMUNICATION
    • H04M1/00Substation equipment, e.g. for use by subscribers
    • H04M1/72Mobile telephones; Cordless telephones, i.e. devices for establishing wireless links to base stations without route selection
    • H04M1/724User interfaces specially adapted for cordless or mobile telephones
    • H04M1/72403User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality
    • H04M1/7243User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages
    • H04M1/72439User interfaces specially adapted for cordless or mobile telephones with means for local support of applications that increase the functionality with interactive means for internal management of messages for image or video messaging

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Business, Economics & Management (AREA)
  • General Business, Economics & Management (AREA)
  • Human Computer Interaction (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)

Abstract

The invention designs a portable high-dynamic imaging fusion system under a high radiation condition, which is characterized in that a mobile terminal such as a mobile phone is utilized to acquire a multi-exposure image sequence, then high-dynamic recovery of images is realized under the single image calibration constraint, and meanwhile, the high-dynamic fusion among multiple images is realized. The basic process of the system comprises: the system regulates and controls the mobile phone to obtain a multi-exposure sequence, and stores the obtained 8-bit low-dynamic image through an image processing pipeline; respectively inputting the images into a dequantization network and a pixel exposure calibration network to obtain 32-bit images, and respectively inputting the obtained 32-bit low-dynamic images into a linearization network and a pixel exposure calibration network to linearize to obtain linear irradiance images; the linear irradiance image is respectively passed through a phantom network and a pixel exposure calibration network to reduce information loss caused by dynamic range clipping; and finally, optimizing the obtained image through an optimizing network to obtain a required high-dynamic image and outputting the high-dynamic image.

Description

Portable high dynamic imaging fusion system under high radiation condition
Technical Field
The invention relates to the technical field of image processing, and provides a portable image acquisition method and a high dynamic range image processing method based on deep learning.
Background
With the continuous development of technology, the quality requirements of images are increasingly improved, and the High Dynamic Range (HDR) display technology is rapidly developed. However, since some HDR devices are expensive, they are not widely used at present, but in the future, HDR display devices will be popular, and people will come to the colorful HDR image era.
Most current research is to fuse multiple exposed Low Dynamic Range (LDR) images using computer image processing techniques to acquire an HDR image. Generally, studies use images with intermediate positions as reference images, while other images are used to display details of the reference images. However, this approach requires that the object being photographed be static, and if there is motion of the object, the effect of the fusion may be "ghosting"; on the other hand, since the acquisition mode of the high dynamic imaging is a multi-input single-output mode, the single low dynamic range image cannot be processed. Therefore, how to design a single LDR method for generating HDR pictures is a very well studied problem.
Recently, with the development of deep learning technology, some researchers have obtained new algorithms with better performance in the process of converting LDR images into HDR images. Yuzhen Niu et al propose a new model based on a Generative Antagonism Network (GAN), called HDRGAN, which can be used to regenerate the missing details of the fused HDR image, eliminating artifacts of the reconstructed image. Seungjun Shin et al employ Convolutional Neural Networks (CNN), which include GoogleNet, resNet and VGG16, detect bright areas and increase their brightness according to the classification of bright and non-bright sources to obtain HDR images. Artificial Neural Networks (ANNs) are abstract models of the human brain. Because of the complexity of the human brain, it is difficult to build accurate mathematical models to describe the functioning of the brain. The pixel exposure calibration network may show different results because of the different activation functions of the different networks and the different numbers of neural cells.
In summary, there is no portable and reliable method for acquiring HDR images. The invention provides a portable image acquisition method and a high dynamic range image processing method based on deep learning.
Disclosure of Invention
The invention aims to solve the problem of conveniently generating a single image HDR image under high radiation condition.
In order to achieve the above purpose, the invention provides a system for performing single-image HDR imaging by pixel exposure and inversion camera pipeline based on deep learning under high radiation condition, and integrates the system into software, the software is provided with a user UI interface, and a user can adjust parameters such as exposure times, duration, focal length and the like on the UI interface to realize the regulation and control of a mobile phone camera on a mobile phone and other terminals and obtain a high dynamic range image through system image processing. And the system is mainly divided into two parts, namely image acquisition and image processing.
The image acquisition section includes four steps:
and 1, displaying a shooting object in real time by using a screen for previewing, taking the shooting object to a camera manager for managing camera equipment through initialization of system service, obtaining the attribute of a rear camera, and calling an open camera to acquire the camera equipment object by calling an open camera method in a callback function.
And 2, utilizing the obtained camera equipment representing the rear camera to establish preview on one hand, after setting preview parameters, displaying a preview result on a screen, and on the other hand, establishing a camera photographing pipeline and a photographing parameter setter. And setting parameters through a parameter setter to set the photographing mode and obtaining a photographing request.
And 3, monitoring button actions, stopping previewing after obtaining a photographing request, performing single photographing action or continuous framing in a camera photographing pipeline to obtain an 8-bit LDR image, and storing picture data into the mobile phone through the pipeline.
And 4, continuously previewing after changing parameters through screen interaction, and waiting for the next shooting.
The image processing section includes four steps:
and 5, the system retrieves the stored image and dequantizes the 8-bit LDR image, namely, the 8-bit LDR image enters a dequantizing network and a parallel pixel exposure calibration network respectively for dequantization, and a 32-bit LDR image is obtained.
And 6, linearizing the 32-bit LDR image obtained in the step 5, namely respectively entering a linearization network and linearizing a pixel exposure calibration network in parallel with the linearization network to obtain a predicted linear irradiance image.
And 7, carrying out illusion on the linear irradiance image obtained in the step 6, namely respectively entering an illusion network and a pixel exposure calibration network parallel to the illusion network so as to prevent information loss caused by dynamic range clipping.
And 8, further optimizing the reconstructed HDR image obtained in the step 7, namely entering an optimizing network for optimization, and finally obtaining an output HDR image.
The invention provides a portable high-dynamic imaging fusion system under a high radiation condition. The method is firstly suitable for high-altitude scenes such as a plateau airport and the like, is integrated into mobile phone software, can directly utilize the mobile phone camera to acquire an 8-bit LDR image, and then obtains an HDR image with a good effect through a series of processing on the LDR image.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of an embodiment of a portable high dynamic imaging fusion system under high radiation conditions.
Fig. 2 is a diagram showing a user UI interface of software in the present invention.
FIG. 3 is an overall flow chart of image acquisition in the present invention.
Fig. 4 is an overall flowchart of image processing in the present invention.
Fig. 5 is an original acquired LDR image.
Fig. 6 is an image after dequantization processing.
Fig. 7 is an image after linearization processing.
Fig. 8 is an image after the illusion processing.
Fig. 9 is an HDR image processed using the present invention.
Fig. 10 is a schematic diagram of the structure of the dequantization network, the pixel exposure calibration network, and the optimization network in the image processing section.
Fig. 11 is a schematic diagram of the structure of the linearization network in the image processing section.
Fig. 12 is a schematic diagram of the structure of the illusion network in the image processing section.
Detailed Description
The invention will be described more fully hereinafter with reference to the accompanying drawings, in which embodiments of the invention are shown. It will be apparent that the described examples are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any creative effort, are within the protection scope of the invention.
The terms "comprising," "having," and any variations thereof, in the description and claims of the invention and in the foregoing drawings, are intended to cover non-exclusive inclusions. For example, a process, method, system, article, or apparatus that comprises a list of steps or elements is not limited to only those listed steps or elements but may alternatively include steps or elements not listed or may include other steps or elements inherent to such process, method, article, or apparatus.
Reference herein to "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least one embodiment of the present application. The appearances of such phrases in various places in the specification are not necessarily all referring to the same embodiment, nor are separate or alternative embodiments mutually exclusive of other embodiments. Those skilled in the art will recognize, either explicitly or implicitly, that the embodiments described herein may be combined with other embodiments.
Embodiments of the present invention are described in detail below.
The overall flow chart of the present invention is shown in fig. 1, and in this embodiment, the steps are as follows:
step 1, an application is opened, and the software displays a user UI interface, as shown in FIG. 2.
The image processing part of the system is based on Google's camera function, wherein the camera provides a camera manager for managing the android system camera, and the initialization is completed by obtaining system services, and meanwhile, the camera information parameters and the numbers of the equipment are obtained.
An object is abstracted from the camera, and the camera is directly connected with a camera in system hardware, monitors the callback state of the camera through a camera callback function, and is responsible for managing a camera capturing channel and receiving a photographing request.
A photographing request session created by the camera equipment is a photographing pipeline of the camera, and is mainly used for processing a photographing request and representing control of camera actions, such as camera preview and photographing, data acquisition and the like. And two interfaces of the callback function and the photographing callback function are provided for monitoring the photographing process while photographing.
The camera photographing represents a photographing request, a photographing request creator is used for creating the photographing request, various parameters of the photographing request including a sensor, a lens, a frame rate, a size, a memory, an exposure time, an exposure mode, a focusing mode, a control algorithm output format, and the like are set, and then transferred to a corresponding photographing pipeline for setting.
And 2, creating a thread of a photographing flow, displaying the currently photographed object for previewing by using a screen in real time, taking a camera manager for managing camera equipment through system service, simultaneously obtaining the attribute of a rear camera, and obtaining the camera equipment object by opening the camera and calling a camera opening method in a callback function by the system.
And 3, the system utilizes the obtained camera equipment representing the rear camera to establish preview on one hand, and after setting preview parameters, the preview result is displayed on a screen, and on the other hand, a camera photographing pipeline and a photographing parameter setter are established.
And 4, setting parameters such as exposure times, focal length and the like on a UI interface by a user through previewing, and obtaining a photographing request by the system. And the user sends out a shooting request after setting the parameters.
And 5, the system performs monitoring button action, and stops previewing after obtaining a photographing request, performs single photographing action or continuous framing in a photographing request pipeline by using the obtained photographing request, stores original image data into the mobile phone through the pipeline, and waits for a next photographing request.
Step 6, the system retrieves the original image and dequantizes the image to obtain a dequantized image, as shown in fig. 6;
the dequantization network adopts a 6-level U-shaped network architecture. Each stage consists of two convolution layers and a leaky linear rectification function (α=0.1), as shown in fig. 10. The output of the last layer is normalized to [ -1.0,1.0] with the Tanh layer, and finally the output of the dequantization network is added to the input LDR image, thereby generating a dequantized LDR image.
The linear rectification function with leakage involved in dequantization is:
step 7, carrying out linearization processing on the 32-bit LDR image obtained in the step 2, wherein a linearization network is shown in fig. 11, and a linear irradiance image is shown in fig. 7;
wherein the CRF function has the following properties: (1) the CRF function is strictly monotonically increasing. (2) F (0) = 0,F (1) =1. Due to the definition of the inverse function, function g=f -1 There are also these two properties. Linearization netThe complex can perform a predictive inverse conditional random field (g=f -1 ) To estimate CRF.
The linearization network takes the nonlinear LDR image, the edge map and the histogram map as inputs, predicts principal component analysis coefficients (PCA coefficients) to reconstruct the inverse CRF, then performs monotonically increasing constraint, and finally obtains a predicted linear irradiance image.
Step 8, performing illusion processing on the predicted linear irradiance image obtained in the step 3 to obtain a reconstructed HDR image, as shown in fig. 8;
among them we have adopted an encoder-decoder architecture with skipped connections as our phantom network. The structure is shown in figure 12. The reconstructed HDR image is composed of h=l lin +αC -1 Formed, wherein L lin As a result of linearizing the networkγ=0.95。
Since the missing values in the overexposed region should always be larger than the existing pixel values, a linear rectification function is added at the end of the network for predicting positive residuals, while since the overexposed mask is a soft mask that allows a e (0, -1), the phantom network can smoothly blend the residuals with the pixel values present around the overexposed region.
Step 9, improving and optimizing the reconstructed HDR image;
the formula involved in the optimization network for optimizing the reconstructed HDR image is the same as the formula involved in the dequantization process.
The optimized image is output and displayed on the GUI interface, and the image is the final required HDR image, as shown in fig. 9.
According to the principle of camera pipeline and pixel imaging and the deep learning principle, the invention designs an inversion camera pipeline (namely, a dequantization network, a linearization network and a phantom network in the figure 1) and a pixel exposure amplifier (namely, a pixel exposure calibration network in the figure 1). Then, by trying to combine the inverse camera pipeline and the pixel exposure booster in parallel, it is successfully achieved that only a cell phone is used to obtain a better HDR image. The method effectively reduces the cost of HDR image acquisition, and meanwhile, the obtained HDR image has better effect, higher running efficiency and remarkable convenience and practicability.
The foregoing has described in detail embodiments of the present invention, the principles, i.e., embodiments, of the present application have been described herein with reference to specific examples, the description of the foregoing embodiments being merely intended to facilitate an understanding of the methods of the present application and their core ideas; also, it is obvious to those skilled in the art that all the inventions utilizing the inventive concept are protected insofar as the variations are within the spirit and scope of the present invention as defined and defined by the appended claims.

Claims (1)

1. A portable high-dynamic imaging fusion system under a high radiation condition is characterized in that a set of portable image acquisition system is designed based on mobile phone APP development and Camera2 application interfaces, a screen is utilized to display a shooting object in real time for previewing, meanwhile, the attribute of a rear Camera is obtained, the exposure times, exposure time and multiple attributes of focal length can be adjusted, and single or multiple shooting is selected; in the adjusting process, displaying the preview result on a screen in real time, monitoring the action of a button, stopping previewing and shooting after a shooting request is obtained, and obtaining an 8-bit low-dynamic image by using the obtained single shooting action or continuous framing in a camera shooting pipeline;
the system is divided into two parts of image acquisition and image processing;
the image acquisition section includes four steps:
step 1, displaying a shooting object in real time by using a screen for previewing, taking the shooting object to a camera manager for managing camera equipment through initialization of system service, simultaneously obtaining the attribute of a rear camera, and calling an open camera to acquire the camera equipment object by calling an open camera method in a callback function;
step 2, utilizing the obtained camera equipment representing the rear camera to establish preview on one hand, after setting preview parameters, displaying preview results on a screen, and on the other hand, establishing a camera photographing pipeline and a photographing parameter setter; setting parameters through a parameter setter to set a photographing mode and obtaining a photographing request;
step 3, monitoring button actions, stopping previewing after obtaining a photographing request, performing single photographing actions or continuous framing in a camera photographing pipeline by using the obtained LDR images with 8 bits, and storing picture data into a mobile phone through the pipeline;
step 4, continuously previewing after changing parameters through screen interaction, and waiting for the next shooting;
the image processing section includes four steps:
step 5, the system retrieves the stored image and dequantizes the 8-bit LDR image, namely, the 8-bit LDR image enters a dequantizing network and a parallel pixel exposure calibration network respectively for dequantization, and a 32-bit LDR image is obtained;
step 6, linearizing the 32-bit LDR image obtained in the step 5, namely respectively entering a linearization network and linearizing a pixel exposure calibration network which is parallel to the linearization network to obtain a predicted linear irradiance image;
step 7, carrying out illusion on the linear irradiance image obtained in the step 6, namely respectively entering an illusion network and a pixel exposure calibration network parallel to the illusion network so as to prevent information loss caused by dynamic range clipping; the encoder-decoder architecture with skipped connections acts as a phantom network;
and 8, further optimizing the reconstructed HDR image obtained in the step 7, namely entering an optimizing network for optimization, and finally obtaining an output HDR image.
CN202210781478.7A 2022-07-04 2022-07-04 Portable high dynamic imaging fusion system under high radiation condition Active CN115297254B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202210781478.7A CN115297254B (en) 2022-07-04 2022-07-04 Portable high dynamic imaging fusion system under high radiation condition

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202210781478.7A CN115297254B (en) 2022-07-04 2022-07-04 Portable high dynamic imaging fusion system under high radiation condition

Publications (2)

Publication Number Publication Date
CN115297254A CN115297254A (en) 2022-11-04
CN115297254B true CN115297254B (en) 2024-03-29

Family

ID=83822690

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202210781478.7A Active CN115297254B (en) 2022-07-04 2022-07-04 Portable high dynamic imaging fusion system under high radiation condition

Country Status (1)

Country Link
CN (1) CN115297254B (en)

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105227858A (en) * 2015-10-30 2016-01-06 维沃移动通信有限公司 A kind of image processing method and mobile terminal
KR20180084686A (en) * 2017-01-16 2018-07-25 전선곤 Apparatus for Acquiring Multiple Exposure Images Using Optical Shutter
WO2019089975A1 (en) * 2017-11-03 2019-05-09 Qualcomm Incorporated Systems and methods for high-dynamic range imaging
CN111292264A (en) * 2020-01-21 2020-06-16 武汉大学 Image high dynamic range reconstruction method based on deep learning
CN111372006A (en) * 2020-03-03 2020-07-03 山东大学 High dynamic range imaging method and system for mobile terminal
CN112435306A (en) * 2020-11-20 2021-03-02 上海北昂医药科技股份有限公司 G banding chromosome HDR image reconstruction method
CN113382169A (en) * 2021-06-18 2021-09-10 荣耀终端有限公司 Photographing method and electronic equipment
WO2022096104A1 (en) * 2020-11-05 2022-05-12 Huawei Technologies Co., Ltd. Permutation invariant high dynamic range imaging

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111709896B (en) * 2020-06-18 2023-04-07 三星电子(中国)研发中心 Method and equipment for mapping LDR video into HDR video

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105227858A (en) * 2015-10-30 2016-01-06 维沃移动通信有限公司 A kind of image processing method and mobile terminal
KR20180084686A (en) * 2017-01-16 2018-07-25 전선곤 Apparatus for Acquiring Multiple Exposure Images Using Optical Shutter
WO2019089975A1 (en) * 2017-11-03 2019-05-09 Qualcomm Incorporated Systems and methods for high-dynamic range imaging
CN111292264A (en) * 2020-01-21 2020-06-16 武汉大学 Image high dynamic range reconstruction method based on deep learning
CN111372006A (en) * 2020-03-03 2020-07-03 山东大学 High dynamic range imaging method and system for mobile terminal
WO2022096104A1 (en) * 2020-11-05 2022-05-12 Huawei Technologies Co., Ltd. Permutation invariant high dynamic range imaging
CN112435306A (en) * 2020-11-20 2021-03-02 上海北昂医药科技股份有限公司 G banding chromosome HDR image reconstruction method
CN113382169A (en) * 2021-06-18 2021-09-10 荣耀终端有限公司 Photographing method and electronic equipment

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
Bayer patterned high dynamic range image reconstruction using adaptive weighting function;Hee Kang 等;《EURASIP Journal on advancesin signal processing 》;20141231;全文 *
Single-Image HDR Reconstruction by Learning to Reverse the Camera Pipeline;Yu-Lun Liu et al;《arXiv:2004.01179v1》;全文 *
一种基于双通道CMOS相机的低照度动态场景HDR融合方法;贺理;陈果;郭宏;金伟其;;红外技术(第04期);全文 *
基于深度学习的高动态范围图像生成技术研究;陈红;《CNKI硕士电子期刊》;正文第10-62页 *

Also Published As

Publication number Publication date
CN115297254A (en) 2022-11-04

Similar Documents

Publication Publication Date Title
Ni et al. Towards unsupervised deep image enhancement with generative adversarial network
CN111641778B (en) Shooting method, device and equipment
WO2019085705A1 (en) Method and device for obtaining exposure compensation value of high-dynamic-range image
WO2020057198A1 (en) Image processing method and device, electronic device and storage medium
US11070742B2 (en) Optimized exposure temporal smoothing for time-lapse mode
CN111311532B (en) Image processing method and device, electronic device and storage medium
CN111107265B (en) Image processing method and device, computer readable medium and electronic equipment
CN111985281A (en) Image generation model generation method and device and image generation method and device
CN112258380A (en) Image processing method, device, equipment and storage medium
CN111986106A (en) High dynamic image reconstruction method based on neural network
CN110956063A (en) Image processing method, device, equipment and storage medium
CN110958363B (en) Image processing method and device, computer readable medium and electronic device
CN111127367A (en) Method, device and system for processing face image
CN115297254B (en) Portable high dynamic imaging fusion system under high radiation condition
WO2016123850A1 (en) Photographing control method for terminal, and terminal
CN112184586A (en) Method and system for rapidly blurring monocular visual image background based on depth perception
CN116416323A (en) Image processing method and device, electronic equipment and storage medium
CN116208733A (en) Video conference interaction method and device
CN115552415A (en) Image processing method, image processing device, electronic equipment and storage medium
CN101594459A (en) Select image system and method thereof
CN111026893A (en) Intelligent terminal, image processing method and computer-readable storage medium
EP4083903B1 (en) Method and apparatus for processing image, and storage medium
WO2022105757A1 (en) Image processing method and apparatus
CN114827477B (en) Method, device, electronic equipment and medium for time-lapse photography
CN117478824B (en) Conference video generation method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant