TW202139140A - Image reconstruction method and apparatus, electronic device and storage medium - Google Patents

Abstract

The present disclosure relates to an image reconstruction method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring event information of a target scene, the event information being used to represent changes in brightness of the target scene within a first brightness range; performing feature extraction on the event information to obtain a first event feature of the target scene; and performing image reconstruction on the first event feature to obtain a reconstructed image of the target scene, wherein the brightness of the reconstructed image is within a second brightness range, and the second brightness range is higher than the first brightness range. The embodiments of the present disclosure may improve the effect of image reconstruction.

Description

Image reconstruction method, image reconstruction device, electronic equipment and computer readable storage medium

The present disclosure relates to the field of computer technology, and in particular to an image reconstruction method, an image reconstruction device, electronic equipment, and computer-readable storage media. This application claims the priority of a Chinese patent application filed with the Chinese Patent Office, the application number is 202010243153.4, and the invention title is "Image reconstruction method and device, electronic equipment and storage media" on March 31, 2020, the entire content of which is incorporated by reference Incorporated in this application.

Traditional image acquisition equipment can acquire images that conform to people's observation habits, such as RGB images or intensity images. However, due to the limitation of its own low dynamic range, the image acquisition device will be underexposed under low light conditions and cannot generate high-quality clear images.

The present disclosure proposes a technical solution for image reconstruction.

According to one aspect of the present disclosure, there is provided an image reconstruction method, including: acquiring event information of a target scene, the event information being used to represent the brightness change of the target scene within a first brightness range; Perform feature extraction to obtain the first event feature of the target scene; perform image reconstruction on the first event feature to obtain a reconstructed image of the target scene, and the brightness of the reconstructed image is within the second brightness range , The second brightness range is higher than the first brightness range.

In a possible implementation manner, performing image reconstruction on the first event feature to obtain a reconstructed image of the target scene includes: performing image reconstruction on the first event feature according to the first noise information and the first event feature The details of an event feature are enhanced to obtain a second event feature; the first event feature and the second event feature are fused to obtain a fusion feature; the fusion feature is image-reconstructed to obtain a reconstruction of the target scene image.

In a possible implementation manner, the image reconstruction method is implemented by an image processing network, the image processing network includes a first feature extraction network and an image reconstruction network, the first feature extraction network The image reconstruction network is used to perform feature extraction on the event information, the image reconstruction network is used to perform image reconstruction on the first event feature, and the image reconstruction method further includes: training the institute according to a preset training set. In the image processing network, the training set includes first sample event information of a plurality of first sample scenes, second sample event information of a plurality of second sample scenes and sample scene images; wherein, the first sample scene The sample event information is obtained in the third brightness range, the second sample event information is obtained in the fourth brightness range, and the sample scene image is obtained in the fourth brightness range, The fourth brightness range is higher than the third brightness range.

In a possible implementation, the image processing network further includes a discrimination network, and the training of the image processing network according to a preset training set includes: The sample event information and the second sample event information of the second sample scene are respectively input into the first feature extraction network to obtain the first sample event feature and the second sample event feature; and the first sample The event feature and the second sample event feature are respectively input to the identification network to obtain a first identification result and a second identification result; according to the first identification result and the second identification result, the image is trained against Deal with the network.

In a possible implementation, the training the image processing network according to a preset training set further includes: inputting the second sample event feature into the image reconstruction network to obtain the second A first reconstructed image of a sample scene; training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training of the image processing network according to a preset training set further includes: combining the second sample event feature And the third noise information is input into the detail enhancement network to obtain the fourth sample event feature; the second sample event feature is fused with the fourth sample event feature to obtain the second sample fusion feature; The two-sample fusion feature is input into the image reconstruction network to obtain the third reconstructed image of the second sample scene; according to the first reconstructed image of the second sample scene, the third reconstructed image and all The sample scene image is used to train the image processing network.

In a possible implementation, the image processing network further includes a second feature extraction network, and the training of the image processing network according to a preset training set further includes: combining the second sample The second sample event information and the second noise information of the scene are input into the second feature extraction network to obtain the third sample event feature; the second sample event feature is fused with the third sample event feature to obtain the first Sample fusion features; input the first sample fusion features into the identification network to obtain a third identification result; according to the first identification result and the third identification result, oppose the training of the image processing network.

In a possible implementation, the training of the image processing network according to a preset training set further includes: inputting the first sample fusion feature into the image reconstruction network to obtain the first A second reconstructed image of a two-sample scene; training the image processing network according to the second reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training of the image processing network according to a preset training set further includes: fusing the first sample The feature and the fourth noise information are input into the detail enhancement network to obtain the fifth sample event feature; the first sample fusion feature and the fifth sample event feature are fused to obtain the third sample fusion feature; The third sample fusion feature is input into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene; according to the second reconstructed image and the fourth reconstructed image of the second sample scene And the sample scene image to train the image processing network.

In a possible implementation, the training the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image includes: According to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image, determine the overall loss of the image processing network; according to the overall loss, determine the Gradient information of the image processing network; according to the gradient information, adjust the network of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network Path parameters, wherein the gradient information of the detail enhancement network is not transmitted to the second feature extraction network.

According to an aspect of the present disclosure, there is provided an image reconstruction device, including:

The event acquisition module is used to acquire event information of the target scene, and the event information is used to indicate the brightness change of the target scene in the first brightness range; the feature extraction module is used to perform feature extraction on the event information , The first event feature of the target scene is obtained; the image reconstruction module is used to perform image reconstruction on the first event feature to obtain a reconstructed image of the target scene, and the brightness of the reconstructed image is at In the second brightness range, the second brightness range is higher than the first brightness range.

In a possible implementation, the image reconstruction module includes: a detail enhancement sub-module for performing detail enhancement on the first event feature according to the first noise information and the first event feature, Obtain a second event feature; a fusion sub-module for fusing the first event feature with the second event feature to obtain a fusion feature; a reconstruction sub-module for performing image reconstruction on the fusion feature, Obtain a reconstructed image of the target scene.

In a possible implementation manner, the image reconstruction device is implemented by an image processing network, the image processing network includes a first feature extraction network and an image reconstruction network, the first feature extraction network The path is used for feature extraction of the event information, the image reconstruction network is used for image reconstruction of the first event feature, and the image reconstruction device further includes:

The training module is used to train the image processing network according to a preset training set. The training set includes first sample event information of a plurality of first sample scenes, and second sample scenes of a plurality of second sample scenes. Sample event information and sample scene images; wherein the first sample event information is acquired within a third brightness range, the second sample event information is acquired within a fourth brightness range, and the sample scene The image is acquired in the fourth brightness range, and the fourth brightness range is higher than the third brightness range.

In a possible implementation, the image processing network further includes an identification network, and the training module includes: a first extraction sub-module configured to convert the first sample of the first sample scene The event information and the second sample event information of the second sample scene are respectively input into the first feature extraction network to obtain the first sample event feature and the second sample event feature; the first discrimination sub-module is used to combine The first sample event feature and the second sample event feature are respectively input to the identification network to obtain a first identification result and a second identification result; the first confrontation training sub-module is used to obtain the first identification result according to the first The identification result and the second identification result are used against training the image processing network.

In a possible implementation, the training module further includes: a first reconstruction sub-module, configured to input the second sample event feature into the image reconstruction network to obtain the second sample scene A first reconstructed image; a first training sub-module for training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training module further includes: a first enhancement sub-module for combining the second sample event features and the third The noise information is input into the detail enhancement network to obtain the fourth sample event feature; the first fusion sub-module is used to fuse the second sample event feature with the fourth sample event feature to obtain a second sample fusion Features; a second reconstruction sub-module, used to input the second sample fusion feature into the image reconstruction network to obtain a third reconstructed image of the second sample scene; a second training sub-module, using Training the image processing network according to the first reconstructed image of the second sample scene, the third reconstructed image, and the sample scene image.

In a possible implementation manner, the image processing network further includes a second feature extraction network, and the training module further includes: a second extraction sub-module configured to convert the second feature extraction network of the second sample scene Two sample event information and second noise information are input into the second feature extraction network to obtain a third sample event feature; a second fusion sub-module is used to combine the second sample event feature with the third sample Event feature fusion to obtain a first sample fusion feature; a second discrimination sub-module for inputting the first sample fusion feature into the discrimination network to obtain a third discrimination result; a second confrontation training sub-module , Used for training the image processing network against the first identification result and the third identification result.

In a possible implementation, the training module further includes: a third reconstruction sub-module, configured to input the first sample fusion feature into the image reconstruction network to obtain the second sample scene The second reconstructed image; the third training sub-module is used to train the image processing network according to the second reconstructed image of the second sample scene and the sample scene image.

In a possible implementation manner, the image processing network further includes a detail enhancement network, and the training module further includes: a second enhancement sub-module for fusing the features of the first sample with the first sample Four noise information is input into the detail enhancement network to obtain the fifth sample event feature; the third fusion sub-module is used to fuse the first sample fusion feature with the fifth sample event feature to obtain the third Sample fusion features; a fourth reconstruction sub-module, used to input the third sample fusion features into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene; a fourth training sub-module , For training the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image.

In a possible implementation manner, the fourth training sub-module is used to determine the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image. The overall loss of the image processing network; according to the overall loss, the gradient information of the image processing network is determined; according to the gradient information, the first feature extraction network and the second feature extraction are adjusted The network, the detail enhancement network and the network parameters of the image reconstruction network, wherein the gradient information of the detail enhancement network is not transmitted to the second feature extraction network.

According to an aspect of the present disclosure, there is provided an electronic device including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the instructions stored in the memory to execute The above-mentioned image reconstruction method.

According to one aspect of the present disclosure, there is provided a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions are executed by a processor to realize the above-mentioned image reconstruction method.

According to an aspect of the present disclosure, there is provided a computer program including computer-readable code, and when the computer-readable code is run in an electronic device, a processor in the electronic device executes the above-mentioned image reconstruction method.

In the embodiments of the present disclosure, it is possible to obtain event information of the target scene in the lower first brightness range; perform feature extraction on the event information to obtain the event feature; perform image reconstruction on the event feature to obtain the target scene in the higher The reconstructed image in the second brightness range can reconstruct a high-quality image under normal lighting conditions through events under dark light conditions, which improves the effect of image reconstruction.

It should be understood that the above general description and the following detailed description are only exemplary and explanatory, rather than limiting the present disclosure. According to the following detailed description of exemplary embodiments with reference to the accompanying drawings, other features and aspects of the present disclosure will become clear.

Various exemplary embodiments, features, and aspects of the present disclosure will be described in detail below with reference to the drawings. The same drawing symbols in the drawings indicate elements with the same or similar functions. Although various aspects of the embodiments are shown in the drawings, the drawings need not be drawn to scale unless otherwise noted.

The dedicated word "exemplary" here means "serving as an example, embodiment, or illustration." Any embodiment described herein as "exemplary" need not be construed as being superior or better than other embodiments.

The term "and/or" in this article is only an association relationship describing related objects, which means that there can be three relationships. For example, A and/or B can mean: A alone exists, A and B exist at the same time, and B exists alone. three conditions. In addition, the term "at least one" herein means any one or any combination of at least two of the multiple, for example, including at least one of A, B, and C, and may mean including those made from A, B, and C Any one or more elements selected in the set.

In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. Those skilled in the art should understand that the present disclosure can also be implemented without certain specific details. In some instances, the methods, means, elements, and circuits well known to those skilled in the art have not been described in detail, so as to highlight the gist of the present disclosure.

In the fields of image shooting, image processing, face recognition, security, etc., it is usually necessary to collect images through image acquisition equipment (such as intensity cameras or cameras, etc.). Images captured by image capture devices under dark conditions (such as night, low light or other dark environments) are prone to underexposure and poor image quality. In this case, a poor quality image can be reconstructed to obtain a high-quality image under normal lighting conditions.

Fig. 1 shows a flowchart of an image reconstruction method according to an embodiment of the present disclosure. As shown in Fig. 1, the image reconstruction method includes:

In step S11, obtain event information of the target scene, where the event information is used to indicate the brightness change of the target scene within a first brightness range;

In step S12, feature extraction is performed on the event information to obtain the first event feature of the target scene;

In step S13, image reconstruction is performed on the first event feature to obtain a reconstructed image of the target scene, the brightness of the reconstructed image is within a second brightness range, and the second brightness range is higher than all. The first brightness range.

In a possible implementation, the image reconstruction method may be executed by electronic equipment such as a terminal device or a server, and the terminal device may be a user equipment (User Equipment, UE), a mobile device, a user terminal, a terminal, or a cellular device. Mobile phones, indoor wireless phones, personal digital assistants (Personal Digital Assistant, PDA), handheld devices, computing devices, in-vehicle devices, wearable devices, etc. The image reconstruction method can call the computer memory stored in the memory through the processor. This is achieved by reading instructions. Alternatively, the image reconstruction method can be executed by a server.

In a possible implementation manner, the target scene may be a geographic area including scenes such as buildings, landscapes, people, and vehicles. The target scene may be under dark light conditions (for example, night or other dark environments), and the image of the target scene collected by an image acquisition device (for example, an intensity camera or a camera, etc.) is underexposed and the image quality is poor. In this case, in step S11, an event collection device (such as an event camera) may be used to obtain event information of the target scene within the first brightness range corresponding to the dark light condition, and the event information is used to represent the target scene The brightness change in the first brightness range. The present disclosure does not limit the specific value of the first brightness range.

(1) 公式（1）中，

和

表示場景中第k個位置的事件資料

的空間座標，

表示事件資料

產生的時間，

表示事件資料

的極性，極性為正表示亮度增強，極性為負表示亮度降低。In a possible implementation, the event camera can record the brightness changes in the scene asynchronously, and output event data in the form of a stream (event stream). The data unit is as follows:

(1) In formula (1),

with

Indicates the event data at the k-th position in the scene

Space coordinates,

Indicates event data

Time generated,

Indicates event data

The polarity is positive, the brightness is increased, and the polarity is negative, the brightness is decreased.

The traditional CNN method can only process rule data in the form of pictures, and cannot be applied to event streams. Therefore, when the target scene is in the first brightness range, the brightness change of the target scene in one or more preset time periods can be collected by the event collection device to obtain event data, and the polarity of each event data can be measured in the spatial dimension. Integrate to obtain single-channel or multi-channel event information.

（2） 公式（2）中，

表示第k個位置的事件資料在預設時間段

內的事件資訊。這樣，對場景中各個位置的事件資料進行積分，可得到單通道的事件資訊（也可稱為事件幀）；對多個預設時間段內各個位置的事件資料進行積分，可得到多通道的事件資訊，例如四通道的事件資訊。為保證資料範圍的一致性，可將各通道的事件資訊分別在空間維度上進行標準化，將標準化後的事件資訊作為目標場景的事件資訊。本公開對事件資訊的通道數量不作限制The integration method is as follows:

(2) In formula (2),

Indicates that the event data at the kth position is in the preset time period

Event information within. In this way, by integrating event data at various locations in the scene, single-channel event information (also called event frames) can be obtained; by integrating event data at various locations in multiple preset time periods, multi-channel event information can be obtained. Event information, such as four-channel event information. In order to ensure the consistency of the data range, the event information of each channel can be standardized in the spatial dimension, and the standardized event information can be regarded as the event information of the target scene. This disclosure does not limit the number of channels for event information

In a possible implementation manner, feature extraction may be performed on the event information in step S12 to obtain the first event feature of the target scene. The first event feature includes at least information representing the structure of the target scene. For example, the feature of the event information can be extracted through a convolutional neural network, which can include multiple convolutional layers, multiple residual layers, etc. The present disclosure does not limit the network structure of the convolutional neural network.

In a possible implementation manner, image reconstruction may be performed on the first event feature in step S13 to obtain a reconstructed image of the target scene. The reconstructed image may be, for example, an intensity image, the brightness of the reconstructed image is within a second brightness range corresponding to normal lighting conditions, and the second brightness range is higher than the first brightness range.

In a possible implementation manner, image reconstruction of the first event feature may be performed, for example, through a deconvolutional neural network, which may include multiple deconvolutional layers, multiple residual layers, and convolutional layers. Etc., the present disclosure does not limit the specific value of the second brightness range and the network structure of the deconvolutional neural network.

According to the embodiment of the present disclosure, it is possible to obtain event information of the target scene in the lower first brightness range; perform feature extraction on the event information to obtain the event feature; perform image reconstruction on the event feature to obtain the target scene in the higher The reconstructed image in the second brightness range can reconstruct a high-quality image under normal lighting conditions through events under dark light conditions, which improves the effect of image reconstruction.

In a possible implementation manner, step S13 may include:

Performing detail enhancement on the first event feature according to the first noise information and the first event feature to obtain a second event feature;

Fusing the first event feature and the second event feature to obtain a fusion feature;

Image reconstruction is performed on the fusion feature to obtain a reconstructed image of the target scene.

For example, the event information obtained under dim light conditions may have more noise interference and partial structural information loss. In this case, the feature of the first event can be enhanced to recover more detailed information.

In a possible implementation, random first noise information can be preset, and additional noise channels are added to the first event feature based on the first noise information. The first event feature after adding the noise channel is input into the detail enhancement network for detail enhancement to obtain the second event feature. The detail enhancement network can be, for example, a residual network, including a convolutional layer and multiple residual layers. The present disclosure does not limit the acquisition method of the first noise information and the specific network structure of the detail enhancement network.

In a possible implementation, the first event feature and the second event feature can be fused, for example, superimposed, to obtain the fusion feature; the fusion feature is input into the deconvolutional neural network for image reconstruction, and the target scene is obtained. Reconstruct the image.

In this way, the detailed information in the features of the first event can be enhanced, and the quality of the reconstructed image can be further improved.

In a possible implementation, the image reconstruction method according to the embodiment of the present disclosure can be implemented by an image processing network, the image processing network includes at least a first feature extraction network and an image reconstruction network, the first The feature extraction network is used for feature extraction of the event information, such as a convolutional neural network; the image reconstruction network is used for image reconstruction of the first event feature, such as a deconvolutional neural network .

It should be understood that the image processing network can adopt other types of networks or models, and those skilled in the art can set it according to actual conditions, and the present disclosure does not limit this.

Before applying the image processing network, the image processing network can be trained.

In a possible implementation, the image reconstruction method according to the embodiment of the present disclosure further includes: training the image processing network according to a preset training set, the training set including the first sample scenes of a plurality of first sample scenes. One sample event information, second sample event information and sample scene images of multiple second sample scenes,

Wherein, the first sample event information is obtained in a third brightness range, the second sample event information is obtained in a fourth brightness range, and the sample scene image is obtained in the fourth brightness range. Acquired within the range, the fourth brightness range is higher than the third brightness range.

For example, a training set may be preset, and the training set includes multiple sample scenes, such as scenes such as buildings, landscapes, people, and vehicles. The sample scene can be divided into a dark light scene (may be called the first sample scene) and a normally illuminated scene (may be called the second sample scene). Each first sample scene includes first sample event information; each second sample scene includes second sample event information and sample scene images. The first sample scene and the second sample scene may be the same or different scenes, which is not limited in the present disclosure.

In a possible implementation manner, when the first sample scene is in the third brightness range corresponding to the dark light condition, the brightness change of the first sample scene can be acquired through an event collection device (for example, an event camera), and the first sample scene can be obtained. The same event information can be used as input to the image processing network. The first sample event information includes information representing the overall structure of the first sample scene. The third brightness range may be the same as or different from the aforementioned first brightness range, which is not limited in the present disclosure.

The first sample event information under dark light conditions includes information representing the overall structure of the first sample scene, but lacks intensity information (that is, brightness information of the image). In this case, the event information of the second sample scene under normal lighting conditions (which can be referred to as the second sample event information) can be introduced, so as to learn the intensity information in the second sample event information through the image processing network.

In a possible implementation manner, when the second sample scene is in the fourth brightness range corresponding to normal lighting conditions, the brightness change of the second sample scene can be acquired by the event collection device to obtain the second sample event information. The fourth brightness range is higher than the third brightness range. Wherein, the fourth brightness range may be the same as or different from the aforementioned second brightness range, which is not limited in the present disclosure.

The method for acquiring the first sample event information of the first sample scene and the second sample event information of the second sample scene may be similar to the method for acquiring the event information of the target scene, and the description will not be repeated here.

In addition, for the first sample scene under dark light conditions, the image quality of the target scene collected by the image acquisition device is poor and cannot be used as supervision information. In this case, the sample scene image of the second sample scene under normal lighting conditions can be introduced as the supervision information of the image processing network. The sample scene image may be acquired by an image acquisition device (for example, a camera) in the fourth brightness range corresponding to normal lighting conditions.

In this way, the training effect of the image processing network can be improved.

In a possible implementation manner, the image processing network further includes an identification network, and the step of training the image processing network according to a preset training set includes:

The first sample event information of the first sample scene and the second sample event information of the second sample scene are respectively input to the first feature extraction network to obtain first sample event features and second samples Event characteristics

Input the first sample event feature and the second sample event feature into the authentication network respectively to obtain a first authentication result and a second authentication result;

According to the first identification result and the second identification result, the image processing network is trained against training.

For example, the identification network in the image processing network is used to identify the output result of the first feature extraction network. That is to say, the first feature extraction network can be trained by adversarial training, so that the first feature extraction network can learn the first sample event information under dark light conditions and the second sample event information under normal lighting conditions. Distribute information together.

In a possible implementation manner, the first sample event information of the first sample scene and the second sample event information of the second sample scene can be separately input into the first feature extraction network for processing, and the first sample can be output Event characteristics and second sample event characteristics; input the first sample event characteristics and the second sample event characteristics into the identification network respectively to obtain the first identification result and the second identification result; according to the first identification result and the second identification result, Confrontation training the image processing network.

In the adversarial training process, the first feature extraction network tries to confuse the first sample event feature and the second sample event feature, and the identification network tries to distinguish the first sample event feature from the second sample event feature, and the two are opposed to each other. mutual improvement.

In this way, the first feature extraction network can be forced to extract the common distribution domain between the feature domain under normal lighting conditions and the feature domain under low light conditions, so that the first sample event feature under low light conditions has normal lighting conditions The distribution characteristics of event information under normal light conditions, and the second sample event characteristics under normal light conditions have the distribution characteristics of event information under dark light conditions. That is, through domain adaptation, the first feature extraction network is suitable for feature extraction of two differently distributed data at the same time. The present disclosure does not limit the selection of the loss function for the confrontation training.

In this way, the first feature extraction network can better extract event features under dark light, and improve the accuracy of the first feature extraction network, so as to use event information under dark light to achieve high-quality image reconstruction .

In a possible implementation, the step of training the image processing network according to a preset training set further includes:

Inputting the feature of the second sample event into the image reconstruction network to obtain a first reconstructed image of the second sample scene;

Training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image.

For example, after the confrontation training, the second sample event features extracted by the first feature extraction network have the characteristics of the distribution of event information under dark light conditions, and the corresponding second sample event information has supervision information (ie , The sample scene image under normal lighting conditions).

In a possible implementation, the step of training the image processing network according to a preset training set further includes:

Inputting the feature of the second sample event into the image reconstruction network to obtain a first reconstructed image of the second sample scene;

Training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image.

For example, after the confrontation training, the second sample event features extracted by the first feature extraction network have the characteristics of the distribution of event information under dark light conditions, and the corresponding second sample event information has supervision information (ie , The sample scene image under normal lighting conditions).

In a possible implementation, the image processing network further includes a second feature extraction network, and the step of training the image processing network according to a preset training set further includes:

Input second sample event information and second noise information of the second sample scene into the second feature extraction network to obtain third sample event features;

Fusing the second sample event feature with the third sample event feature to obtain a first sample fusion feature;

Input the fusion feature of the first sample into the authentication network to obtain a third authentication result;

According to the first identification result and the third identification result, the image processing network is trained against training.

For example, the event information of the first sample under dim light conditions may have a certain amount of noise interference, while the event information of the second sample under normal lighting conditions has lower noise. In this case, additional noise channels can be introduced for the second sample event information to improve the generalization of the network.

In a possible implementation, the image processing network further includes a second feature extraction network, such as a convolutional image processing network, including multiple convolutional layers and multiple residual layers. The network structure of the extraction network is not restricted.

In a possible implementation, random second noise information can be preset, and a noise channel is added to the second sample event information based on the second noise information. The second sample event information after adding the noise channel is input into the second feature extraction network for feature extraction, and the third sample event feature is output; the second sample event feature is fused with the third sample event feature to obtain Fusion features of the first sample. In this way, the feature enhancement of the second sample event feature can be achieved.

In a possible implementation, the fusion feature of the first sample is input into the identification network to obtain the third identification result; further, according to the first identification result and the third identification result, the image processing network is opposed to training road. The specific process of confrontation training will not be repeated.

In this way, the accuracy of the first feature extraction network can be further improved.

In a possible implementation, the image processing network further includes a second feature extraction network, and the step of training the image processing network according to a preset training set further includes:

Inputting the first sample fusion feature into the image reconstruction network to obtain a second reconstructed image of the second sample scene;

Training the image processing network according to the second reconstructed image of the second sample scene and the sample scene image.

For example, after the confrontation training, the first sample fusion features extracted by the first feature extraction network and the second feature extraction network have the characteristics of the distribution of event information under dark light conditions, and the corresponding second The sample event information has supervision information (ie, the sample scene image under normal lighting conditions).

In a possible implementation, the fusion feature of the first sample can be input into the image reconstruction network for processing, and the second reconstructed image of the second sample scene is output; the second reconstructed image according to the second sample scene and The difference between the sample scene images can determine the network loss of the first feature extraction network, the second feature extraction network, and the image reconstruction network, such as L1 loss; further, it can be adjusted inversely according to the network loss The network parameters of the first feature extraction network, the second feature extraction network, and the image reconstruction network realize the training of the first feature extraction network, the second feature extraction network, and the image reconstruction network.

In the actual training process, alternate training can also be performed. That is, in each round of iterative calculations, the network parameters of the identification network are adjusted in the reverse direction according to the loss of the counter network; then the network parameters of the first feature extraction network, the second feature extraction network, and the image reconstruction network are adjusted Path loss, reversely adjust the network parameters of the first feature extraction network, the second feature extraction network, and the image reconstruction network. In this training, the output of the identification network will still be obtained as guidance information, but the identification will not be updated The parameters of the network. In this way, after multiple rounds of iterative calculations, the trained image processing network can be obtained when the training conditions (such as network convergence) are met.

In this way, the training process of the entire image processing network can be realized, and a high-precision image processing network can be obtained.

In a possible implementation manner, the image processing network further includes a detail enhancement network, and the step of training the image processing network according to a preset training set may further include:

Input the second sample event feature and the third noise information into the detail enhancement network to obtain the fourth sample event feature;

Fusing the second sample event feature with the fourth sample event feature to obtain a second sample fusion feature;

Input the second sample fusion feature into the image reconstruction network to obtain a third reconstructed image of the second sample scene;

Training the image processing network according to the first reconstructed image of the second sample scene, the third reconstructed image, and the sample scene image.

For example, the detail enhancement network can be introduced to enhance the details of the event features in order to recover more detailed image information (such as local structural information). The detail enhancement network may be, for example, a residual network, including a convolutional layer and multiple residual layers. The present disclosure does not limit the network structure of the detail enhancement network.

In a possible implementation manner, in the case where the second feature extraction network is not introduced, the second sample event feature can be directly used for detail enhancement. A random third noise information can be preset, and a noise channel can be added to the second sample event feature based on the third noise information. The second sample event feature after adding the noise channel is input into the detail enhancement network for processing to obtain the fourth sample event feature; the second sample event feature is fused with the fourth sample event feature to obtain the second sample fusion feature; The second sample fusion feature is input into the image reconstruction network to obtain a third reconstructed image of the second sample scene.

In a possible implementation manner, the image processing network is trained according to the first reconstructed image, the third reconstructed image, and the sample scene image of the sample scene.

Among them, according to the difference between the third reconstructed image and the sample scene image, the first loss of the first feature extraction network, the detail enhancement network, and the image reconstruction network can be determined; according to the third reconstructed image and the sample The difference between the scene images and the difference between the first reconstructed image and the sample scene image can determine the second loss of the first feature extraction network, the detail enhancement network, and the image reconstruction network. The second loss can ensure that the quality of the third reconstructed image after the detail enhancement is introduced is better than the quality of the first reconstructed image when the detail enhancement is not introduced, ensuring that the detail enhancement network can play an expected role.

In a possible implementation, the total loss of the first feature extraction network, the detail enhancement network, and the image reconstruction network can be determined based on the first loss and the second loss, for example, the first loss and the second loss are weighted The sum is determined as the overall loss; further, the network parameters of the first feature extraction network, the detail enhancement network, and the image reconstruction network can be adjusted inversely according to the overall loss to realize the first feature extraction network and the detail enhancement network And training of image reconstruction network.

In the actual training process, alternate training can also be performed. That is, in each round of iterative operations, the identification network is trained against; the first feature extraction network, the detail enhancement network and the image reconstruction network are trained again, and the output of the identification network is used as guidance information, but the identification network is not updated Parameters. After multiple rounds of iterative calculations, the trained image processing network can be obtained when the training conditions (such as network convergence) are met.

In this way, the details of the reconstructed image can be enhanced, and the quality of the reconstructed image obtained by the trained image processing network can be further improved.

In a possible implementation, the step of training the image processing network according to a preset training set may further include:

Input the fusion feature of the first sample and the fourth noise information into the detail enhancement network to obtain the event feature of the fifth sample;

Fuse the first sample fusion feature with the fifth sample event feature to obtain a third sample fusion feature;

Input the third sample fusion feature into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene;

Training the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image.

For example, when the second feature extraction network has been introduced, the first sample fusion feature can be used for detail enhancement. A random fourth noise information can be preset, and a noise channel is added to the first sample fusion feature based on the fourth noise information. Input the fusion feature of the first sample after adding the noise channel to the detail enhancement network for processing to obtain the event feature of the fifth sample; fuse the fusion feature of the first sample with the event feature of the fifth sample to obtain the fusion feature of the third sample; The third sample fusion feature is input into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene.

In a possible implementation manner, an image processing network is trained based on the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image. This step can include:

Determine the overall loss of the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image;

Determine the gradient information of the image processing network according to the total loss;

Adjusting the network parameters of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network according to the gradient information,

Wherein, the gradient information of the detail enhancement network is not transmitted to the second feature extraction network.

For example, according to the difference between the fourth reconstructed image and the sample scene image, the third loss of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network can be determined ; According to the difference between the fourth reconstructed image and the sample scene image, and the difference between the second reconstructed image and the sample scene image, the first feature extraction network, the second feature extraction network, and the details can be determined The fourth loss of the enhancement network and the image reconstruction network. The fourth loss can ensure that the quality of the fourth reconstructed image after the detail enhancement is introduced is better than the quality of the second reconstructed image when the detail enhancement is not introduced, ensuring that the detail enhancement network can play an expected role.

In a possible implementation, the total loss of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network can be determined based on the third loss and the fourth loss. For example, the third loss The weighted sum of the loss and the fourth loss is determined as the overall loss; according to the overall loss, the gradient information of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network can be determined, and further, The gradient information can be transmitted backwards in the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network, so as to adjust the first feature extraction network, the second feature extraction network, The network parameters of the detail enhancement network and the image reconstruction network realize the training of the first feature extraction network, the second feature extraction network, the detail enhancement network and the image reconstruction network.

In a possible implementation, noise channels are added to the inputs of the second feature extraction network and the detail enhancement network. Therefore, in order to reduce the impact of the early training stage on the learning effect, when the gradient information is transmitted backwards, The stop gradient between the detail enhancement network and the second feature extraction network can reduce the mutual interference between the detail enhancement network and the second feature extraction network, effectively reducing the loops in the information flow , To reduce the probability of mode collapse.

In the actual training process, alternate training can also be performed. That is, in each round of iterative operation, the discriminating network is trained against the training. Retrain the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network. The output of the identification network is used as guidance information, but the parameters of the identification network are not updated. After multiple rounds of iterative calculations, the trained image processing network can be obtained when the training conditions (such as network convergence) are met.

In this way, the details of the reconstructed image can be enhanced, and the quality of the reconstructed image obtained by the trained image processing network can be further improved.

Fig. 2 shows a schematic diagram of the network training process of the image reconstruction method according to an embodiment of the present disclosure. 2, the image processing network according to the disclosed embodiments of the present embodiment comprises a first feature extraction network E _C, the second feature extraction network E _P, D network authentication, detail enhancement and image network T _e Rebuild the network R.

In an example, for any set of the first sample and the second sample scenario scenario, the event may be the first sample in low light conditions information feature extractor 21 receives the first E _C in the web, and outputs the first sample Event feature X _LE ; input the second sample event information 22 under normal lighting conditions into the first feature extraction network E _C that shares the parameters for processing, and output the second sample event feature X _C ; for the second sample under normal lighting conditions event information 22 is added after the noise information 23, the second input parameter characteristic not shared network E _P extraction processing, output of the third characteristic _X-P sample event; event wherein the second sample and the third sample _C X X event characteristics _p is superimposed to obtain the first sample fusion feature X _DE ; the first sample event feature X _LE and the first sample fusion feature X _DE are respectively input into the identification network D for identification, and the respective identification results (not shown) out).

（3） 公式（3）中，

和

分別表示第一樣本事件特徵X_LE 和第一樣本融合特徵X_DE 對應的損失。In the example, the discrimination network D is trained against the discrimination result according to the discrimination result. The network loss L _{D is} expressed as follows:

(3) In formula (3),

with

Respectively represent the loss corresponding to the first sample event feature X _LE and the first sample fusion feature X _DE.

；同時，對第一樣本融合特徵X_DE 添加雜訊資訊24後，輸入細節增強網路T_e 中，輸出第五樣本事件特徵

；將第一樣本融合特徵X_DE 與第五樣本事件特徵

融合後，輸入圖像重建網路R中，輸出第四重建圖像

。In the example, the first sample fusion feature X _{DE is} input into the image reconstruction network R, and the second reconstructed image is output

; At the same time, after adding noise information 24 to the first sample fusion feature X _DE , input it into the detail enhancement network _Te , and output the fifth sample event feature

; Fuse the feature X _{DE of the} first sample with the event feature of the fifth sample

After fusion, enter the image reconstruction network R, and output the fourth reconstructed image

.

、第四重建圖像

及所述樣本場景圖像

（未示出），可確定第一特徵提取網路E_C 、第二特徵提取網路E_P 、細節增強網路T_e 及圖像重建網路R的總體損失

（也可稱為重建損失），表示如下：

（4） 公式（4）中，

表示亮度重建損失，可以為第四重建圖像

與所述樣本場景圖像

之間的L1損失，以及第二重建圖像

與所述樣本場景圖像

之間的L1損失的和。

表示細節增強網路的殘差損失，可以為

與-X_p 之間的L1損失（表示為

）。

表示排名損失，可以為第四重建圖像

與所述樣本場景圖像

之間的L1損失，以及第二重建圖像

與所述樣本場景圖像

之間的L1損失的差。β和γ表示超參數項，本領域技術人員可根據實際情況設置In the example, according to the second reconstructed image

, The fourth reconstructed image

And the sample scene image

(Not shown), may determine the first feature extraction network E _C, the second feature extraction network E _P, the overall loss of detail enhancement, and image reconstruction network T _e of the web R

(It can also be called reconstruction loss), which is expressed as follows:

(4) In formula (4),

Indicates the loss of brightness reconstruction, which can be the fourth reconstructed image

With the sample scene image

L1 loss between and the second reconstructed image

With the sample scene image

The sum of L1 losses between.

Represents the residual loss of the detail enhancement network, which can be

L1 loss between -X _{p (expressed as}

).

Represents the ranking loss, which can be the fourth reconstructed image

With the sample scene image

L1 loss between and the second reconstructed image

With the sample scene image

The difference between the L1 loss. β and γ represent hyperparameter items, which can be set by those skilled in the art according to the actual situation

的第一項用於確保網路能夠恢復出正確的圖像，第二項用於保證細節增強網路的精度，第三項用於保證網路在引入細節增強網路T_e 後的重構效果更好，使得細節增強網路T_e 能真正地起到細節增強的作用。in,

The first is used to ensure the correct web to recover the image, the second network to ensure the accuracy of detail enhancement, for securing the third web reconstructed after introduction detail enhancement T _e of the web The effect is better, so that the detail enhancement network _Te can really play the role of detail enhancement.

（5）In an example, the overall optimization goal of the image processing network according to the embodiment of the present disclosure can be expressed as follows:

(5)

分別表示用於第一特徵提取網路E_C 、第二特徵提取網路E_P 、圖像重建網路R及細節增強網路T_e 的參數；

表示鑒別網路D的參數；

是相應的超參數權重，本領域技術人員可根據實際情況設置。根據本公開實施例，可使用對抗式訓練交替優化這兩類參數，可例如採用隨機批次處理梯度下降的方式進行訓練，本公開對此不作限制。經訓練後，可得到高精度的圖像處理網路。In formula (5),

Denote a first feature extracting network E _C, the second feature extraction network E _P, the image reconstruction of the network R and detail enhancement parameters T _e of the web;

Indicates the parameters of the authentication network D;

Is the corresponding hyperparameter weight, which can be set by those skilled in the art according to the actual situation. According to the embodiments of the present disclosure, adversarial training can be used to alternately optimize the two types of parameters, and training can be performed, for example, by using random batch processing gradient descent, which is not limited in the present disclosure. After training, a high-precision image processing network can be obtained.

According to the image reconstruction method of the embodiment of the present disclosure, by combining the domain self-adjustment method with the event camera, image reconstruction is performed using event information under dark light conditions to obtain high-quality images under normal lighting conditions, thereby improving the image The effect of reconstruction. In the training process, this method does not need to perform supervised training on intensity images under dark light, realizes an unsupervised network framework, and reduces the difficulty of data set construction. This method enhances the dark light distribution area in the event feature through the detail enhancement network, reduces the noise interference, enhances the local details, and improves the effect of image reconstruction and training.

The network framework of the image reconstruction method according to the embodiments of the present disclosure does not depend on event information, and is also applicable to other tasks based on domain self-adjustment methods, such as image style transformation, semantic segmentation domain self-adjustment, and so on. Just change the corresponding input data and replace the image reconstruction network with the network structure corresponding to the respective tasks.

The image reconstruction method according to the embodiment of the present disclosure can be applied to the fields of image shooting, image processing, face recognition, security, etc., to realize image reconstruction under dark light conditions.

For example, the shooting system of electronic equipment (such as smart phones) using related technologies is based on intensity cameras, which cannot be imaged in low light conditions. Using flash as an aid to take pictures or record videos will bring about a great increase in energy consumption, and flash The dazzling light is very unfriendly to the people in the scene. The highly dynamic event camera does not require additional light source assistance, and the energy consumption is very low. The event camera can be set to obtain event information under dark light conditions, and through the image reconstruction method of the embodiment of the present disclosure, a clear image is generated according to the event information, thereby realizing image shooting under dark light conditions.

For example, the image reconstruction method of the embodiment of the present disclosure can be used as an upstream algorithm of various image processing algorithms. Image processing tasks such as face recognition, object detection, semantic segmentation, etc. will fail due to the inability to obtain high-quality intensity images in low light conditions. This image reconstruction method can reconstruct the intensity image under dark light based on the event information under dark light conditions, so that the above algorithm can continue to be applied.

For example, a large number of intensity cameras are used in the security field of cities, and there will be many blind spots that cannot be clearly detected in shadow areas and dark light conditions. The event camera can be set to obtain event information under dark light conditions, and through the image reconstruction method of the embodiment of the present disclosure, a clear image is generated according to the event information, thereby improving the effect of security detection and ensuring city safety.

It can be understood that the various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle and logic. The length is limited, and the details of this disclosure will not be repeated. Those skilled in the art can understand that, in the above-mentioned image reconstruction method of the specific embodiment, the specific execution order of each step should be determined by its function and possible internal logic.

In addition, the present disclosure also provides image reconstruction devices, electronic equipment, computer-readable storage media, and programs, all of which can be used to implement any of the image reconstruction methods provided in the present disclosure. For the corresponding technical solutions and descriptions, refer to the corresponding methods in the method section. Record, not repeat it.

Fig. 3 shows a block diagram of an image reconstruction device according to an embodiment of the present disclosure. As shown in Fig. 3, the image reconstruction device includes:

The event acquisition module 31 is configured to acquire event information of a target scene, where the event information is used to indicate a brightness change of the target scene within a first brightness range;

The feature extraction module 32 is configured to perform feature extraction on the event information to obtain the first event feature of the target scene;

The image reconstruction module 33 is configured to perform image reconstruction on the first event feature to obtain a reconstructed image of the target scene, the brightness of the reconstructed image is within a second brightness range, and the second brightness The range is higher than the first brightness range.

In a possible implementation, the image reconstruction module 33 includes: a detail enhancement sub-module for performing detail enhancement on the first event feature according to the first noise information and the first event feature , Obtain a second event feature; a fusion sub-module, used to fuse the first event feature with the second event feature to obtain a fusion feature; a reconstruction sub-module, used to perform image reconstruction on the fusion feature To obtain a reconstructed image of the target scene.

In a possible implementation manner, the image reconstruction device is implemented by an image processing network, the image processing network includes a first feature extraction network and an image reconstruction network, the first feature extraction network The path is used for feature extraction of the event information, the image reconstruction network is used for image reconstruction of the first event feature, and the image reconstruction device further includes:

The training module is used to train the image processing network according to a preset training set. The training set includes first sample event information of a plurality of first sample scenes, and second sample scenes of a plurality of second sample scenes. Sample event information and sample scene images; wherein the first sample event information is acquired within a third brightness range, the second sample event information is acquired within a fourth brightness range, and the sample scene The image is acquired in the fourth brightness range, and the fourth brightness range is higher than the third brightness range.

In a possible implementation, the image processing network further includes an identification network, and the training module includes: a first extraction sub-module configured to convert the first sample of the first sample scene The event information and the second sample event information of the second sample scene are respectively input into the first feature extraction network to obtain the first sample event feature and the second sample event feature; the first discrimination sub-module is used to combine The first sample event feature and the second sample event feature are respectively input to the identification network to obtain a first identification result and a second identification result; the first confrontation training sub-module is used to obtain the first identification result according to the first The identification result and the second identification result are used against training the image processing network.

In a possible implementation, the training module further includes: a first reconstruction sub-module, configured to input the second sample event feature into the image reconstruction network to obtain the second sample scene A first reconstructed image; a first training sub-module for training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image.

In a possible implementation, the image processing network further includes a detail enhancement network, and the training module further includes: a first enhancement sub-module for combining the second sample event features and the third The noise information is input into the detail enhancement network to obtain the fourth sample event feature; the first fusion sub-module is used to fuse the second sample event feature with the fourth sample event feature to obtain a second sample fusion Features; a second reconstruction sub-module, used to input the second sample fusion feature into the image reconstruction network to obtain a third reconstructed image of the second sample scene; a second training sub-module, using Training the image processing network according to the first reconstructed image of the second sample scene, the third reconstructed image, and the sample scene image.

In a possible implementation manner, the image processing network further includes a second feature extraction network, and the training module further includes: a second extraction sub-module configured to convert the second feature extraction network of the second sample scene Two sample event information and second noise information are input into the second feature extraction network to obtain a third sample event feature; a second fusion sub-module is used to combine the second sample event feature with the third sample Event feature fusion to obtain a first sample fusion feature; a second discrimination sub-module for inputting the first sample fusion feature into the discrimination network to obtain a third discrimination result; a second confrontation training sub-module , Used for training the image processing network against the first identification result and the third identification result.

In a possible implementation, the training module further includes: a third reconstruction sub-module, configured to input the first sample fusion feature into the image reconstruction network to obtain the second sample scene The second reconstructed image; the third training sub-module is used to train the image processing network according to the second reconstructed image of the second sample scene and the sample scene image.

In a possible implementation manner, the image processing network further includes a detail enhancement network, and the training module further includes: a second enhancement sub-module for fusing the features of the first sample with the first sample Four noise information is input into the detail enhancement network to obtain the fifth sample event feature; the third fusion sub-module is used to fuse the first sample fusion feature with the fifth sample event feature to obtain the third Sample fusion features; a fourth reconstruction sub-module, used to input the third sample fusion features into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene; a fourth training sub-module , For training the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image.

In a possible implementation manner, the fourth training sub-module is used to determine the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image. The overall loss of the image processing network; according to the overall loss, the gradient information of the image processing network is determined; according to the gradient information, the first feature extraction network and the second feature extraction are adjusted The network, the detail enhancement network and the network parameters of the image reconstruction network, wherein the gradient information of the detail enhancement network is not transmitted to the second feature extraction network.

In some embodiments, the functions or modules included in the image reconstruction apparatus provided in the embodiments of the present disclosure can be used to execute the methods described in the above method embodiments, and for specific implementation, refer to the description of the above method embodiments. For the sake of brevity, I won't repeat them here.

The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions are executed by a processor to realize the above-mentioned image reconstruction method. The computer-readable storage medium may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium.

An embodiment of the present disclosure also provides an electronic device, including: a processor; a memory for storing executable instructions of the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above image Reconstruction method.

The embodiments of the present disclosure also provide a computer program product, including computer-readable code. When the computer-readable code runs on the device, the processor in the device executes the image reconstruction method for implementing the image reconstruction method provided by any of the above embodiments. instruction.

The embodiments of the present disclosure also provide another computer program product for storing computer-readable instructions, which when executed cause the computer to perform the operation of the image reconstruction method provided by any of the above-mentioned embodiments.

Electronic devices can be provided as terminals, servers, or other types of devices.

FIG. 4 shows a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcasting terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and other terminals.

4, the electronic device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input/output interface 812, a sensor component 814, and communication Components 816.

The processing component 802 generally controls the overall operations of the electronic device 800, such as operations associated with display, telephone calls, data communication, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the foregoing method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support the operation of the electronic device 800. Examples of such data include instructions of any application or method used to operate on the electronic device 800, contact information, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage devices or their combination, such as static random access memory (SRAM), electronically erasable rewritable read-only memory (EEPROM), erasable Programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, floppy disk or CD-ROM.

The power supply component 806 provides power for various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with the generation, management, and distribution of power for the electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, sliding, and gestures on the touch panel. The touch sensor can not only sense the boundary of the touch or slide action, but also detect the duration and pressure related to the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and/or a rear camera. When the electronic device 800 is in an operation mode, such as a shooting mode or a video mode, the front camera and/or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a microphone (MIC). When the electronic device 800 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive external audio signals. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 further includes a speaker for outputting audio signals.

The input/output interface 812 provides an interface between the processing component 802 and a peripheral interface module. The peripheral interface module may be a keyboard, a click wheel, a button, and the like. These buttons may include, but are not limited to: home button, volume button, start button, and lock button.

The sensor component 814 includes one or more sensors for providing the electronic device 800 with various aspects of state evaluation. For example, the sensor component 814 can detect the on/off state of the electronic device 800 and the relative positioning of the components. For example, the component is the display and the keypad of the electronic device 800. The sensor component 814 can also detect the electronic device 800 or The position of a component of the electronic device 800 changes, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration/deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects when there is no physical contact. The sensor component 814 may also include a light sensor, such as a complementary metal oxide semiconductor (CMOS) or charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor component 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second-generation mobile communication technology (2G) or a third-generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASIC), digital signal processor (DSP), digital signal processing device (DSPD), programmable logic device (PLD), On-site programmable gate array (FPGA), controller, microcontroller, microprocessor or other electronic components are used to implement the above-mentioned image reconstruction method.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as a memory 804 including computer program instructions, which can be executed by the processor 820 of the electronic device 800 to complete the image reconstruction method described above. .

FIG. 5 shows a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. 5, the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions that can be executed by the processing component 1922, such as application programs. The application program stored in the memory 1932 may include one or more modules each corresponding to a set of commands. In addition, the processing component 1922 is configured to execute instructions to perform the above-mentioned image reconstruction method.

The electronic device 1900 may also include a power component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input and output (I/O) Interface 1958. The electronic device 1900 can operate operating systems based on storage in the memory 1932, such as Microsoft's server operating system (Windows Server ^TM ), Apple’s graphical user interface operating system (Mac OS X ^TM ), multi-user and multi-stroke Computer operating system (Unix ^TM ), free and open source UNIX-like operating system (Linux ^TM ), open source UNIX-like operating system (FreeBSD ^TM ) or similar.

In an exemplary embodiment, there is also provided a non-volatile computer-readable storage medium, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to complete the image reconstruction method described above. .

The present disclosure may be a system, method, and/or computer program product. The computer program product may include a computer-readable storage medium loaded with computer-readable program instructions for enabling the processor to implement various aspects of the present disclosure.

The computer-readable storage medium may be a tangible device that can hold and store instructions used by the instruction execution device. The computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (non-exhaustive list) include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable and programmable Read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital audio-visual disc (DVD), memory stick, Floppy disks, mechanical encoding devices, such as punch cards on which instructions are stored or raised structures in the grooves, and any suitable combination of the above. The computer-readable storage medium used here is not interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (for example, light pulses through fiber optic cables), or passing through Electrical signals transmitted by wires.

The computer-readable program instructions described here can be downloaded from a computer-readable storage medium to various computing/processing devices, or downloaded to an external computer or via a network, such as the Internet, a local area network, a wide area network, and/or a wireless network. External storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and/or edge servers. The network interface card or network interface in each computing/processing device receives computer-readable program instructions from the network and forwards the computer-readable program instructions for computer-readable storage in each computing/processing device In the media.

The computer program instructions used to perform the operations of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or any of one or more programming languages. Combining source code or object code written, the programming language includes object-oriented programming languages-such as Smalltalk, C++, etc., and conventional procedural programming languages-such as "C" language or similar programming languages. Computer-readable program instructions can be executed entirely on the user's computer, partly on the user's computer, executed as a stand-alone software package, partly on the user's computer and partly executed on a remote computer, or completely remotely executed. Run on the end computer or server. In the case of a remote computer, the remote computer can be connected to the user’s computer through any kind of network—including a local area network (LAN) or a wide area network (WAN)—or, it can be connected to an external computer (for example, using Internet service providers to connect via the Internet). In some embodiments, the electronic circuit is personalized by using the status information of computer-readable program instructions, such as programmable logic circuit, field programmable gate array (FPGA), or programmable logic array (PLA). The electronic circuit can execute computer-readable program instructions to realize various aspects of the present disclosure.

Herein, various aspects of the present disclosure are described with reference to flowcharts and/or block diagrams of image reconstruction methods, image reconstruction devices (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block of the flowchart and/or block diagram and the combination of each block in the flowchart and/or block diagram can be implemented by computer-readable program instructions.

These computer-readable program instructions can be provided to the processors of general-purpose computers, special-purpose computers, or other programmable data processing devices, thereby producing a machine that allows these instructions to be executed by the processors of the computer or other programmable data processing devices At this time, a device that implements the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams is produced. It is also possible to store these computer-readable program instructions in a computer-readable storage medium. These instructions make the computer, programmable data processing device and/or other equipment work in a specific manner, so that the computer-readable medium storing the instructions is It includes an article of manufacture, which includes instructions for implementing various aspects of the functions/actions specified in one or more blocks in the flowchart and/or block diagram.

It is also possible to load computer-readable program instructions into a computer, other programmable data processing device, or other equipment, so that a series of operation steps are executed on the computer, other programmable data processing device, or other equipment to generate a computer The process of implementation enables instructions executed on a computer, other programmable data processing device, or other equipment to implement the functions/actions specified in one or more blocks in the flowcharts and/or block diagrams.

The flowcharts and block diagrams in the drawings show the possible implementation architecture, functions, and operations of the system, method, and computer program product according to multiple embodiments of the present disclosure. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of an instruction, and the module, program segment, or part of an instruction includes one or more Executable instructions for logic functions. In some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two consecutive blocks can actually be executed basically in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, as well as the combination of the blocks in the block diagram and/or flowchart, can be used to perform specified functions or actions based on dedicated hardware. The system can be implemented, or it can be implemented by a combination of dedicated hardware and computer instructions.

The computer program product can be implemented by hardware, software, or a combination thereof. In an optional embodiment, the computer program product is specifically embodied as a computer storage medium. In another optional embodiment, the computer program product is specifically embodied as a software product, such as a software development kit (SDK), etc. Wait.

Without violating logic, different embodiments of the present disclosure can be combined with each other, and the description of different embodiments is emphasized. For the part of the description, reference may be made to the records of other embodiments.

The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Without departing from the scope and spirit of the illustrated embodiments, many modifications and changes are obvious to those of ordinary skill in the art. The choice of terms used herein is intended to best explain the principles, practical applications, or improvements to technologies in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

:第二重建圖像

:第四重建圖像

:第五樣本事件特徵 21:第一樣本事件資訊 22:第二樣本事件資訊 23:雜訊資訊 24:雜訊資訊 31:事件獲取模組 32:特徵提取模組 33:圖像重建模組 800:電子設備 802:處理組件 804:記憶體 806:電源組件 808:多媒體組件 810:音訊組件 812:輸入/輸出介面 814:感測器組件 816:通信組件 820:處理器 1900:電子設備 1922:處理組件 1926:電源組件 1932:記憶體 1950:網路介面 1958:輸入輸出介面S11 ~ S13: Process Step D: Identification Network E _C: the first network feature extraction E _P: a second network feature extraction R: image reconstruction network T _e: Detail Enhancement Network X _LE: the first sample Event feature X _C : Second sample event feature X _p : Third sample event feature X _DE : First sample fusion feature

: The second reconstructed image

: The fourth reconstructed image

: Fifth sample event feature 21: First sample event information 22: Second sample event information 23: Noise information 24: Noise information 31: Event acquisition module 32: Feature extraction module 33: Image reconstruction module 800: electronic device 802: processing component 804: memory 806: power supply component 808: multimedia component 810: audio component 812: input/output interface 814: sensor component 816: communication component 820: processor 1900: electronic device 1922: Processing component 1926: Power component 1932: Memory 1950: Network interface 1958: Input and output interface

Other features and effects of the present invention will be clearly presented in the embodiments with reference to the drawings, in which: Fig. 1 shows a flowchart of an image reconstruction method according to an embodiment of the present disclosure. Fig. 2 shows a schematic diagram of the network training process of the image reconstruction method according to an embodiment of the present disclosure. Fig. 3 shows a block diagram of an image reconstruction device according to an embodiment of the present disclosure. Fig. 4 shows a block diagram of an electronic device according to an embodiment of the present disclosure. Fig. 5 shows a block diagram of an electronic device according to an embodiment of the present disclosure.

S11~S13: Process steps

Claims (13)

An image reconstruction method, including: Acquiring event information of a target scene, where the event information is used to indicate a brightness change of the target scene within a first brightness range; Performing feature extraction on the event information to obtain the first event feature of the target scene; Perform image reconstruction on the first event feature to obtain a reconstructed image of the target scene, the brightness of the reconstructed image is within a second brightness range, and the second brightness range is higher than the first brightness range . The image reconstruction method according to claim 1, wherein performing image reconstruction on the first event feature to obtain a reconstructed image of the target scene includes: Performing detail enhancement on the first event feature according to the first noise information and the first event feature to obtain a second event feature; Fusing the first event feature and the second event feature to obtain a fusion feature; Image reconstruction is performed on the fusion feature to obtain a reconstructed image of the target scene. The image reconstruction method according to claim 1 or 2, wherein the image reconstruction method is implemented by an image processing network, and the image processing network includes a first feature extraction network and an image reconstruction network The first feature extraction network is used for feature extraction of the event information, and the image reconstruction network is used for image reconstruction of the first event feature: The image reconstruction method further includes: training the image processing network according to a preset training set, the training set including first sample event information of a plurality of first sample scenes, and a plurality of second sample scenes The second sample event information and sample scene images of; Wherein, the first sample event information is obtained in a third brightness range, the second sample event information is obtained in a fourth brightness range, and the sample scene image is obtained in the fourth brightness range. Acquired within the range, the fourth brightness range is higher than the third brightness range. The image reconstruction method according to claim 3, wherein the image processing network further includes a discrimination network, and the training of the image processing network according to a preset training set includes: The first sample event information of the first sample scene and the second sample event information of the second sample scene are respectively input to the first feature extraction network to obtain first sample event features and second samples Event characteristics Input the first sample event feature and the second sample event feature into the authentication network respectively to obtain a first authentication result and a second authentication result; According to the first identification result and the second identification result, the image processing network is trained against training. The image reconstruction method according to claim 4, wherein the training of the image processing network according to a preset training set further includes: Inputting the feature of the second sample event into the image reconstruction network to obtain a first reconstructed image of the second sample scene; Training the image processing network according to the first reconstructed image of the second sample scene and the sample scene image. The image reconstruction method according to claim 5, wherein the image processing network further includes a detail enhancement network, and the training of the image processing network according to a preset training set further includes: Input the second sample event feature and the third noise information into the detail enhancement network to obtain the fourth sample event feature; Fusing the second sample event feature with the fourth sample event feature to obtain a second sample fusion feature; Input the second sample fusion feature into the image reconstruction network to obtain a third reconstructed image of the second sample scene; Training the image processing network according to the first reconstructed image of the second sample scene, the third reconstructed image, and the sample scene image. The image reconstruction method according to claim 4, wherein the image processing network further includes a second feature extraction network, and the training of the image processing network according to a preset training set further includes: Input second sample event information and second noise information of the second sample scene into the second feature extraction network to obtain third sample event features; Fusing the second sample event feature with the third sample event feature to obtain a first sample fusion feature; Input the fusion feature of the first sample into the authentication network to obtain a third authentication result; According to the first identification result and the third identification result, the image processing network is trained against training. The image reconstruction method according to claim 7, wherein the training of the image processing network according to a preset training set further includes: Inputting the first sample fusion feature into the image reconstruction network to obtain a second reconstructed image of the second sample scene; Training the image processing network according to the second reconstructed image of the second sample scene and the sample scene image. The image reconstruction method according to claim 8, wherein the image processing network further includes a detail enhancement network, and the training of the image processing network according to a preset training set further includes: Input the fusion feature of the first sample and the fourth noise information into the detail enhancement network to obtain the event feature of the fifth sample; Fuse the first sample fusion feature with the fifth sample event feature to obtain a third sample fusion feature; Input the third sample fusion feature into the image reconstruction network to obtain a fourth reconstructed image of the second sample scene; Training the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image. The image reconstruction method according to claim 9, wherein the image is trained according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image Handling the network, including: Determine the overall loss of the image processing network according to the second reconstructed image of the second sample scene, the fourth reconstructed image, and the sample scene image; Determine the gradient information of the image processing network according to the total loss; Adjusting the network parameters of the first feature extraction network, the second feature extraction network, the detail enhancement network, and the image reconstruction network according to the gradient information, Wherein, the gradient information of the detail enhancement network is not transmitted to the second feature extraction network. An image reconstruction device includes: An event acquisition module for acquiring event information of a target scene, where the event information is used to indicate a brightness change of the target scene within a first brightness range; The feature extraction module is used to perform feature extraction on the event information to obtain the first event feature of the target scene; The image reconstruction module is configured to perform image reconstruction on the first event feature to obtain a reconstructed image of the target scene, the brightness of the reconstructed image is within a second brightness range, and the second brightness range Higher than the first brightness range. An electronic device including: processor; Memory used to store executable instructions of the processor; Wherein, the processor is configured to call instructions stored in the memory to execute the image reconstruction method described in any one of request items 1 to 10. A computer-readable storage medium has computer program instructions stored thereon, and when the computer program instructions are executed by a processor, the image reconstruction method described in any one of request items 1 to 10 is realized.

Images