CN112085768B - Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium - Google Patents

Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium Download PDF

Info

Publication number
CN112085768B
CN112085768B CN202010909623.6A CN202010909623A CN112085768B CN 112085768 B CN112085768 B CN 112085768B CN 202010909623 A CN202010909623 A CN 202010909623A CN 112085768 B CN112085768 B CN 112085768B
Authority
CN
China
Prior art keywords
information
snn
optical flow
deconvolution
input
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
CN202010909623.6A
Other languages
Chinese (zh)
Other versions
CN112085768A (en
Inventor
施路平
杨哲宇
盛凯枫
赵蓉
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Lynxi Technology Co Ltd
Original Assignee
Beijing Lynxi Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Lynxi Technology Co Ltd filed Critical Beijing Lynxi Technology Co Ltd
Priority to CN202010909623.6A priority Critical patent/CN112085768B/en
Publication of CN112085768A publication Critical patent/CN112085768A/en
Priority to TW110132594A priority patent/TW202211163A/en
Priority to PCT/CN2021/116133 priority patent/WO2022048582A1/en
Application granted granted Critical
Publication of CN112085768B publication Critical patent/CN112085768B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/20—Analysis of motion
    • G06T7/269—Analysis of motion using gradient-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T9/00—Image coding
    • G06T9/002—Image coding using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Evolutionary Computation (AREA)
  • Multimedia (AREA)
  • Artificial Intelligence (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Image Analysis (AREA)

Abstract

The invention provides an optical flow information prediction method, an optical flow information prediction device, electronic equipment and a storage medium, wherein the optical flow information prediction method comprises the following steps: acquiring a DVS pulse signal; extracting characteristic information of the DVS pulse signal by using SNN; and predicting first optical flow information of the characteristic information by using the SNN. The invention can improve the time resolution of the optical flow information.

Description

Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to a method and apparatus for predicting optical flow information, an electronic device, and a storage medium.
Background
The optical flow information prediction refers to predicting the motion of an object corresponding to a pixel point in an image by using an image signal acquired by a camera. Plays an important role in the fields of image processing, object segmentation, motion detection, navigation and the like. Currently, optical flow information prediction mainly predicts optical flow information by using an analog discharge rate signal of an image signal, which results in a lower temporal resolution of the predicted optical flow information.
Disclosure of Invention
The embodiment of the invention provides a method, a device, electronic equipment and a storage medium for predicting optical flow information, which are used for solving the problem that the time resolution of the predicted optical flow information is lower.
In a first aspect, an embodiment of the present invention provides a method for predicting optical flow information, including:
acquiring dynamic vision receptor (Dynamic Vision Sensor, DVS) pulse signals;
extracting characteristic information of the DVS pulse signal using a pulse neural network (Spiking Neural Network, SNN);
and predicting first optical flow information of the characteristic information by using the SNN.
After the predicting the first optical flow information of the feature information using the SNN, the method further includes:
and encoding the first optical flow information of the target pixel by using two neurons to obtain encoded information of the first neuron and the second neuron, wherein the first neuron is used for representing the optical flow direction of the target pixel, the second neuron is used for representing the optical flow intensity of the target pixel, and the target pixel is any pixel corresponding to the DVS pulse signal.
Optionally, a first correspondence exists between the optical flow direction represented by the first neuron and a discharge time, and a second correspondence exists between the optical flow intensity represented by the second neuron and the discharge time, where the discharge time is an output time of optical flow information of the target pixel.
Optionally, the extracting the characteristic information of the DVS pulse signal using SNN includes:
inputting the DVS pulse signal to the SNN, updating the neuron information according to the input pulse signal, and extracting the characteristic information of the neuron information.
Optionally, the inputting the DVS pulse signal to the SNN, updating the neuron information according to the input pulse signal, and extracting the characteristic information of the neuron information includes:
inputting the DVS pulse signal to the SNN through a feedforward connection, updating the neuron information in a leakage integration LIF mode according to the input pulse signal, and extracting the characteristic information of the neuron information.
Optionally, the SNN includes a plurality of convolution layers, and extracting the characteristic information of the DVS pulse signal using the SNN includes:
and extracting characteristic information of the DVS pulse signal by using the plurality of convolution layers, wherein the input of the latter convolution layer in the two adjacent convolution layers comprises the output of the former convolution layer.
Optionally, the SNN includes a plurality of deconvolution layers, and the predicting the first optical flow information of the feature information using the SNN includes:
and predicting first optical flow information of the characteristic information by using the deconvolution layers, wherein the input of the later deconvolution layer in the two adjacent deconvolution layers comprises the prediction result of the previous deconvolution layer.
Optionally, the sampling result of the previous deconvolution layer in the two adjacent deconvolution layers is inserted into the prediction result of the next deconvolution layer, wherein the sampling result is obtained by sampling the prediction result of the previous deconvolution layer.
Optionally, the input of at least one deconvolution layer of the plurality of deconvolution layers further comprises characteristic information conveyed by the deconvolution layer.
Optionally, the at least one deconvolution layer includes an i-th deconvolution layer in a first order, and the input of the i-th deconvolution layer includes characteristic information that is transmitted in the SNN in a second order of the i-th convolution layer, wherein the first order is an order from output to input in the SNN, and the second order is an order from input to output in the SNN.
Optionally, the acquiring the DVS pulse signal includes:
acquiring DVS pulse signals output by a camera and APS signals of an activated pixel receptor;
the method further comprises the steps of:
second optical flow information of the APS signal is predicted using an analog neural network ANN.
Optionally, the SNN includes a plurality of convolution layers, and the input of at least one convolution layer of the plurality of convolution layers further includes feature information transferred by the ANN; and/or
The SNN includes a plurality of deconvolution layers, and the input of at least one deconvolution layer of the plurality of deconvolution layers further includes characteristic information conveyed by the ANN.
Optionally, the ANN is matched with the model structure of the SNN, the at least one convolution layer includes an ith convolution layer in a second order, and the input of the ith convolution layer includes characteristic information transferred by the ith convolution layer in the ANN in the second order; and/or
The ANN is matched with the model structure of the SNN, the at least one deconvolution layer comprises an ith deconvolution layer in a second sequence, and the input of the ith deconvolution layer comprises characteristic information transmitted by the ith deconvolution layer in the second sequence in the ANN;
wherein the second order is an order from input to output.
In a second aspect, an embodiment of the present invention provides an optical flow information prediction apparatus, including:
the acquisition module is used for acquiring DVS pulse signals;
the extracting module is used for extracting the characteristic information of the DVS pulse signal by using SNN;
and the first prediction module is used for predicting first optical flow information of the characteristic information by using the SNN.
Optionally, the apparatus further comprises:
the coding module is used for coding first optical flow information of a target pixel by using two neurons to obtain information coded by the first neuron and the second neuron, wherein the first neuron is used for representing the optical flow direction of the target pixel, the second neuron is used for representing the optical flow intensity of the target pixel, and the target pixel is any pixel corresponding to the DVS pulse signal.
Optionally, a first correspondence exists between the optical flow direction represented by the first neuron and a discharge time, and a second correspondence exists between the optical flow intensity represented by the second neuron and the discharge time, where the discharge time is an output time of optical flow information of the target pixel.
Optionally, the extracting module is configured to input the DVS pulse signal to the SNN, update neuron information according to the input pulse signal, and extract characteristic information of the neuron information.
Optionally, the extracting module is configured to input the DVS pulse signal to the SNN through a feed-forward connection, update the neuron information in a leakage integration distribution LIF manner according to the input pulse signal, and extract characteristic information of the neuron information.
Optionally, the SNN includes a plurality of convolution layers, and the extracting module is configured to extract characteristic information of the DVS pulse signal using the plurality of convolution layers, where an input of a later one of two adjacent convolution layers includes an output of a previous one of the two adjacent convolution layers.
Optionally, the SNN includes a plurality of deconvolution layers, and the first prediction module is configured to predict the first optical flow information of the feature information using the plurality of deconvolution layers, where an input of a later deconvolution layer of two adjacent deconvolution layers includes a prediction result of a previous deconvolution layer.
Optionally, the sampling result of the previous deconvolution layer in the two adjacent deconvolution layers is inserted into the prediction result of the next deconvolution layer, wherein the sampling result is obtained by sampling the prediction result of the previous deconvolution layer.
Optionally, the input of at least one deconvolution layer of the plurality of deconvolution layers further comprises characteristic information conveyed by the deconvolution layer.
Optionally, the at least one deconvolution layer includes an i-th deconvolution layer in a first order, and the input of the i-th deconvolution layer includes characteristic information that is transmitted in the SNN in a second order of the i-th convolution layer, wherein the first order is an order from output to input in the SNN, and the second order is an order from input to output in the SNN.
Optionally, the acquisition module is used for acquiring a DVS pulse signal output by the camera and an active pixel receptor APS signal;
the apparatus further comprises:
and the second prediction module is used for predicting second optical flow information of the APS signal by using the analog neural network ANN.
Optionally, the SNN includes a plurality of convolution layers, and the input of at least one convolution layer of the plurality of convolution layers further includes feature information transferred by the ANN; and/or
The SNN includes a plurality of deconvolution layers, and the input of at least one deconvolution layer of the plurality of deconvolution layers further includes characteristic information conveyed by the ANN.
Optionally, the ANN is matched with the model structure of the SNN, the at least one convolution layer includes an ith convolution layer in a second order, and the input of the ith convolution layer includes characteristic information transferred by the ith convolution layer in the ANN in the second order; and/or
The ANN is matched with the model structure of the SNN, the at least one deconvolution layer comprises an ith deconvolution layer in a second sequence, and the input of the ith deconvolution layer comprises characteristic information transmitted by the ith deconvolution layer in the second sequence in the ANN;
wherein the second order is an order from input to output.
In a third aspect, an embodiment of the present invention provides an electronic device, including: the optical flow information prediction method comprises a memory, a processor and a program or an instruction stored in the memory and capable of running on the processor, wherein the program or the instruction realizes the steps in the optical flow information prediction method provided by the embodiment of the invention when being executed by the processor.
In a fourth aspect, an embodiment of the present invention provides a readable storage medium having stored thereon a program or instructions that, when executed by a processor, implement the steps in the optical flow information prediction method provided by the embodiment of the present invention.
In the embodiment of the invention, DVS pulse signals are acquired; extracting characteristic information of the DVS pulse signal by using SNN; and predicting first optical flow information of the characteristic information by using the SNN. The characteristic information of the DVS pulse signals is extracted to predict, so that the extremely high time resolution characteristic of the DVS pulse signals is utilized, and the time resolution of optical flow information is improved.
Drawings
FIG. 1 is a flowchart of an optical flow information prediction method according to an embodiment of the present invention;
FIG. 2 is a schematic illustration of a neuron according to an embodiment of the present invention;
FIG. 3 is a schematic diagram of an SNN according to an embodiment of the present invention;
FIG. 4 is a schematic diagram of an ANN and SNN according to an embodiment of the present invention;
FIG. 5 is a block diagram of an optical flow information prediction apparatus according to an embodiment of the present invention;
fig. 6 is a block diagram of an electronic device according to an embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are some, but not all, of the embodiments of the present application. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
The terms first, second and the like in the description and in the claims, are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate, such that embodiments of the present application may be implemented in sequences other than those illustrated or described herein, and that the "first" and "second" distinguished objects generally are of the type and do not limit the number of objects, e.g., the first object may be one or more.
Referring to fig. 1, fig. 1 is a flowchart of an optical flow information prediction method provided by an embodiment of the present invention, as shown in fig. 1, including the following steps:
step 101, acquiring a DVS pulse signal.
The DVS pulse signal may be an image signal (may also be referred to as Event information) acquired by a DVS, and the DVS may also be referred to as an Event Camera (Event Camera). The DVS is different from the traditional camera in that the shutter is used for controlling the frame rate that all pixels record light intensity according to frames, the DVS is sensitive to the light intensity change rate, each pixel independently records the change of the light intensity logarithmic value at the pixel, and when the change exceeds a threshold value, a positive or negative pulse is generated. Due to the asynchronous nature of DVS, it is not fast-thresholded, with extremely high temporal resolution, such as some DVS frame rates of about 1,000,000fps.
Alternatively, the DVS pulse signal may be a DVS pulse signal acquired by a camera that combines an active pixel receptor (Active Pixel Sensor, APS) with DVS, where the camera that combines APS with DVS can record both single frame images and DVS pulse signals, for example: DAViS camera.
The DVS pulse signal may be a continuous pulse signal, for example: the DVS acquires a continuous signal.
And 102, extracting characteristic information of the DVS pulse signal by using SNN.
The SNN may be a pre-trained network model.
The extracting of the feature information of the DVS pulse signal using the SNN may be extracting of the feature information of the DVS pulse signal using a convolution layer in the SNN. In addition, extracting the feature information may also be referred to as encoding the DVS pulse signal.
Of course, in the embodiment of the present invention, the method is not limited to extracting the characteristic information of the DVS pulse signal by using the convolution layer in the SNN, for example: other pre-trained network layers can be adopted to extract the characteristic information of the DVS pulse signals, and the characteristic information can be specifically determined in the process of model creation and training.
Step 103, predicting first optical flow information of the feature information by using the SNN.
The first optical flow information for predicting the feature information using the SNN may be first optical flow information for predicting the feature information using a deconvolution layer of the SNN. In addition, predicting the first optical flow information of the feature information may also be referred to as decoding the feature information.
Of course, in the embodiment of the present invention, the use of the deconvolution layer in the SNN to predict the optical flow information is not limited, for example: other network layers of pre-training may also be employed to predict optical flow information, particularly as may be determined during model creation and training.
In addition, the first optical flow information may include optical flow information of one or more pixels, and may specifically represent motion information of one or more pixels corresponding to an object in space.
The first optical flow information of the feature information may be also referred to as optical flow information of the DVS pulse signal.
In the embodiment of the invention, the SNN can be used for extracting the characteristic information of the DVS pulse signal and predicting the first optical flow information of the characteristic information, so that the extremely high time resolution characteristic of the DVS pulse signal is utilized, and the time resolution of the optical flow information is improved. Further, the discreteness and sparsity of the DVS pulse signals are utilized, so that the processing efficiency of predicting optical flow information is improved.
It should be noted that the embodiment of the present application may be applied to an electronic device, and the electronic device may have a capability of collecting DVS pulse signals, for example: including DVS or DAViS cameras, although this is not limiting, such as: the electronic device may be configured to receive DVS pulse signals transmitted by other devices. The electronic device may be, in particular, an electronic device such as a mobile phone, a monitoring device, an image capturing device, or a computer, which is not limited thereto.
As an alternative embodiment, after the predicting the first optical flow information of the feature information using the SNN, the method further includes:
and encoding the first optical flow information of the target pixel by using two neurons to obtain encoded information of the first neuron and the second neuron, wherein the first neuron is used for representing the optical flow direction of the target pixel, the second neuron is used for representing the optical flow intensity of the target pixel, and the target pixel is any pixel corresponding to the DVS pulse signal.
The encoding of the first optical flow information of the target pixel using the two neurons may be converting the first optical flow information of the target pixel into information encoded by the two neurons to obtain two neurons for representing the optical flow direction of the target pixel and for representing the optical flow intensity of the target pixel.
In addition, since the target pixel is any pixel corresponding to the DVS pulse signal, it is possible to realize that the optical flow information of each pixel is represented by two neurons.
In this embodiment, since two neurons of the pixel point for representing the optical flow direction of the target pixel and for representing the optical flow intensity of the target pixel can be obtained, the discrete type of the DVS pulse signal and the sparsity of the optical flow can be fully utilized, and the processing efficiency and the time resolution of the optical flow estimation can be improved.
Optionally, a first correspondence exists between the optical flow direction represented by the first neuron and a discharge time, and a second correspondence exists between the optical flow intensity represented by the second neuron and the discharge time, where the discharge time is an output time of optical flow information of the target pixel.
In this embodiment, it is possible to realize determination of the optical flow direction and the optical flow intensity from the discharge time. In addition, the first correspondence relationship and the second correspondence relationship may be predefined.
For example: taking a specific period T as an example, as shown in fig. 2, the upper neuron 201 represents the optical flow direction θ, and the lower neuron 202 represents the optical flow intensity r, where the relationship between the discharge time T and the optical flow direction θ is:
the relationship between the discharge time t and the optical flow intensity r (pixel) is:
wherein, alpha and beta are constants.
It should be noted that the above two formulas are merely examples, and the embodiments of the present invention are not limited thereto, for example: constants may be added or subtracted from the above-described formulas of the optical flow intensity and the optical flow direction, and may be specifically set according to the actual scene of the application. The first correspondence relationship and the second correspondence relationship are not limited to be determined by a formula, and may be, for example: the mapping table of the optical flow direction and the discharge time can also be established, and the mapping table of the optical flow intensity and the discharge time can be established, and the optical flow direction and the optical flow intensity can be determined through table lookup.
Further, the discharge time T may be a time position of the specific period T, and the discharge time T is not limited to this, for example: or a specific time of outputting the optical flow information.
In addition, in the embodiment of the present invention, after determining the optical flow direction and the optical flow intensity, components u and v in the optical flow x, y directions may be determined, for example: can be obtained from u=r cos (θ), v=r sin (θ), respectively.
As an alternative embodiment, the extracting the characteristic information of the DVS pulse signal using the SNN includes:
inputting the DVS pulse signal to the SNN, updating the neuron information according to the input pulse signal, and extracting the characteristic information of the neuron information.
The inputting of the DVS pulse signal to the SNN may be a continuous DVS pulse signal to the SNN. The updating of the neuron information according to the input pulse signal may be updating of the neuron information in the SNN according to the input pulse signal to convert the input pulse signal into the neuron information.
In this embodiment, since the neuron information is updated according to the input pulse signal, and the characteristic information of the neuron information is extracted, the discrete type of the DVS pulse signal and the sparsity of the optical flow can be fully utilized, and the processing efficiency and the time resolution of the optical flow estimation can be improved.
Further, in this embodiment, in combination with the above-described encoding of the first optical flow information of the target pixel using two neurons, a neuron input-neuron output (spike-in-spike-out) processing mode in which all the SNN networks are used from input to output may be implemented, so as to further improve the processing efficiency and the time resolution of optical flow estimation.
Optionally, the inputting the DVS pulse signal to the SNN, updating the neuron information according to the input pulse signal, and extracting the characteristic information of the neuron information includes:
inputting the DVS pulse signal to the SNN through a feed-forward connection, updating the neuron information according to the input pulse signal in a leakage integration distribution (Leaky Integrate and Fire, LIF) mode, and extracting characteristic information of the neuron information.
The inputting the DVS pulse signal to the SNN through the feedforward connection and updating the neuron information in the LIF manner according to the input pulse signal may be that the DVS pulse signal is continuously input through the feedforward connection and updates the neuron information in the LIF manner.
It should be noted that, in the embodiment of the present invention, updating the neuron information in the LIF manner is not limited, for example: the updating of the neuron state may be performed by other methods, which is not limited thereto.
As an alternative embodiment, the SNN includes a plurality of convolution layers, and the extracting the characteristic information of the DVS pulse signal using the SNN includes:
and extracting characteristic information of the DVS pulse signal by using the plurality of convolution layers, wherein the input of the latter convolution layer in the two adjacent convolution layers comprises the output of the former convolution layer.
As shown in fig. 3, the above-mentioned multiple convolution layers may be used, where the first convolution layer receives the input DVS pulse signal 301 or updated neuron information, and then extracts the received DVS pulse signal or updated neuron information through the multiple convolution layers.
It should be noted that, the convolutional layer may also be referred to as a convolutional layer for extracting the feature information of the DVS pulse signal, and encoding the DVS pulse signal to obtain the feature information. And each convolutional layer may also be referred to as an encoder.
In addition, in the case of updating the neuron information according to the pulse signal, the characteristic information of the DVS pulse signal may be extracted using the plurality of convolution layers.
In this embodiment, since the characteristic information of the DVS pulse signal is extracted using the plurality of convolution layers, the accuracy of the characteristic information can be improved.
Of course, in the embodiment of the present invention, the method is not limited to extracting the characteristic information of the DVS pulse signal by using a plurality of convolution layers, for example: in some scenarios, feature information may be extracted using one convolutional layer or using convolutional layers and other neural network layers.
As an alternative embodiment, the SNN includes a plurality of deconvolution layers, and the predicting the first optical flow information of the feature information using the SNN includes:
and predicting first optical flow information of the characteristic information by using the deconvolution layers, wherein the input of the later deconvolution layer in the two adjacent deconvolution layers comprises the prediction result of the previous deconvolution layer.
The above-mentioned deconvolution layers can be shown in fig. 3, where the prediction result of the previous deconvolution layer is input to the next deconvolution layer, so as to implement one-time prediction of the optical flow by each deconvolution layer, thereby improving the accuracy of optical flow information prediction.
Note that, the optical flow information of the deconvolution layer prediction feature information may also be referred to as deconvolution layer decoding the feature information to obtain the optical flow information. And each deconvolution layer may also be referred to as a decoder.
Of course, in the embodiment of the present invention, the prediction of the first optical flow information using a plurality of deconvolution layers is not limited, for example: in some scenarios, the first optical flow information may be predicted using one deconvolution layer or using a combination of deconvolution layers and other neural network layers.
Optionally, the sampling result of the previous deconvolution layer in the two adjacent deconvolution layers is inserted into the prediction result of the next deconvolution layer, wherein the sampling result is obtained by sampling the prediction result of the previous deconvolution layer.
In this embodiment, it may be implemented to insert the sampling result of the previous deconvolution layer into the prediction result of the next deconvolution layer, for example: such as the information corresponding to the front header 303 shown on the right side of fig. 3. Thus, the sampling result of the previous deconvolution layer is inserted into the prediction result of the next deconvolution layer, so that the accuracy of the prediction result can be further improved.
Optionally, the input of at least one deconvolution layer of the plurality of deconvolution layers further includes characteristic information conveyed by the deconvolution layer.
In this embodiment, the deconvolution layer may further include the feature information of the lower layer transferred by the convolution layer, besides the information submitted by the upper layer, so that the input of the deconvolution layer is richer, and the prediction result is further improved.
Optionally, the at least one deconvolution layer includes an i-th deconvolution layer in a first order, and the characteristic information included in the input of the i-th deconvolution layer is characteristic information transmitted in a second order of the i-th convolution layer in the SNN, wherein the first order is an order from output to input in the SNN, and the second order is an order from input to output in the SNN.
The first order may be understood as a reverse order, for example, as shown in fig. 3, where the 1 st deconvolution layer receives the characteristic information 302 transmitted by the 1 st convolution layer, the 2 nd deconvolution layer receives the characteristic information transmitted by the 2 nd convolution layer, and the 3 rd deconvolution layer receives the characteristic information transmitted by the 3 rd convolution layer, that is, the 1 st deconvolution layer, the 2 nd deconvolution layer, and the 3 rd deconvolution layer are arranged in the order from output to input in SNN. Therefore, the deconvolution layer can obtain the characteristic information of a lower layer, so that the accuracy of the prediction result is further improved.
Of course, in the embodiment of the present invention, the above-mentioned characteristic information transfer relationship is not limited, for example: it is also possible that the 1 st convolution layer is passed to the 1 st deconvolution layer, the 2 nd convolution layer is passed to the 2 nd deconvolution layer, etc.
As an alternative embodiment, the acquiring the DVS pulse signal includes:
acquiring DVS pulse signals and APS signals output by a camera;
the method further comprises the steps of:
second optical flow information of the APS signal is predicted using ANN.
The camera may be a combination of DVS and APS, and the output of the camera includes both DVS pulse signals and conventional APS signals.
The ANN may be trained in advance, and is not limited in the embodiment of the present invention, and is used for predicting the network model of the optical flow information of the APS signal.
In this embodiment, it is possible to realize both the first optical flow information that can predict DVS pulse signals and the second optical flow information that can predict APS signals, thereby improving the prediction capability of the optical flow information.
As an optional implementation manner, the input of at least one convolution layer of the plurality of convolution layers in the SNN further includes feature information transferred by the ANN; and/or
The input of at least one deconvolution layer of the plurality of deconvolution layers in the SNN further includes characteristic information conveyed by the ANN.
The characteristic information transmitted by the ANN, which is further included in the input of the convolution layer, may be characteristic information of an APS signal extracted by the convolution layer in the ANN model, and the characteristic information transmitted by the ANN, which is further included in the input of the deconvolution layer, may be characteristic information of deconvolution layer prediction of the ANN model.
In this embodiment, since the input of at least one convolution layer further includes the feature information transferred by the ANN, and/or the input of at least one deconvolution layer further includes the feature information transferred by the ANN, prediction is performed by using the combined feature information, so that accuracy of optical flow information prediction may be further improved. And the high spatial resolution of the APS signal can be further utilized to further improve the calculation accuracy of the optical flow information.
Optionally, the ANN is matched with the model structure of the SNN, the at least one convolution layer includes an ith convolution layer in a second order, and the input of the ith convolution layer includes feature information transferred by the ith convolution layer in the second order in the ANN; and/or
The ANN is matched with the model structure of the SNN, the at least one deconvolution layer comprises an ith deconvolution layer in a second sequence, and the input of the ith deconvolution layer comprises characteristic information transmitted by the ith deconvolution layer in the second sequence in the ANN;
wherein the second order is an order from input to output.
In this embodiment, as shown in fig. 4, SNN may be located below the upper ANN in fig. 4, 401 and 402 may represent an APS signal and a DVS pulse signal, feature information may be transferred from the 1 st convolution layer in the ANN to the 1 st convolution layer in the SNN in fig. 4, feature information may be transferred from the 2 nd convolution layer to the 2 nd convolution layer in the SNN, …, feature information may be transferred from the 1 st deconvolution layer in the ANN to the 1 st deconvolution layer in the SNN, and feature information may be transferred from the 2 nd deconvolution layer to the 2 nd deconvolution layer in the SNN, specifically, feature information corresponding to an arrow represented by 403 in fig. 4 may be transferred. Of course, fig. 4 is merely an illustration.
Of course, in the embodiment of the present invention, the feature information is not limited to be transferred through the above sequence, for example: it may also be that the 2 nd convolution layer in the ANN conveys the characteristic information to the 1 st convolution layer in the SNN, the 3 rd convolution layer conveys the characteristic information to the 2 nd convolution layer in the SNN, and so on.
In the embodiment of the invention, DVS pulse signals are acquired; extracting characteristic information of the DVS pulse signal by using SNN; and predicting first optical flow information of the characteristic information by using the SNN. The characteristic information of the DVS pulse signals is extracted to predict, so that the extremely high time resolution characteristic of the DVS pulse signals is utilized, and the time resolution of optical flow information is improved.
Referring to fig. 5, fig. 5 is a block diagram of an optical flow information prediction apparatus according to an embodiment of the present invention, and as shown in fig. 5, an optical flow information prediction apparatus 500 includes:
an acquisition module 501 for acquiring dynamic vision receptor DVS pulse signals;
an extracting module 502, configured to extract feature information of the DVS pulse signal using a pulse neural network SNN;
a first prediction module 503, configured to predict first optical flow information of the feature information using the SNN.
Optionally, the apparatus further includes:
The coding module is used for coding first optical flow information of a target pixel by using two neurons to obtain information coded by the first neuron and the second neuron, wherein the first neuron is used for representing the optical flow direction of the target pixel, the second neuron is used for representing the optical flow intensity of the target pixel, and the target pixel is any pixel corresponding to the DVS pulse signal.
Optionally, a first correspondence exists between the optical flow direction represented by the first neuron and a discharge time, and a second correspondence exists between the optical flow intensity represented by the second neuron and the discharge time, where the discharge time is an output time of optical flow information of the target pixel.
Optionally, the extracting module is configured to input the DVS pulse signal to the SNN, update neuron information according to the input pulse signal, and extract feature information of the neuron information.
Optionally, the extracting module is configured to input the DVS pulse signal to the SNN through a feed-forward connection, update the neuron information in a leakage integration release LIF manner according to the input pulse signal, and extract characteristic information of the neuron information.
Optionally, the SNN includes a plurality of convolution layers, and the extracting module is configured to extract characteristic information of the DVS pulse signal using the plurality of convolution layers, where an input of a later one of two adjacent convolution layers includes an output of a previous one of the two adjacent convolution layers.
Optionally, the SNN includes a plurality of deconvolution layers, and the first prediction module is configured to predict the first optical flow information of the feature information using the plurality of deconvolution layers, where an input of a later deconvolution layer of two adjacent deconvolution layers includes a prediction result of a previous deconvolution layer.
Optionally, the sampling result of the previous deconvolution layer in the two adjacent deconvolution layers is inserted into the prediction result of the next deconvolution layer, wherein the sampling result is obtained by sampling the prediction result of the previous deconvolution layer.
Optionally, the input of at least one deconvolution layer of the plurality of deconvolution layers further includes characteristic information conveyed by the deconvolution layer.
Optionally, the at least one deconvolution layer includes an i-th deconvolution layer in a first order, and the characteristic information included in the input of the i-th deconvolution layer is characteristic information transmitted in a second order of the i-th convolution layer in the SNN, wherein the first order is an order from output to input in the SNN, and the second order is an order from input to output in the SNN.
Optionally, the acquiring module is configured to acquire a DVS pulse signal output by the camera and an active pixel receptor APS signal;
the apparatus further comprises:
and the second prediction module is used for predicting second optical flow information of the APS signal by using the analog neural network ANN.
Optionally, the SNN includes a plurality of convolution layers, and the input of at least one convolution layer of the plurality of convolution layers further includes feature information transferred by the ANN; and/or
The SNN includes a plurality of deconvolution layers, and the input of at least one deconvolution layer of the plurality of deconvolution layers further includes characteristic information conveyed by the ANN.
Optionally, the ANN is matched with the model structure of the SNN, the at least one convolution layer includes an ith convolution layer in a second order, and the input of the ith convolution layer includes feature information transferred by the ith convolution layer in the second order in the ANN; and/or
The ANN is matched with the model structure of the SNN, the at least one deconvolution layer comprises an ith deconvolution layer in a second sequence, and the input of the ith deconvolution layer comprises characteristic information transmitted by the ith deconvolution layer in the second sequence in the ANN;
wherein the second order is an order from input to output.
The optical flow information prediction device provided by the embodiment of the present invention can implement each process in the method embodiment of fig. 1, and in order to avoid repetition, a description thereof will not be repeated here.
It should be noted that, the optical flow information prediction apparatus in the embodiment of the present invention may be an apparatus, or may be a component, an integrated circuit, or a chip in an electronic device.
Referring to fig. 6, fig. 6 is a block diagram of an electronic device according to an embodiment of the present invention, and as shown in fig. 6, an electronic device 600 includes: the optical flow information prediction method comprises a memory 601, a processor 602 and a program or an instruction stored in the memory 601 and capable of being executed on the processor 602, wherein the program or the instruction realizes the steps in the optical flow information prediction method when being executed by the processor 602.
The embodiment of the invention also provides a readable storage medium, on which a program or an instruction is stored, which when executed by a processor, implements each process of the optical flow information prediction method embodiment described above, and can achieve the same technical effects, so that repetition is avoided, and no further description is given here.
It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Furthermore, it should be noted that the scope of the methods and apparatus in the embodiments of the present application is not limited to performing the functions in the order shown or discussed, but may also include performing the functions in a substantially simultaneous manner or in an opposite order depending on the functions involved, e.g., the described methods may be performed in an order different from that described, and various steps may also be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.
From the above description of the embodiments, it will be clear to those skilled in the art that the above-described embodiment method may be implemented by means of software plus a necessary general hardware platform, but of course may also be implemented by means of hardware, but in many cases the former is a preferred embodiment. Based on such understanding, the technical solution of the present application may be embodied essentially or in a part contributing to the prior art in the form of a software product stored in a storage medium (such as ROM/RAM, magnetic disk, optical disk), including several instructions for causing a terminal (which may be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to perform the method described in the embodiments of the present application.
The embodiments of the present application have been described above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, which are merely illustrative and not restrictive, and many forms may be made by those of ordinary skill in the art without departing from the spirit of the present application and the scope of the claims, which are also within the protection of the present application.

Claims (26)

1. An optical flow information prediction method, comprising:
acquiring a dynamic vision receptor DVS pulse signal;
extracting characteristic information of the DVS pulse signal by using a pulse neural network SNN;
predicting first optical flow information of the feature information using the SNN;
after the predicting the first optical flow information of the feature information using the SNN, the method further includes:
and encoding the first optical flow information of the target pixel by using two neurons to obtain encoded information of the first neuron and the second neuron, wherein the first neuron is used for representing the optical flow direction of the target pixel, the second neuron is used for representing the optical flow intensity of the target pixel, and the target pixel is any pixel corresponding to the DVS pulse signal.
2. The method of claim 1, wherein a first correspondence exists between the optical flow direction represented by the first neuron and a discharge time, and a second correspondence exists between the optical flow intensity represented by the second neuron and the discharge time, wherein the discharge time is an output time of optical flow information of the target pixel.
3. The method of claim 1, wherein extracting the feature information of the DVS pulse signal using the SNN comprises:
inputting the DVS pulse signal to the SNN, updating the neuron information according to the input pulse signal, and extracting the characteristic information of the neuron information.
4. The method of claim 3, wherein the inputting the DVS pulse signal to the SNN and updating the neuron information according to the input pulse signal, extracting characteristic information of the neuron information, comprises:
inputting the DVS pulse signal to the SNN through a feedforward connection, updating the neuron information in a leakage integration LIF mode according to the input pulse signal, and extracting the characteristic information of the neuron information.
5. The method of claim 1, wherein the SNN includes a plurality of convolution layers, the extracting feature information of the DVS pulse signal using the SNN includes:
and extracting characteristic information of the DVS pulse signal by using the plurality of convolution layers, wherein the input of the latter convolution layer in the two adjacent convolution layers comprises the output of the former convolution layer.
6. The method of claim 5, wherein the SNN includes a plurality of deconvolution layers, the predicting the first optical flow information of the feature information using the SNN comprising:
And predicting first optical flow information of the characteristic information by using the deconvolution layers, wherein the input of the later deconvolution layer in the two adjacent deconvolution layers comprises the prediction result of the previous deconvolution layer.
7. The method of claim 6, wherein the sampling result of a previous deconvolution layer of the two adjacent deconvolution layers is inserted into the prediction result of a subsequent deconvolution layer, wherein the sampling result is a result of sampling the prediction result of the previous deconvolution layer.
8. The method of claim 6, wherein the input of at least one deconvolution layer of the plurality of deconvolution layers further comprises characteristic information conveyed by the deconvolution layer.
9. The method of claim 8, wherein the at least one deconvolution layer comprises an ith deconvolution layer in a first order, an input of the ith deconvolution layer comprising characteristic information that is transmitted in the SNN at an ith convolutional layer in a second order, wherein the first order is an order from output to input in the SNN, and the second order is an order from input to output in the SNN.
10. The method of any of claims 1 to 9, wherein the acquiring DVS pulse signals comprises:
Acquiring DVS pulse signals output by a camera and APS signals of an activated pixel receptor;
the method further comprises the steps of:
second optical flow information of the APS signal is predicted using an analog neural network ANN.
11. The method of claim 10, wherein the SNN comprises a plurality of convolutional layers, an input of at least one of the plurality of convolutional layers further comprising characteristic information conveyed by the ANN; and/or
The SNN includes a plurality of deconvolution layers, and the input of at least one deconvolution layer of the plurality of deconvolution layers further includes characteristic information conveyed by the ANN.
12. The method of claim 11, wherein the ANN matches a model structure of the SNN, the at least one convolution layer comprising an ith convolution layer in a second order, an input of the ith convolution layer comprising characteristic information conveyed in the ANN by the ith convolution layer in the second order; and/or
The ANN is matched with the model structure of the SNN, the at least one deconvolution layer comprises an ith deconvolution layer in a second sequence, and the input of the ith deconvolution layer comprises characteristic information transmitted by the ith deconvolution layer in the second sequence in the ANN;
wherein the second order is an order from input to output.
13. An optical flow information prediction apparatus, comprising:
the acquisition module is used for acquiring dynamic vision receptor DVS pulse signals;
the extraction module is used for extracting the characteristic information of the DVS pulse signal by using the pulse neural network SNN;
a first prediction module for predicting first optical flow information of the feature information using the SNN;
the apparatus further comprises:
the coding module is used for coding first optical flow information of a target pixel by using two neurons to obtain information coded by the first neuron and the second neuron, wherein the first neuron is used for representing the optical flow direction of the target pixel, the second neuron is used for representing the optical flow intensity of the target pixel, and the target pixel is any pixel corresponding to the DVS pulse signal.
14. The apparatus of claim 13, wherein a first correspondence exists between the optical flow direction represented by the first neuron and a discharge time, and a second correspondence exists between the optical flow intensity represented by the second neuron and the discharge time, wherein the discharge time is an output time of optical flow information of the target pixel.
15. The apparatus of claim 13, wherein the extraction module is configured to input the DVS pulse signal to the SNN, and to update neuron information based on the input pulse signal to extract characteristic information of the neuron information.
16. The apparatus of claim 15, wherein the extraction module is configured to input the DVS pulse signal to the SNN via a feed-forward connection, and update neuron information in a leaky integrate-and-dispense LIF manner based on the input pulse signal, and extract characteristic information of the neuron information.
17. The apparatus of claim 13, wherein the SNN comprises a plurality of convolution layers, the extraction module to extract characteristic information of the DVS pulse signal using the plurality of convolution layers, wherein an input of a subsequent one of two adjacent convolution layers comprises an output of a previous one of the convolution layers.
18. The apparatus of claim 17, wherein the SNN comprises a plurality of deconvolution layers, the first prediction module to predict first optical flow information for the feature information using the plurality of deconvolution layers, wherein an input of a subsequent deconvolution layer of two adjacent deconvolution layers comprises a prediction result of a previous deconvolution layer.
19. The apparatus of claim 18, wherein a sample of a previous deconvolution layer of two adjacent deconvolution layers is inserted into a prediction of a subsequent deconvolution layer, wherein the sample is a result of sampling the prediction of the previous deconvolution layer.
20. The apparatus of claim 19, wherein the input of at least one deconvolution layer of the plurality of deconvolution layers further comprises characteristic information conveyed by the deconvolution layer.
21. The apparatus of claim 20, wherein the at least one deconvolution layer comprises an ith deconvolution layer in a first order, an input of the ith deconvolution layer comprising characteristic information that is transmitted in the SNN at an ith convolutional layer in a second order, wherein the first order is an order from output to input in the SNN, and the second order is an order from input to output in the SNN.
22. The apparatus of any one of claims 13 to 21, wherein the acquisition module is configured to acquire a DVS pulse signal output by a camera and an active pixel receptor APS signal;
the apparatus further comprises:
and the second prediction module is used for predicting second optical flow information of the APS signal by using the analog neural network ANN.
23. The apparatus of claim 22, wherein the SNN comprises a plurality of convolutional layers, an input of at least one of the plurality of convolutional layers further comprising characteristic information conveyed by the ANN; and/or
The SNN includes a plurality of deconvolution layers, and the input of at least one deconvolution layer of the plurality of deconvolution layers further includes characteristic information conveyed by the ANN.
24. The apparatus of claim 23, wherein the ANN matches a model structure of the SNN, the at least one convolution layer comprising an ith convolution layer in a second order, an input of the ith convolution layer comprising characteristic information conveyed in the ANN by the ith convolution layer in the second order; and/or
The ANN is matched with the model structure of the SNN, the at least one deconvolution layer comprises an ith deconvolution layer in a second sequence, and the input of the ith deconvolution layer comprises characteristic information transmitted by the ith deconvolution layer in the second sequence in the ANN;
wherein the second order is an order from input to output.
25. An electronic device, comprising: memory, a processor, and a program or instructions stored on the memory and executable on the processor, which when executed by the processor, implement the steps in the optical flow information prediction method according to any one of claims 1 to 12.
26. A readable storage medium, wherein a program or instructions is stored on the readable storage medium, which when executed by a processor, implements the steps in the optical flow information prediction method according to any one of claims 1 to 12.
CN202010909623.6A 2020-09-02 2020-09-02 Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium Active CN112085768B (en)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN202010909623.6A CN112085768B (en) 2020-09-02 2020-09-02 Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium
TW110132594A TW202211163A (en) 2020-09-02 2021-09-02 Method and device for optical flow information prediction, electronic device, and storage medium
PCT/CN2021/116133 WO2022048582A1 (en) 2020-09-02 2021-09-02 Method and device for optical flow information prediction, electronic device, and storage medium

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202010909623.6A CN112085768B (en) 2020-09-02 2020-09-02 Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium

Publications (2)

Publication Number Publication Date
CN112085768A CN112085768A (en) 2020-12-15
CN112085768B true CN112085768B (en) 2023-12-26

Family

ID=73731791

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202010909623.6A Active CN112085768B (en) 2020-09-02 2020-09-02 Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium

Country Status (3)

Country Link
CN (1) CN112085768B (en)
TW (1) TW202211163A (en)
WO (1) WO2022048582A1 (en)

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112085768B (en) * 2020-09-02 2023-12-26 北京灵汐科技有限公司 Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium
CN112702588B (en) * 2020-12-21 2023-04-07 北京灵汐科技有限公司 Dual-mode image signal processor and dual-mode image signal processing system
CN116109489A (en) * 2021-11-09 2023-05-12 华为技术有限公司 Denoising method and related equipment
CN114581490B (en) * 2022-04-29 2022-09-20 北京大学 Scene setting method and device, control equipment and readable storage medium

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011248704A (en) * 2010-05-28 2011-12-08 Nippon Telegr & Teleph Corp <Ntt> Optical flow estimation device, optical flow estimation method and optical flow estimation program
CN109903315A (en) * 2019-03-08 2019-06-18 腾讯科技（深圳）有限公司 Method, apparatus, equipment and readable storage medium storing program for executing for light stream prediction
CN110210563A (en) * 2019-06-04 2019-09-06 北京大学 The study of pattern pulse data space time information and recognition methods based on Spike cube SNN
CN111127522A (en) * 2019-12-30 2020-05-08 亮风台（上海）信息科技有限公司 Monocular camera-based depth optical flow prediction method, device, equipment and medium
CN111340081A (en) * 2020-02-19 2020-06-26 中国科学院光电技术研究所 Optical flow estimation-based neural network aberration prediction method
WO2020155741A1 (en) * 2019-01-29 2020-08-06 清华大学 Fusion structure and method of convolutional neural network and pulse neural network

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110555861B (en) * 2019-08-09 2023-04-25 北京字节跳动网络技术有限公司 Optical flow calculation method and device and electronic equipment
CN112085768B (en) * 2020-09-02 2023-12-26 北京灵汐科技有限公司 Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011248704A (en) * 2010-05-28 2011-12-08 Nippon Telegr & Teleph Corp <Ntt> Optical flow estimation device, optical flow estimation method and optical flow estimation program
WO2020155741A1 (en) * 2019-01-29 2020-08-06 清华大学 Fusion structure and method of convolutional neural network and pulse neural network
CN109903315A (en) * 2019-03-08 2019-06-18 腾讯科技（深圳）有限公司 Method, apparatus, equipment and readable storage medium storing program for executing for light stream prediction
CN110210563A (en) * 2019-06-04 2019-09-06 北京大学 The study of pattern pulse data space time information and recognition methods based on Spike cube SNN
CN111127522A (en) * 2019-12-30 2020-05-08 亮风台（上海）信息科技有限公司 Monocular camera-based depth optical flow prediction method, device, equipment and medium
CN111340081A (en) * 2020-02-19 2020-06-26 中国科学院光电技术研究所 Optical flow estimation-based neural network aberration prediction method

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
Chankyu Lee 等.Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks.. arXiv:2003.06696v1.2020,第1-16页. *
Spike-FlowNet: Event-based Optical Flow Estimation with Energy-Efficient Hybrid Neural Networks.;Chankyu Lee 等;arXiv:2003.06696v1;第1-16页 *

Also Published As

Publication number Publication date
CN112085768A (en) 2020-12-15
TW202211163A (en) 2022-03-16
WO2022048582A1 (en) 2022-03-10

Similar Documents

Publication Publication Date Title
CN112085768B (en) Optical flow information prediction method, optical flow information prediction device, electronic equipment and storage medium
CN110569695B (en) Image processing method and device based on loss assessment image judgment model
CN109636754B (en) Extremely-low-illumination image enhancement method based on generation countermeasure network
CN110110601B (en) Video pedestrian re-recognition method and device based on multi-time space attention model
CN112464807A (en) Video motion recognition method and device, electronic equipment and storage medium
CN110580428A (en) image processing method, image processing device, computer-readable storage medium and electronic equipment
CN111292264A (en) Image high dynamic range reconstruction method based on deep learning
CN110650291B (en) Target focus tracking method and device, electronic equipment and computer readable storage medium
CN112183166B (en) Method and device for determining training samples and electronic equipment
CN108804658B (en) Image processing method and device, storage medium and electronic equipment
WO2014201971A1 (en) Object detection method and device for online training
KR20170038144A (en) Attention detection apparatus and attention detection method
CN112200057A (en) Face living body detection method and device, electronic equipment and storage medium
US20230104196A1 (en) Device and a Method for Processing Data Sequences Using a Convolutional Neural Network
Milford et al. Long exposure localization in darkness using consumer cameras
WO2023005760A1 (en) Systems and methods for performing computer vision task using sequence of frames
CN112766028B (en) Face fuzzy processing method and device, electronic equipment and storage medium
CN110956063A (en) Image processing method, device, equipment and storage medium
CN114266952B (en) Real-time semantic segmentation method based on deep supervision
CN114255177B (en) Exposure control method, device, equipment and storage medium in imaging
CN112668504A (en) Action recognition method and device and electronic equipment
CN113691731B (en) Processing method and device and electronic equipment
WO2022265321A1 (en) Methods and systems for low light media enhancement
CN111144312B (en) Image processing method, device, equipment and medium
EP4248657A1 (en) Methods and systems for low light media enhancement

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant