WO2022165873A1 - 一种仿视网膜中央凹与外周的联合采样方法及装置 - Google Patents

一种仿视网膜中央凹与外周的联合采样方法及装置 Download PDF

Info

Publication number
WO2022165873A1
WO2022165873A1 PCT/CN2021/077573 CN2021077573W WO2022165873A1 WO 2022165873 A1 WO2022165873 A1 WO 2022165873A1 CN 2021077573 W CN2021077573 W CN 2021077573W WO 2022165873 A1 WO2022165873 A1 WO 2022165873A1
Authority
WO
WIPO (PCT)
Prior art keywords
light intensity
information
texture
dynamic
sampling
Prior art date
Application number
PCT/CN2021/077573
Other languages
English (en)
French (fr)
Inventor
田永鸿
康照东
李家宁
周晖晖
张伟
朱林
��昌毅
Original Assignee
北京大学
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 北京大学 filed Critical 北京大学
Publication of WO2022165873A1 publication Critical patent/WO2022165873A1/zh

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
    • H04N19/42—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation

Definitions

  • the invention relates to the technical field of communication, in particular to a combined sampling method and device for simulating retinal fovea and periphery.
  • Neuromorphic vision sensors have the advantages of high temporal resolution, high dynamic range, low data redundancy, and low power consumption. ) has broad application prospects; its corresponding research direction, neuromorphic vision, is also an interdisciplinary and research hotspot in the fields of computational neuroscience and computer vision.
  • the purpose of the embodiments of the present invention is to provide a combined sampling method, device, computer equipment and storage medium for simulating the fovea and periphery of the retina, so as to solve the above technical problems.
  • the embodiments of the present application provide a joint sampling method for simulating the fovea and periphery of the retina, and the method includes:
  • the preset coding mode is used for coding, and an asynchronous pulse signal corresponding to the pulse signal is output;
  • the asynchronous pulse signal is decoded by the decoding module, and the encoded data of the change information and the encoded data of the texture information are separated;
  • Visual task analysis is performed according to the encoded data of the change information and the encoded data of the texture information, and corresponding analysis results are obtained.
  • the visual tasks include the task of perceiving the current scene change and the task of reconstructing images and videos.
  • the method further includes:
  • the asynchronous pulse signal is characterized by a biological vision sampling model, and the asynchronous pulse array signal is a discrete lattice sparse in space and time.
  • the biological vision sampling model includes a peripheral retinal physiological structure and function model and a retinal foveal physiological structure and function model
  • the characterization of the asynchronous pulse signal by the biological vision sampling model includes:
  • the pulse coding representation of the dynamic changes of objects in different spatial positions in the current scene is perceived;
  • the control module switches between the dynamic perception mode and the texture perception mode, and according to the current scene The scene situation automatically perceives the dynamic changes of objects and the fusion coding representation of texture information.
  • the method further includes:
  • the light intensity information in the neighborhood of the receptive field is sampled and encoded to obtain the corresponding sampling result and encoding result.
  • the method further includes:
  • the dynamic information flow and the texture information flow are both adopt an event encoding format
  • the event encoding format includes quadruplets of abscissa, ordinate, time and polarity, and each quadruple is encoded with a preset number of bits
  • the dynamic information stream and the texture information stream are controlled and processed by the control module to generate a corresponding mixed coded stream, and the mixed coded stream adopts the event coding format.
  • the method further includes:
  • the mixed coded stream is processed using a corresponding preset processing method, and the preset processing method includes performing mixed coded stream separation processing on the mixed coded stream and processing the mixed coded stream.
  • Stream for scene reconstruction processing includes performing mixed coded stream separation processing on the mixed coded stream and processing the mixed coded stream.
  • the method further includes:
  • the first light intensity value and the second light intensity value are compared, and if the absolute value of the difference between the first light intensity value and the second light intensity value exceeds a preset threshold, the event, and the current light intensity value of the current receptive field is recorded by the circuit in the sampling device.
  • the embodiments of the present application provide a combined sampling device that mimics the fovea and periphery of the retina, the device comprising:
  • the acquisition module is used to acquire the light intensity change information and object texture information of different spatial positions in the current scene
  • an encoding module configured to encode the light intensity change information and the object texture information acquired by the acquisition module according to a preset encoding method, to obtain a pulse signal corresponding to the optical signal;
  • the preset coding mode is used for coding, and an asynchronous pulse signal corresponding to the pulse signal is output;
  • a decoding module for decoding the asynchronous pulse signal encoded by the encoding module, and separating the encoded data of the change information and the encoded data of the texture information;
  • the analysis module is configured to analyze the visual task according to the encoded data of the change information and the encoded data of the texture information separated by the decoding module, and obtain a corresponding analysis result.
  • an embodiment of the present application provides a computer device, including a memory and a processor, where computer-readable instructions are stored in the memory, and when the computer-readable instructions are executed by the processor, the processor causes the processor to Perform the above method steps.
  • an embodiment of the present application provides a storage medium storing computer-readable instructions.
  • the computer-readable instructions are executed by one or more processors, the one or more processors perform the above method steps.
  • light intensity change information and object texture information at different spatial positions in the current scene are obtained; the light intensity change information and object texture information are encoded according to a preset encoding method to obtain a pulse signal corresponding to the light signal;
  • the preset coding method is used for coding, and the asynchronous pulse signal corresponding to the pulse signal is output; the asynchronous pulse signal is decoded by the decoding module,
  • the change information encoded data and the texture information encoded data are separated; the visual task analysis is performed according to the change information encoded data and the texture information encoded data, and the corresponding analysis results are obtained.
  • the visual tasks include the task of perceiving the current scene change and the task of reconstructing images and videos. Therefore, using the sampling method provided in the embodiment of the present application, the optical signal can be perceptually encoded and a pulse array can be generated, and at the same time, the decoding module at the decoding end can complete pulse type separation and scene brightness reconstruction according to the encoding rules. Therefore, the sampling method has high performance.
  • the advantages of temporal resolution, high dynamic range, and low power consumption can be used in application scenarios such as high-speed motion blur and extreme lighting. It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the invention.
  • FIG. 1 is a schematic flowchart of a joint sampling method for imitating retinal fovea and periphery provided by an embodiment of the present disclosure
  • Fig. 2 is a kind of processing flow chart of dynamic visual pulse signal sampling coding and decoding provided by an embodiment of the present invention
  • Fig. 3 is a kind of processing flow chart of dynamic perception and texture perception of a signal provided by an embodiment of the present invention
  • FIG. 4 is a process flow diagram of generating a mixed event on a signal perception result through a control circuit according to an embodiment of the present invention
  • FIG. 5 is a flowchart of a process for performing scene brightness reconstruction on a mixed event stream according to an embodiment of the present invention
  • FIG. 6 is a schematic flowchart of a joint sampling method for imitating the fovea and the periphery under a specific application scenario provided by an embodiment of the present disclosure
  • FIG. 7 is a schematic structural diagram of a combined sampling device imitating the fovea and periphery of the retina provided by an embodiment of the present disclosure.
  • an embodiment of the present disclosure provides a joint sampling method of the fovea and the periphery of the retina, and the joint sampling method of the fovea and the periphery of the retina specifically includes the following method steps:
  • S101 Acquire light intensity change information and object texture information at different spatial positions in the current scene.
  • S102 Encode the light intensity change information and the object texture information according to a preset encoding method to obtain a pulse signal corresponding to the optical signal.
  • S103 In the case of automatically switching between the dynamic sensing mode and the texture sensing mode through the control module, use a preset coding method to perform coding, and output an asynchronous pulse signal corresponding to the pulse signal.
  • S104 Decode the asynchronous pulse signal by the decoding module, and separate the encoded data of the change information and the encoded data of the texture information.
  • the decoding module is a pulse signal decoding module, and the decoding module is used for event separation.
  • the process of event separation is as follows: maintain a timestamp for each location, and perform a timestamp on the mixed event stream through the timestamp. Separation, get dynamic information event flow and texture information event flow.
  • the decoding module is a pulse signal decoding module, and the decoding module is also used to perform brightness reconstruction.
  • the process of brightness reconstruction is as follows: a stack is maintained for each position, and continuous dynamic events are recorded through the maintained stack. And after the next texture event comes, restore the brightness information of the previous period of time.
  • the decoding module is a pulse signal decoding module, and the process of event separation and brightness reconstruction through the decoding module specifically includes the following steps:
  • the decoding module reads the events generated by each position in chronological order
  • the dynamic event indicates the direction of light intensity change, and the light intensity cannot be directly reconstructed. At this time, it is pushed into the stack and waits for the subsequent asynchronous reference frame;
  • Texture events represent the result of light intensity integration over a period of time, and the light intensity can be directly reconstructed and used as the above asynchronous reference frame to restore the light intensity within the time range of dynamic events in the stack.
  • S105 Perform visual task analysis according to the encoded data of the change information and the encoded data of the texture information, and obtain a corresponding analysis result.
  • the visual task includes the task of perceiving the current scene change and the task of reconstructing the image and video.
  • the sampling method provided by the embodiment of the present disclosure further includes the following steps: sampling and recording the dynamic changes of light intensity and texture feature information in the form of pulses; Characterized, the asynchronous pulse array signal is a discrete lattice sparse in space and time.
  • the biological vision sampling model includes a peripheral retinal physiological structure and function model and a retinal foveal physiological structure and function model
  • the characterization of the asynchronous pulse signal by the biological vision sampling model includes the following steps:
  • the control module switches between the dynamic perception mode and the texture perception mode, and automatically perceives the dynamic changes and textures of objects according to the current scene. Fusion coding representation of information.
  • the retinal peripheral physiological structure and function model is used to perceive the dynamic change information of the object.
  • the process of perceiving the dynamic change information of objects through a model of the peripheral physiological structure and function of the retina is as follows:
  • the circuit In order to adopt the joint sampling method used in the embodiments of the present disclosure, the circuit in the sampling apparatus for joint sampling.
  • the model of retinal peripheral physiological structure and function includes a joint sampling module for joint sampling, and the joint sampling module is specifically used for: when the light intensity change exceeds a threshold, the event quadruple of the event is issued , where the event quadruple of the event includes abscissa, ordinate, and time, which represent the location and time of the event, and the polarity represents the direction of light intensity change, that is, dynamic information flow.
  • the fovea-like physiological structure and function model includes a joint sampling model for joint sampling, and the joint sampling model is specifically used for: when the light intensity integral exceeds a threshold, the event quadruple of the event is released, wherein the event's The event quadruple includes the abscissa, ordinate, and time to represent the location and time of the event, and the polarity to represent the type of integrator used, that is, the texture information flow.
  • the control module is a pulse signal generator control module. If the event is a dynamic event, after the dynamic event quadruple enters the control module, the time stamp of the event and the time stamp recorded by the control module are obtained. Time interval, if the time interval is greater than the above-mentioned pre-configured predetermined value, the dynamic event quadruple is ignored, otherwise, the dynamic event quadruple is output.
  • the texture type event quadruple is directly output.
  • control module In a practical application scenario, no matter what kind of event is output from the control module, it follows a 64-bit encoding format, and the control module updates the recorded timestamp and resets the integrator in the above texture vision sampling module.
  • the sampling method provided by the embodiment of the present disclosure further includes the following steps:
  • the events in the dynamic information flow and the texture information flow are simultaneously processed by the control module in the joint sampling device for joint sampling.
  • the time interval between each dynamic event and the previous event must not exceed a certain predetermined value.
  • the time interval between each texture type event and the previous event must be greater than the predetermined value; and the scene change of the current scene is judged according to the next event type and timestamp, and the perception mode is automatically switched by the control module according to the scene change of the current scene. to the perception mode that matches the scene transitions of the current scene.
  • the fovea-like physiological structure and function model is used to perceive the texture information of objects, that is, the light intensity of the current receptive field is integrated, and when the integrated value exceeds a certain set threshold, an event is issued, Also reset the integrator.
  • the sampling method provided by the embodiment of the present disclosure further includes the following steps: sampling and encoding the light intensity information in the neighborhood of the receptive field to obtain corresponding sampling results and encoding results.
  • the sampling method provided by the embodiment of the present disclosure further includes the following steps:
  • each quadruple is encoded with a preset number of bits.
  • the preset number can be set to 64, then each quadruple use 64-bit encoding;
  • the dynamic information stream and the texture information stream are controlled and processed by the control module to generate the corresponding mixed code stream.
  • the mixed code stream adopts the event encoding format.
  • the event encoding format includes abscissa, ordinate, time and polarity quadruplets.
  • Each The quadruplets are encoded using a preset number of bits. In a specific application scenario, the preset number may be set to 64, and each quadruple is encoded using 64 bits.
  • the quadruple encoded by the event quadruple can also be replaced by the pulse plane, that is, all pixel position pulses in a certain sampling space-time are encoded with a "0" or "1" mark, in When the scene changes greatly, the transmission bandwidth is saved.
  • the dynamic information stream and the texture information stream are both encoded in the event encoding format.
  • the circuit states of the circuits in the sampling device for joint sampling are updated synchronously.
  • the mixed coded stream also uses the above-mentioned 64-bit event quadruple encoding, and the decoding module at the decoding end determines the specific meaning of each event quadruple and performs subsequent processing.
  • the sampling method provided by the embodiment of the present disclosure further includes the following steps:
  • the type of the mixed encoding stream is judged, and the type of the mixed encoding stream is obtained;
  • a corresponding preset processing method is used to process the mixed coded stream, and the preset processing method includes performing mixed coded stream separation processing on the mixed coded stream and scene reconstruction processing on the mixed coded stream.
  • the sampling method provided by the embodiment of the present disclosure further includes the following steps:
  • the first light intensity value and the second light intensity value are compared, and if the absolute value of the difference between the first light intensity value and the second light intensity value exceeds a preset threshold, the event is issued, and the event is passed through the sampling device.
  • the circuit records the current light intensity value of the current receptive field.
  • DVS Dynamic Vision Sensor, dynamic vision sensor
  • DVS Dynamic Vision Sensor, dynamic vision sensor
  • the integral vision sensor is a visual sensor that imitates the perception of fine texture of objects by foveal cells.
  • the emitted neural pulse signals are described by spatiotemporal pulse array signals, that is, event representation. Compared with traditional fixed frame rate cameras, it has the advantages of high time resolution, low data redundancy, high dynamic range, and low power consumption.
  • the sampling method provided by the embodiment of the present application proposes a scheme of simulating the joint sampling of the fovea and the periphery of the retina from the perspective of functional design.
  • the embodiment of the present application provides a processing flow of dynamic visual pulse signal sampling encoding and decoding, as shown in FIG. 2 , including the following processing steps:
  • Step 1 For the photoelectric conversion module, the optical signal is processed by the sampling chip, and the light intensity value is logarithmically encoded. If the light intensity value is I, the logarithm ln(I) is taken by the circuit design, thereby generating a time Continuous electrical signal; input the logarithmic electrical signal to the differential sensing module, convert the original light intensity value into an electrical signal and input it to the texture sensing module, wherein the differential sensing module and the texture sensing module run simultaneously;
  • Step 2.1 For the differential sensing module, compare the magnitude I of the electrical signal at the current moment with the magnitude I 0 of the electrical signal recorded by the circuit, if
  • Step 2.2 For the texture perception module, two integrators are used at the same time to integrate I and I max -I respectively. If the integral value of any integrator reaches ⁇ i , the control module will be issued with the current coordinate, time and polarity.
  • I is the size of the electrical signal at the current moment
  • I max and ⁇ i are the preset maximum light intensity and the preset integration threshold. If the integrator that integrates I first reaches the threshold, the polarity is 1, otherwise the polarity is 0, as shown in Figure 3;
  • Step 3 For the control module, record the reference time t 0 , and for the event issued by the differential sensing module, compare the time interval between the event timestamp t and t 0 with the preset time interval t w , if tt 0 ⁇ t w , Then output the event, otherwise it will not output; for the event issued by the texture perception module, the event will be output directly; if there is an event output, update t 0 to the event timestamp t, and at the same time, the two integrators in the texture perception module will be updated. reset, as shown in Figure 4.
  • Step 4 The event stream output from the control module is a mixed stream of dynamic events and texture events using the same kind of representation, and the encoding of the scene light intensity is completed;
  • Step 5 Use the preset thresholds ⁇ d and ⁇ i used in encoding, the preset light intensity I max and the preset time interval t w as parameters, and transmit them to the decoding module, specify the range of the abscissa and ordinate in the quadruple, and then set the The events in the mixed event stream are input to the decoding module in turn, and the decoding module maintains a separate reference time t 0 for each set of abscissa and ordinate pairs to distinguish the type of each event, and the stack S is used to reconstruct the position at any time.
  • grayscale value
  • Step 6 For each position in the scene, the decoding module sequentially reads the timestamp t of the event stream generated by the position in chronological order. If tt 0 ⁇ t w , the event is a dynamic event, otherwise the event is Integral event; after reading an event, update t 0 to the current event timestamp t;
  • Step 7.1 If the current event is a dynamic event, push the time and polarity of the event into the stack and wait for future texture events to decode a reference light intensity value;
  • Step 7.2 If the current event is a textured event, if the polarity is 1, the light intensity value I from time t 0 to time t is ⁇ i /(tt 0 ), if the polarity is 0, then the light intensity value I from time t 0 to time t is ⁇ i /(tt 0 ).
  • the light intensity value I at time is I max - ⁇ i /(tt 0 ); if there are elements in the stack later, according to the light intensity value at time t 0 and the polarity of the top element of the stack, the second from the top of the stack can be obtained.
  • the change of light intensity from time t' to t 0 of each element if the polarity of the top element of the stack is 1, the light intensity increases from time t' to time t 0 , and the value of light intensity at time t' is Ie - ⁇ d ; If the polarity is 0, the light intensity decreases from time t' to time t 0 , and the light intensity value at time t' is Ie ⁇ d .
  • the light intensity value at any time can be obtained according to the light intensity change in the time interval of every two adjacent elements in the stack;
  • Step 8 By separating the mixed streams and considering the meanings of different types of event streams, the decoding module completes the decoding of the mixed streams and reconstruction of the scene light intensity, as shown in FIG. 5 .
  • FIG. 6 it is a schematic flowchart of a joint sampling method for simulating the fovea and the periphery of the retina in a specific application scenario provided by an embodiment of the present disclosure.
  • the joint sampling method of the fovea and the periphery under the specific application scenario provided by the embodiment of the present disclosure specifically includes the following steps:
  • Step 1 input the optical signal into the dynamic sensing module for processing to obtain a dynamic pulse array signal; and input the optical signal into the texture sensing module for processing to obtain a textured pulse array signal;
  • the dynamic sensing module is a model of human
  • the perception principle of dynamic events in the periphery of the retina of the eye according to the light intensity change information in the spatial neighborhood, converts the light signal in a period of time into a pulse array signal with neuromorphic representation
  • the texture perception module is modeled after the human retina fovea The perception principle of texture information, according to the texture information of the static objects in the scene, convert the light signal in a period of time into a pulse array signal with neuromorphic representation
  • Step 2 Convert the dynamic pulse array signal and the texture pulse
  • the array signals are all input into the control coding for processing to obtain a mixed pulse array signal, and the scene reconstruction is performed according to the mixed pulse array signal, or a machine vision analysis is performed, wherein the control module, according to a certain coding rule, converts the above The pulse array
  • the optical signal is converted into a spatiotemporal pulse array signal, which is a sparse spatiotemporal lattice.
  • the optical signal can be perceptually encoded and a pulse array can be generated, and the decoding module at the decoding end can complete the process of pulse type separation and scene brightness reconstruction according to the encoding rules.
  • the sampling method provided by the embodiment of the present application can jointly represent, simultaneously perceive dynamic and static light intensity information, meet the requirements of machine vision perception and human vision perception, and has the advantages of high temporal resolution, high dynamic range, and low power consumption.
  • the sampling method provided by the embodiment of the present application can also flexibly switch between the dynamic sampling mode and the static sampling mode.
  • light intensity change information and object texture information at different spatial positions in the current scene are obtained; the light intensity change information and object texture information are encoded according to a preset encoding method, and a pulse signal corresponding to the light signal is obtained;
  • the preset coding method is used for coding, and the asynchronous pulse signal corresponding to the pulse signal is output; the asynchronous pulse signal is decoded by the decoding module,
  • the change information encoded data and the texture information encoded data are separated; the visual task analysis is performed according to the change information encoded data and the texture information encoded data, and the corresponding analysis results are obtained.
  • the visual tasks include the task of perceiving the current scene change and the task of reconstructing images and videos. Therefore, using the sampling method provided in the embodiment of the present application, the optical signal can be perceptually encoded and a pulse array can be generated, and at the same time, the decoding module at the decoding end can complete pulse type separation and scene brightness reconstruction according to the encoding rules. Therefore, the sampling method has high performance.
  • the advantages of temporal resolution, high dynamic range, and low power consumption can be used in application scenarios such as high-speed motion blur and extreme lighting.
  • the following is an embodiment of the combined sampling device for imitating the fovea and the periphery according to the present invention, which can be used to execute the embodiment of the joint sampling method for imitating the fovea and the periphery of the present invention.
  • the embodiment of the combined sampling device for imitating retinal fovea and periphery of the present invention please refer to the embodiment of the combined sampling method for imitating retinal fovea and periphery of the present invention.
  • FIG. 7 shows a schematic structural diagram of a fovea-periphery joint sampling device provided by an exemplary embodiment of the present invention.
  • the combined sampling device of the fovea and the periphery of the retina can be realized by software, hardware or a combination of the two to become all or a part of the terminal.
  • the fovea-periphery-simulated joint sampling device includes an acquisition module 701 , an encoding module 702 , a decoding module 703 and an analysis module 704 .
  • the acquisition module 701 is used to acquire the light intensity change information and object texture information of different spatial positions in the current scene;
  • an encoding module 702 configured to encode the light intensity change information and object texture information acquired by the acquisition module 701 according to a preset encoding method, to obtain a pulse signal corresponding to the optical signal;
  • the preset coding mode is used for coding, and an asynchronous pulse signal corresponding to the pulse signal is output;
  • the decoding module 703 is used to decode the asynchronous pulse signal encoded by the encoding module 702, and separate the encoded data of the change information and the encoded data of the texture information;
  • the analysis module 704 is configured to perform visual task analysis according to the change information encoded data and texture information encoded data separated by the decoding module 703, and obtain corresponding analysis results.
  • the visual tasks include the task of perceiving current scene changes and the task of reconstructing images and videos.
  • the device further includes:
  • a sampling module (not shown in FIG. 7 ), used for sampling and recording the dynamic changes of light intensity and texture feature information in the form of pulses;
  • the characterization module (not shown in FIG. 7 ) is used to characterize the asynchronous pulse signal through the biological visual sampling model, and the asynchronous pulse array signal is a discrete lattice sparse in space and time.
  • the biological vision sampling model includes a peripheral retinal physiological structure and function model and a retinal foveal physiological structure and function model
  • the characterization module is specifically used for:
  • the control module switches between the dynamic perception mode and the texture perception mode, and automatically perceives the dynamic changes and textures of objects according to the current scene. Fusion coding representation of information.
  • sampling module is also used to:
  • the encoding module 702 is further configured to: encode the light intensity information in the neighborhood of the receptive field to obtain a corresponding encoding result.
  • the obtaining module 701 is also used for:
  • the device also includes:
  • the generation module (not shown in FIG. 7 ) is used to generate a dynamic information flow and a texture information flow according to the dynamic change of the light intensity and the texture information obtained by the obtaining module 701.
  • the encoding format includes quadruplets of abscissa, ordinate, time and polarity, and each quadruple is encoded with a preset number of bits;
  • the dynamic information flow and the texture information flow are controlled and processed by the control module, and the corresponding mixed coding flow is generated, and the mixed coding adopts the event coding format.
  • the obtaining module 701 is further configured to: obtain the mixed encoding stream;
  • the device also includes: a judgment module (not shown in FIG. 7 ) for judging the type of the mixed encoding stream acquired by the acquisition module 701 according to the time interval between the two event quadruplets at the same position, to obtain the mixed encoding the type of stream;
  • the processing module (not shown in FIG. 7 ) is used to process the mixed coded stream by using a corresponding preset processing method according to the type of the mixed coded stream determined by the judgment module.
  • the coded stream is subjected to mixed coded stream separation processing and the mixed coded stream is subjected to scene reconstruction processing.
  • the obtaining module 701 is configured to: obtain the first light intensity value of the current receptive field, and obtain the second light intensity value recorded by the circuit in the sampling device for joint sampling;
  • the device also includes:
  • a comparison module (not shown in FIG. 7 ) is used to compare the first light intensity value and the second light intensity value acquired by the acquisition module 701. If the difference between the first light intensity value and the second light intensity value is If the absolute value exceeds the preset threshold, the event will be released;
  • the recording module (not shown in FIG. 7 ) is used to record the current light intensity value of the current receptive field through the circuit in the sampling device.
  • the combined sampling device for imitating the fovea and the periphery performs the joint sampling method for imitating the fovea and the periphery
  • only the division of the above functional modules is used for illustration.
  • the above-mentioned function allocation can be completed by different function modules according to requirements, that is, the internal structure of the device is divided into different function modules, so as to complete all or part of the functions described above.
  • the combined sampling device for imitating retinal fovea and periphery provided by the above-mentioned embodiments and the embodiment of the joint sampling method for imitating retinal fovea and periphery belong to the same concept, and the embodiment and implementation process are detailed in the joint sampling method for imitating retinal fovea and periphery. Examples are not repeated here.
  • the acquisition module is used to acquire the light intensity change information and object texture information of different spatial positions in the current scene;
  • the encoding module is used to obtain the light intensity change information and object texture information obtained by the acquisition module according to a preset encoding method. Encoding is performed to obtain a pulse signal corresponding to the optical signal; and in the case of automatic switching between the dynamic perception mode and the texture perception mode through the control module, the preset encoding method is used for encoding, and the asynchronous corresponding to the pulse signal is output.
  • the decoding module is used to decode the asynchronous pulse signal encoded by the encoding module, and separate the encoded data of the change information and the encoded data of the texture information; and the analysis module is used to encode the encoded data of the change information and the encoded data of the texture information according to the encoded data of the decoding module
  • the data is analyzed for visual tasks, and the corresponding analysis results are obtained.
  • the visual tasks include the task of perceiving the current scene changes and the task of reconstructing images and videos. Therefore, using the sampling device provided by the embodiment of the present application, the optical signal can be perceptually encoded and a pulse array can be generated.
  • the decoding module at the decoding end can complete the pulse type separation and scene brightness reconstruction according to the encoding rules. Therefore, the sampling method has high performance.
  • the advantages of temporal resolution, high dynamic range, and low power consumption can be used in application scenarios such as high-speed motion blur and extreme lighting.
  • a computer device includes a memory, a processor, and a computer program stored in the memory and running on the processor.
  • the processor executes the computer program, the following steps are implemented: acquiring the current scene Light intensity change information and object texture information at different spatial positions; encode the light intensity change information and object texture information according to the preset coding method to obtain a pulse signal corresponding to the light signal; in the dynamic perception mode and texture perception through the control module
  • the preset encoding method is used for encoding, and the asynchronous pulse signal corresponding to the pulse signal is output; the asynchronous pulse signal is decoded by the decoding module, and the encoded data of the change information and the encoded data of the texture information are separated.
  • Carry out visual task analysis according to the encoded data of the change information and the encoded data of the texture information, and obtain the corresponding analysis results.
  • the visual task includes the task of perceiving the current scene change and the task of reconstructing the image and video.
  • a storage medium storing computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors perform the following steps: obtaining the current scene Light intensity change information and object texture information at different spatial positions; encode the light intensity change information and object texture information according to the preset coding method to obtain a pulse signal corresponding to the light signal; in the dynamic perception mode and texture perception through the control module In the case of automatic switching between modes, the preset encoding method is used for encoding, and the asynchronous pulse signal corresponding to the pulse signal is output; the asynchronous pulse signal is decoded by the decoding module, and the encoded data of the change information and the encoded data of the texture information are separated. ; Carry out visual task analysis according to the encoded data of the change information and the encoded data of the texture information, and obtain the corresponding analysis results.
  • the visual task includes the task of perceiving the current scene change and the task of reconstructing the image and video.
  • the realization of all or part of the processes in the methods of the above embodiments can be accomplished by instructing relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the program is During execution, it may include the processes of the embodiments of the above-mentioned methods.
  • the aforementioned storage medium may be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (Read-Only Memory, ROM), or a random access memory (Random Access Memory, RAM) or the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)

Abstract

一种仿视网膜中央凹与外周的联合采样方法、装置、计算机设备和存储介质。所述方法包括：通过解码模块对异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；根据变化信息编码数据和纹理信息编码数据进行视觉任务分析，得到对应的分析结果，视觉任务包括感知当前场景变化的任务和重构图像视频的任务。因此，该采样方法能够对光信号进行感知编码并生成脉冲阵列，同时解码端的解码模块可根据编码规则完成脉冲类型分离及场景亮度重构，因此，该采样方法具有高时间分辨率、高动态范围、低功耗等优点，能够应用于高速运动模糊与极端光照等应用场景中。

Description

一种仿视网膜中央凹与外周的联合采样方法及装置 技术领域
本发明涉及通信技术领域，特别涉及一种仿视网膜中央凹与外周的联合采样方法及装置。
背景技术
神经形态视觉传感器具有高时域分辨率、高动态范围、低数据冗余和低功耗等优势，在自动驾驶、无人机视觉导航等多个领域(尤其是在高速运动或极端光照场景下)具有广阔的应用前景；其对应的研究方向，神经形态视觉，也是计算神经科学与计算机视觉领域的交叉学科与研究热点。
在现有的各类神经形态相机中，大部分设计均只基于一种神经形态视觉传感器的原理，只感知物体运动信息或纹理信息，无法做到同时感知；而通过DVS与传统图像相结合的DAVIS系列相机存在帧率不匹配、无法对齐等问题；通过DVS触发积分型传感器的ATIS系列相机存在动态事件丢失、低光照区域过长时间积分等问题，无论何种设计，均无法匹配人眼感知动态与纹理的生物学特征。
发明内容
本发明实施例的目的在于提供一种仿视网膜中央凹与外周的联合采样方法、装置、计算机设备和存储介质，以解决上述技术问题。
第一方面，本申请实施例提供了一种仿视网膜中央凹与外周的联合采样方法，所述方法包括：
获取当前场景中不同空间位置的光强变化信息和物体纹理信息；
根据预设编码方式对所述光强变化信息和所述物体纹理信息进行编码，得到与光信号对应的脉冲信号；
在通过控制模块在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用所述预设编码方式进行编码，并输出与所述脉冲信号对应的异步脉冲信号；
通过解码模块对所述异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；
根据所述变化信息编码数据和所述纹理信息编码数据进行视觉任务分析，得到对应的分析结果，所述视觉任务包括感知当前场景变化的任务和重构图像视频的任务。
在一种实施方式中，所述方法还包括：
以脉冲的形式采样并记录光强动态变化以及纹理特征信息；或者，
通过生物视觉采样模型对所述异步脉冲信号进行表征，所述异步脉冲阵列信号为时空稀疏的离散点阵。
在一种实施方式中，所述生物视觉采样模型包括仿视网膜外周生理结构与功能模型和仿视网膜中央凹生理结构与功能模型，所述通过生物视觉采样模型对所述异步脉冲信号进行表征包括：
根据所述仿视网膜外周生理结构与功能模型，感知当前场景中不同空间位置下的物体动态变化的脉冲编码表征；或者，
根据所述仿视网膜中央凹生理结构与功能模型，感知当前场景中不同空间位置下的物体纹理结构的编码表征；或者，
根据所述仿视网膜外周生理结构与功能模型和所述仿视网膜中央凹生理结构与功能模型，通过所述控制模块在所述动态感知模式和所述纹理感知模式之间进行切换，并根据当前场景的场景情况自动感知物体动态变化与纹理信息的融合编码表征。
在一种实施方式中，所述方法还包括：
对感受野邻域内的光强信息进行采样与编码，得到对应的采样结果和编码结果。
在一种实施方式中，所述方法还包括：
在每个感受野内获取对应的光强动态变化和纹理信息，并根据所述光强动态变化和所述纹理信息生成动态信息流与纹理信息流，所述动态信息流和所述纹理信息流均采用事件编码格式，所述事件编码格式包括横坐标、纵坐标、时间和极性四元组，每个四元组使用预设数量比特编码；
通过所述控制模块对所述动态信息流和所述纹理信息流进行控制处理，生成对应的混合编码流，所述混合编码流采用所述事件编码格式。
在一种实施方式中，所述方法还包括：
获取所述混合编码流；
根据同一位置两个事件四元组之间的时间间隔，判断所述混合编码流的类型，得到所述混合编码流的类型；
根据所述混合编码流的类型，对所述混合编码流采用对应的预设处理方式进行处理，所述预设处理方式包括对所述混合编码流进行混合编码流分离处理和对所述混合编码流进行场景重构处理。
在一种实施方式中，所述方法还包括：
获取当前感受野的第一光强值，以及获取用于进行联合采样的采样装置中的电路所记录的第二光强值；
将所述第一光强值和所述第二光强值进行比较，若所述第一光强值和所述第二光强值之间差值的绝对值超过预设阈值，则发放该事件，并通过所述采样装置中的所述电路记录当前感受野的当前光强值。
第二方面，本申请实施例提供了一种仿视网膜中央凹与外周的联合采样装置，所述装置包括：
获取模块，用于获取当前场景中不同空间位置的光强变化信息和物体纹理信息；
编码模块，用于根据预设编码方式对所述获取模块获取的所述光强变化信息和所述物体纹理信息进行编码，得到与光信号对应的脉冲信号；以及
在通过控制模块在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用所述预设编码方式进行编码，并输出与所述脉冲信号对应的异步脉冲信号；
解码模块，用于对所述编码模块编码出的所述异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；
分析模块，用于根据所述解码模块分离出的所述变化信息编码数据和所述纹理信息编码数据进行视觉任务分析，得到对应的分析结果，所述视觉任务包 括感知当前场景变化的任务和重构图像视频的任务。
第三方面，本申请实施例提供一种计算机设备，包括存储器和处理器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行上述的方法步骤。
第四方面，本申请实施例提供一种存储有计算机可读指令的存储介质，所述计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行上述的方法步骤。
本申请实施例提供的技术方案可以包括以下有益效果：
在本申请实施例中，获取当前场景中不同空间位置的光强变化信息和物体纹理信息；根据预设编码方式对光强变化信息和物体纹理信息进行编码，得到与光信号对应的脉冲信号；在通过控制模块在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用预设编码方式进行编码，并输出与脉冲信号对应的异步脉冲信号；通过解码模块对异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；根据变化信息编码数据和纹理信息编码数据进行视觉任务分析，得到对应的分析结果，视觉任务包括感知当前场景变化的任务和重构图像视频的任务。因此，采用本申请实施例提供的采样方法，能够对光信号进行感知编码并生成脉冲阵列，同时解码端的解码模块可根据编码规则完成脉冲类型分离及场景亮度重构，因此，该采样方法具有高时间分辨率、高动态范围、低功耗等优点，能够应用于高速运动模糊与极端光照等应用场景中。应当理解的是，以上的一般描述和后文的细节描述仅是示例性和解释性的，并不能限制本发明。
附图说明
此处的附图被并入说明书中并构成本说明书的一部分，示出了符合本发明的实施例，并与说明书一起用于解释本发明的原理。
图1是本公开实施例提供的一种仿视网膜中央凹与外周的联合采样方法的流程示意图；
图2是本发明实施例提供的一种动态视觉脉冲信号采样编码与解码的处理流程图；
图3是本发明实施例提供的一种对信号进行动态感知与纹理感知的处理流程图；
图4是本发明实施例提供的一种对信号感知结果通过控制电路生成混合事件的处理流程图；
图5是本发明实施例提供的一种对混合事件流进行场景亮度重构的处理流程图；
图6是本公开实施例提供的具体应用场景下的仿视网膜中央凹与外周的联合采样方法的流程示意图；
图7是本公开实施例提供的一种仿视网膜中央凹与外周的联合采样装置的结构示意图。
具体实施方式
以下描述和附图充分地示出本发明的具体实施方案，以使本领域的技术人 员能够实践它们。
应当明确，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其它实施例，都属于本发明保护的范围。
下面结合附图详细说明本公开的可选实施例。
如图1所示，本公开实施例提供一种仿视网膜中央凹与外周的联合采样方法，该仿视网膜中央凹与外周的联合采样方法具体包括以下方法步骤：
S101：获取当前场景中不同空间位置的光强变化信息和物体纹理信息。
S102：根据预设编码方式对光强变化信息和物体纹理信息进行编码，得到与光信号对应的脉冲信号。
S103：在通过控制模块在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用预设编码方式进行编码，并输出与脉冲信号对应的异步脉冲信号。
S104：通过解码模块对异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据。
在本申请实施例中，解码模块为脉冲信号解码模块，该解码模块用于进行事件分离，事件分离的过程具体如下所述：对每个位置维护一个时间戳，通过时间戳对混合事件流进行分离，得到动态信息事件流与纹理信息事件流。
此外，解码模块为脉冲信号解码模块，该解码模块还用于进行亮度重构，亮度重构的过程具体如下所述：对每个位置维护一个栈，通过该维护的栈记录连续的动态事件，并在下一个纹理事件到来后，还原之前一段时间的亮度信息。
在实际应用场景中，解码模块为脉冲信号解码模块，通过该解码模块进行事件分离和亮度重构的过程具体包括以下步骤：
解码模块按时间顺序依次读取每个位置产生的事件；
当同一位置两个相邻事件时间间隔不大于预先配置的预定值时，则判断出后产生事件为动态型事件；
当同一位置两个相邻事件时间间隔大于预先的预定值时，则判断出后产生事件为纹理型事件；
动态型事件表示光强变化方向，无法直接重构光强，此时，入栈并等待后续异步参考帧；
纹理型事件表示一段时间内光强积分结果，可以直接重构光强，并作为上述异步参考帧，还原栈内动态型事件的时间范围内的光强大小。
S105：根据变化信息编码数据和纹理信息编码数据进行视觉任务分析，得到对应的分析结果，视觉任务包括感知当前场景变化的任务和重构图像视频的任务。
在一种可能的实现方式中，本公开实施例提供的采样方法还包括以下步骤：以脉冲的形式采样并记录光强动态变化以及纹理特征信息；或者，通过生物视觉采样模型对异步脉冲信号进行表征，异步脉冲阵列信号为时空稀疏的离散点阵。
在一种可能的实现方式中，生物视觉采样模型包括仿视网膜外周生理结构与功能模型和仿视网膜中央凹生理结构与功能模型，通过生物视觉采样模型对异步脉冲信号进行表征包括以下步骤：
根据仿视网膜外周生理结构与功能模型，感知当前场景中不同空间位置下的物体动态变化的脉冲编码表征；或者，
根据仿视网膜中央凹生理结构与功能模型，感知当前场景中不同空间位置下的物体纹理结构的编码表征；或者，
根据仿视网膜外周生理结构与功能模型和仿视网膜中央凹生理结构与功能模型，通过控制模块在动态感知模式和纹理感知模式之间进行切换，并根据当前场景的场景情况自动感知物体动态变化与纹理信息的融合编码表征。
在本申请实施例中，仿视网膜外周生理结构与功能模型用于感知物体动态变化信息。在实际应用场景中，通过仿视网膜外周生理结构与功能模型感知物体动态变化信息的过程具体如下所述：
将当前感受野的光强大小与电路所记录的光强值进行比较，二者差值的绝对值超过预先设置的某个阈值时，则发放事件，同时电路记录当前光强大小，其中，电路为采用本公开实施例所使用的联合采样方法，进行联合采样的采样装置中的电路。
在本申请实施例中，仿视网膜外周生理结构与功能模型包括用于进行联合采样的联合采样模块，该联合采样模块具体用于：当光强变化超过阈值时，发放该事件的事件四元组，其中，该事件的事件四元组包括横坐标、纵坐标、时间表示为该事件产生位置与时间，极性表示光强变化方向，即动态信息流。
仿视网膜中央凹生理结构与功能模型包括用于进行联合采样的联合采样模型，该联合采样模型具体用于：当光强积分超过阈值时，发放该事件的事件四元组，其中，该事件的事件四元组包括横坐标、纵坐标、时间表示为该事件产生位置与时间，极性表示所使用的积分器类型，即纹理信息流。
在本申请实施例中，控制模块为脉冲信号生成器控制模块，若事件为动态型事件，该动态性事件四元组进入该控制模块后，得到该事件时间戳与控制模块记录的时间戳的时间间隔，若该时间间隔大于上述预先配置的预定值时，则忽略该动态型事件四元组，否则，则输出该动态型事件四元组。
若事件为纹理型事件，该纹理型事件四元组进入该控制模块后，直接输出该纹理型事件四元组。
在实际应用场景中，无论何种事件从控制模块输出，其均遵循64比特编码格式，控制模块更新记录的时间戳并重置上述纹理视觉采样模块中的积分器。
在一种可能的实现方式中，本公开实施例提供的采样方法还包括以下步骤：
通过用于进行联合采样的联合采样装置中的控制模块同时处理动态信息流与纹理信息流中的事件，处理后每个动态型事件距离前一个事件的时间间隔一定不超过某个预定值，每个纹理型事件距离前一个事件的时间间隔一定大于该预定值；以及根据下一个事件类型与时间戳判断当前场景的场景变化情况，并根据当前场景的场景变化情况，通过控制模块自动切换感知模式至与当前场景的场景变换情况匹配的感知模式。
在本申请实施例中，仿视网膜中央凹生理结构与功能模型用于感知物体纹理信息，即，对当前感受野的光强进行积分，当积分值超过某个设定阈值时，则发放事件，同时对积分器进行重置。
在一种可能的实现方式中，本公开实施例提供的采样方法还包括以下步骤：对感受野邻域内的光强信息进行采样与编码，得到对应的采样结果和编码结果。
在一种可能的实现方式中，本公开实施例提供的采样方法还包括以下步骤：
在每个感受野内获取对应的光强动态变化和纹理信息，并根据光强动态变化和纹理信息生成动态信息流与纹理信息流，动态信息流和纹理信息流均采用事件编码格式，事件编码格式包括横坐标、纵坐标、时间和极性四元组，每个 四元组使用预设数量比特编码，在一种具体应用场景下，可以将预设数量设置为64，则每个四元组使用64比特编码；
通过控制模块对动态信息流和纹理信息流进行控制处理，生成对应的混合编码流，混合编码流采用事件编码格式，事件编码格式包括横坐标、纵坐标、时间和极性四元组，每个四元组使用预设数量比特编码，在一种具体应用场景下，可以将预设数量设置为64，则每个四元组使用64比特编码。
在一种可能的实现方式中，事件四元组编码的四元组也可以用脉冲平面代替，即：将某个采样时空的所有像素位置脉冲以“0”或“1”标记进行编码，在场景大幅度变化时，节约传输带宽。其中，动态信息流和纹理信息流均采用事件编码格式进行编码。
在实际应用场景中，在生成混合编码流的同时，同步更新用于进行联合采样的采样装置中的电路的电路状态。
在本申请实施例中，混合编码流同样采用上述64比特事件四元组编码，由解码端的解码模块判断每个事件四元组的具体涵义并进行后续处理。
在一种可能的实现方式中，本公开实施例提供的采样方法还包括以下步骤：
获取混合编码流；
根据同一位置两个事件四元组之间的时间间隔，判断混合编码流的类型，得到混合编码流的类型；
根据混合编码流的类型，对混合编码流采用对应的预设处理方式进行处理，预设处理方式包括对混合编码流进行混合编码流分离处理和对混合编码流进行场景重构处理。
在本申请实施例中，预设处理方式除了上述两种常见的预设处理方式之外，还可以根据不同应用场景的需求，引入其它的预设处理方式，在此不再赘述。
在一种可能的实现方式中，本公开实施例提供的采样方法还包括以下步骤：
获取当前感受野的第一光强值，以及获取用于进行联合采样的采样装置中的电路所记录的第二光强值；
将第一光强值和第二光强值进行比较，若第一光强值和第二光强值之间差值的绝对值超过预设阈值，则发放该事件，并通过采样装置中的电路记录当前感受野的当前光强值。
DVS(Dynamic Vision Sensor，动态视觉传感器)是模仿神经元脉冲发放和视网膜外周细胞对亮度变化敏感机理的视觉传感器；积分型视觉传感器是模仿视网膜中央凹细胞对物体精细纹理感知的视觉传感器，二者发放的神经脉冲信号以时空脉冲阵列信号描述，即事件表征，相对传统固定帧率相机具有高时间分辨率、低数据冗余、高动态范围、低功耗等优势。
现有的动态视觉传感器相机或积分型视觉传感器，无法使用同种表征同时表示两种事件，或存在动态信息丢失、暗场景无法感知等问题。为解决上述问题，本申请实施例提供的采样方法，从功能设计上提出了一种仿视网膜中央凹与外周的联合采样的方案。
本申请实施例提供了一种动态视觉脉冲信号采样编码与解码的处理流程，如图2所示，包括如下处理步骤：
步骤1：对于光电转换模块，通过采样芯片对光信号进行处理，对光强值进行对数编码，如光强值为I，则通过电路设计取对数ln(I)，从而生成在时间上连续的电信号；将取对数后的电信号输入到差分感知模块，将原有光强值 转为电信号输入到纹理感知模块，其中，差分感知模块与纹理感知模块同时运行；
步骤2.1：对于差分感知模块，比较当前时刻电信号大小I与电路所记录的电信号大小I 0，若|I-I 0∣≥θ d，即光强变化超过阈值，其中，θ d为预设的动态感知阈值大小，则向控制模块发放包括当前坐标、时间与极性的四元组。若I＞I 0，则极性为1，否则极性为0，发放四元组后，将I 0更新为当前电信号大小I，如图3所示；
步骤2.2：对于纹理感知模块，同时使用两个积分器，分别对I与I max-I进行积分，若任意积分器积分值到达θ i，则向控制模块发放包括当前坐标、时间与极性的四元组，其中，I为当前时刻电信号大小，I max与θ i为预设最大光强与预设积分阈值大小。若对I进行积分的积分器，首先到达阈值，则极性为1，否则极性为0，如图3所示；
步骤3：对于控制模块，记录参考时间t 0，对于差分感知模块发放的事件，比较该事件时间戳t与t 0的时间间隔与预设时间间隔t w的大小，若t-t 0≤t w，则输出该事件，否则不输出；对于纹理感知模块发放的事件，则直接输出该事件；若有事件输出，则更新t 0为该事件时间戳t，同时，将纹理感知模块中两个积分器重置，如图4所示。
步骤4：从控制模块输出的事件流为使用同种表征的动态型事件与纹理型事件的混合流，完成对场景光强的编码；
步骤5：将编码使用的预设阈值θ d与θ i、预设光强I max与预设时间间隔t w作为参数，传输给解码模块，指定四元组中横纵坐标的范围，之后将混合事件流中的事件依次输入解码模块，解码模块对每组横纵坐标对维护一个单独的参考时间t 0用于区分每个事件的类型，以及栈S用于重构该位置在任意时间的灰度值；
步骤6：对场景中的每个位置，解码模块依次读取该位置按时间先后顺序产生的事件流的时间戳t，若t-t 0≤t w，则该事件为动态型事件，否则该事件为积分型事件；读取一个事件后，更新t 0为当前事件时间戳t；
步骤7.1：若当前事件为动态型事件，则将该事件的时间与极性成对入栈并等待未来的纹理型事件解码出可参考的光强值；
步骤7.2：若当前事件为纹理型事件，若极性为1，则t 0时刻到t时刻的光强值I为θ i/(t-t 0)，若极性为0，则t 0时刻到t时刻的光强值I为I max-θ i/(t-t 0)；之后若栈中有元素，根据t 0时刻的光强值与栈顶元素的极性，可得到自栈顶向下第二个元素的时间t′到t 0时间内光强的变化情况，若栈顶元素极性为1，则t′时刻到t 0时刻光强增大，t′时刻光强值为Ie -θd；若极性为0，则t′时刻到t 0时刻光强减小，t′时刻光强值为Ie θd。依据此方法，可根据栈内每两个相邻元素时间间隔内的光强变化情况，得到任何时间的光强值大小；
步骤8：通过对混合流的分离及考虑不同类型事件流的含义，解码模块完成对混合流的解码及场景光强重构，如图5所示。
如图6所示，是本公开实施例提供的具体应用场景下的仿视网膜中央凹与外周的联合采样方法的流程示意图。
如图6所示，本公开实施例提供的具体应用场景下的仿视网膜中央凹与外周的联合采样方法具体包括以下步骤：
步骤1：将光信号输入至动态感知模块中进行处理，得到动态型脉冲阵列信号；以及光信号输入至纹理感知模块中进行处理，得到纹理型脉冲阵列信号；其中，动态感知模块，即仿照人眼视网膜外周对动态事件的感知原理，根据空间邻域内的光强变化信息，将一段时间间隔内的光信号转换为具有神经形态表示的脉冲阵列信号；纹理感知模块，即仿照人眼视网膜中央凹对纹理信息的感知原理，根据场景中物体静态情况下的纹理信息，将一段时间间隔内的光信号转换为具有神经形态表示的脉冲阵列信号；步骤2：将动态型脉冲阵列信号和纹理型脉冲阵列信号均输入至控制编码中进行处理，得到混合型脉冲阵列信号，并根据混合型脉冲阵列信号进行场景重构，或者进行机器视觉分析，其中，控制模块，即根据一定的编码规则，将上述两种感知模块发放的脉冲阵列信号转换为基于统一表征的混合脉冲流，并可根据该规则分离出动态型脉冲与纹理型脉冲、完成场景亮度重构及其他视觉任务，可兼顾面向人类视觉观看与机器视觉分析。
正如图6所示，通过如图6所示的采样过程，将光信号转换为时空脉冲阵列信号，即时空稀疏的点阵。正如图6所示，通过本申请实施例提供的采样方法，可对光信号进行感知编码并生成脉冲阵列，同时解码端的解码模块可根据编码规则完成脉冲类型分离及场景亮度重构过程。
本申请实施例提供的采样方法，能够联合表征，同时感知动态和静态光强信息，同时满足机器视觉感知和人类视觉感知的需求，具有高时间分辨率、高动态范围、低功耗等优势。此外，本申请实施例提供的采样方法，还能够在动态采样模式和静态采样模式之间进行灵活切换。
在本公开实施例中，获取当前场景中不同空间位置的光强变化信息和物体纹理信息；根据预设编码方式对光强变化信息和物体纹理信息进行编码，得到与光信号对应的脉冲信号；在通过控制模块在动态感知模式和纹理感知模式之 间进行自动切换的情况下，使用预设编码方式进行编码，并输出与脉冲信号对应的异步脉冲信号；通过解码模块对异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；根据变化信息编码数据和纹理信息编码数据进行视觉任务分析，得到对应的分析结果，视觉任务包括感知当前场景变化的任务和重构图像视频的任务。因此，采用本申请实施例提供的采样方法，能够对光信号进行感知编码并生成脉冲阵列，同时解码端的解码模块可根据编码规则完成脉冲类型分离及场景亮度重构，因此，该采样方法具有高时间分辨率、高动态范围、低功耗等优点，能够应用于高速运动模糊与极端光照等应用场景中。
下述为本发明仿视网膜中央凹与外周的联合采样装置实施例，可以用于执行本发明仿视网膜中央凹与外周的联合采样方法实施例。对于本发明仿视网膜中央凹与外周的联合采样装置实施例中未披露的细节，请参照本发明仿视网膜中央凹与外周的联合采样方法实施例。
请参见图7，其示出了本发明一个示例性实施例提供的仿视网膜中央凹与外周的联合采样装置的结构示意图。该仿视网膜中央凹与外周的联合采样装置可以通过软件、硬件或者两者的结合实现成为终端的全部或一部分。该仿视网膜中央凹与外周的联合采样装置包括获取模块701、编码模块702、解码模块703和分析模块704。
具体而言，获取模块701，用于获取当前场景中不同空间位置的光强变化信息和物体纹理信息；
编码模块702，用于根据预设编码方式对获取模块701获取的光强变化信息和物体纹理信息进行编码，得到与光信号对应的脉冲信号；以及
在通过控制模块(在图7中未示出)在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用预设编码方式进行编码，并输出与脉冲信号对应的异步脉冲信号；
解码模块703，用于对编码模块702编码出的异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；
分析模块704，用于根据解码模块703分离出的变化信息编码数据和纹理信息编码数据进行视觉任务分析，得到对应的分析结果，视觉任务包括感知当前场景变化的任务和重构图像视频的任务。
可选的，所述装置还包括：
采样模块(在图7中未示出)，用于以脉冲的形式采样并记录光强动态变化以及纹理特征信息；
表征模块(在图7中未示出)用于：通过生物视觉采样模型对异步脉冲信号进行表征，异步脉冲阵列信号为时空稀疏的离散点阵。
可选的，生物视觉采样模型包括仿视网膜外周生理结构与功能模型和仿视网膜中央凹生理结构与功能模型，表征模块具体用于：
根据仿视网膜外周生理结构与功能模型，感知当前场景中不同空间位置下的物体动态变化的脉冲编码表征；或者，
根据仿视网膜中央凹生理结构与功能模型，感知当前场景中不同空间位置下的物体纹理结构的编码表征；或者，
根据仿视网膜外周生理结构与功能模型和仿视网膜中央凹生理结构与功能模型，通过控制模块在动态感知模式和纹理感知模式之间进行切换，并根据当前场景的场景情况自动感知物体动态变化与纹理信息的融合编码表征。
可选的，采样模块还用于：
对感受野邻域内的光强信息进行采样，得到对应的采样结果；
编码模块702还用于：感受野邻域内的光强信息进行编码，得到对应的编码结果。
可选的，获取模块701还用于：
在每个感受野内获取对应的光强动态变化和纹理信息；
所述装置还包括：
生成模块(在图7中未示出)，用于根据获取模块701获取的光强动态变化和纹理信息生成动态信息流与纹理信息流，动态信息流和纹理信息流均采用事件编码格式，事件编码格式包括横坐标、纵坐标、时间和极性四元组，每个四元组使用预设数量比特编码；
通过控制模块对动态信息流和纹理信息流进行控制处理，生成对应的混合编码流，混合编码采用事件编码格式。
可选的，获取模块701还用于：获取混合编码流；
所述装置还包括：判断模块(在图7中未示出)，用于根据同一位置两个事件四元组之间的时间间隔，判断获取模块701获取的混合编码流的类型，得到混合编码流的类型；
处理模块(在图7中未示出)，用于根据判断模块判断出的混合编码流的类型，对混合编码流采用对应的预设处理方式进行处理，处理模块经预设处理方式包括对混合编码流进行混合编码流分离处理和对混合编码流进行场景重构处理。
可选的，获取模块701用于：获取当前感受野的第一光强值，以及获取用于进行联合采样的采样装置中的电路所记录的第二光强值；
所述装置还包括：
比较模块(在图7中未示出)，用于将获取模块701获取的第一光强值和第二光强值进行比较，若第一光强值和第二光强值之间差值的绝对值超过预设阈值，则发放该事件；
记录模块(在图7中未示出)，用于通过采样装置中的电路记录当前感受野的当前光强值。
需要说明的是，上述实施例提供的仿视网膜中央凹与外周的联合采样装置在执行仿视网膜中央凹与外周的联合采样方法时，仅以上述各功能模块的划分进行举例说明，实际应用中，可以根据需要而将上述功能分配由不同的功能模块完成，即将设备的内部结构划分成不同的功能模块，以完成以上描述的全部或者部分功能。另外，上述实施例提供的仿视网膜中央凹与外周的联合采样装置与仿视网膜中央凹与外周的联合采样方法实施例属于同一构思，其体现实现过程详见仿视网膜中央凹与外周的联合采样方法实施例，这里不再赘述。
在本公开实施例中，获取模块用于获取当前场景中不同空间位置的光强变化信息和物体纹理信息；编码模块用于根据预设编码方式对获取模块获取的光强变化信息和物体纹理信息进行编码，得到与光信号对应的脉冲信号；以及在通过控制模块在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用预设编码方式进行编码，并输出与脉冲信号对应的异步脉冲信号；解码模块用于对编码模块编码出的异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；以及分析模块用于根据解码模块分离出的变化信息编码数据和纹理信息编码数据进行视觉任务分析，得到对应的分析结果，视觉任务包括感知当前场景变化的任务和重构图像视频的任务。因此，采用本申请实施例 提供的采样装置，能够对光信号进行感知编码并生成脉冲阵列，同时解码端的解码模块可根据编码规则完成脉冲类型分离及场景亮度重构，因此，该采样方法具有高时间分辨率、高动态范围、低功耗等优点，能够应用于高速运动模糊与极端光照等应用场景中。
在一个实施例中，提出了一种计算机设备，计算机设备包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，处理器执行计算机程序时实现以下步骤：获取当前场景中不同空间位置的光强变化信息和物体纹理信息；根据预设编码方式对光强变化信息和物体纹理信息进行编码，得到与光信号对应的脉冲信号；在通过控制模块在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用预设编码方式进行编码，并输出与脉冲信号对应的异步脉冲信号；通过解码模块对异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；根据变化信息编码数据和纹理信息编码数据进行视觉任务分析，得到对应的分析结果，视觉任务包括感知当前场景变化的任务和重构图像视频的任务。
在一个实施例中，提出了一种存储有计算机可读指令的存储介质，该计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行以下步骤：获取当前场景中不同空间位置的光强变化信息和物体纹理信息；根据预设编码方式对光强变化信息和物体纹理信息进行编码，得到与光信号对应的脉冲信号；在通过控制模块在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用预设编码方式进行编码，并输出与脉冲信号对应的异步脉冲信号；通过解码模块对异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；根据变化信息编码数据和纹理信息编码数据进行视觉任务分析，得到对应的分析结果，视觉任务包括感知当前场景变化的任务和重构图像视频的任务。
本领域普通技术人员可以理解实现上述实施例方法中的全部或部分流程，是可以通过计算机程序来指令相关的硬件来完成，该计算机程序可存储于一计算机可读取存储介质中，该程序在执行时，可包括如上述各方法的实施例的流程。其中，前述的存储介质可为磁碟、光盘、只读存储记忆体(Read-Only Memory，ROM)等非易失性存储介质，或随机存储记忆体(Random Access Memory，RAM)等。
以上实施例的各技术特征可以进行任意的组合，为使描述简洁，未对上述实施例中的各个技术特征所有可能的组合都进行描述，然而，只要这些技术特征的组合不存在矛盾，都应当认为是本说明书记载的范围。
以上所述实施例仅表达了本发明的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对本发明专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本发明构思的前提下，还可以做出若干变形和改进，这些都属于本发明的保护范围。因此，本发明专利的保护范围应以所附权利要求为准。

Claims (10)

  1. 一种仿视网膜中央凹与外周的联合采样方法，其特征在于，所述方法包括：
    获取当前场景中不同空间位置的光强变化信息和物体纹理信息；
    根据预设编码方式对所述光强变化信息和所述物体纹理信息进行编码，得到与光信号对应的脉冲信号；
    在通过控制模块在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用所述预设编码方式进行编码，并输出与所述脉冲信号对应的异步脉冲信号；
    通过解码模块对所述异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；
    根据所述变化信息编码数据和所述纹理信息编码数据进行视觉任务分析，得到对应的分析结果，所述视觉任务包括感知当前场景变化的任务和重构图像视频的任务。
  2. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    以脉冲的形式采样并记录光强动态变化以及纹理特征信息；或者，
    通过生物视觉采样模型对所述异步脉冲信号进行表征，所述异步脉冲阵列信号为时空稀疏的离散点阵。
  3. 根据权利要求2所述的方法，其特征在于，所述生物视觉采样模型包括仿视网膜外周生理结构与功能模型和仿视网膜中央凹生理结构与功能模型，所述通过生物视觉采样模型对所述异步脉冲信号进行表征包括：
    根据所述仿视网膜外周生理结构与功能模型，感知当前场景中不同空间位置下的物体动态变化的脉冲编码表征；或者，
    根据所述仿视网膜中央凹生理结构与功能模型，感知当前场景中不同空间位置下的物体纹理结构的编码表征；或者，
    根据所述仿视网膜外周生理结构与功能模型和所述仿视网膜中央凹生理结构与功能模型，通过所述控制模块在所述动态感知模式和所述纹理感知模式之间进行切换，并根据当前场景的场景情况自动感知物体动态变化与纹理信息的 融合编码表征。
  4. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    对感受野邻域内的光强信息进行采样与编码，得到对应的采样结果和编码结果。
  5. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    在每个感受野内获取对应的光强动态变化和纹理信息，并根据所述光强动态变化和所述纹理信息生成动态信息流与纹理信息流，所述动态信息流和所述纹理信息流均采用事件编码格式，所述事件编码格式包括横坐标、纵坐标、时间和极性四元组，每个四元组使用预设数量比特编码；
    通过所述控制模块对所述动态信息流和所述纹理信息流进行控制处理，生成对应的混合编码流，所述混合编码流采用所述事件编码格式。
  6. 根据权利要求5所述的方法，其特征在于，所述方法还包括：
    获取所述混合编码流；
    根据同一位置两个事件四元组之间的时间间隔，判断所述混合编码流的类型，得到所述混合编码流的类型；
    根据所述混合编码流的类型，对所述混合编码流采用对应的预设处理方式进行处理，所述预设处理方式包括对所述混合编码流进行混合编码流分离处理和对所述混合编码流进行场景重构处理。
  7. 根据权利要求1所述的方法，其特征在于，所述方法还包括：
    获取当前感受野的第一光强值，以及获取用于进行联合采样的采样装置中的电路所记录的第二光强值；
    将所述第一光强值和所述第二光强值进行比较，若所述第一光强值和所述第二光强值之间差值的绝对值超过预设阈值，则发放该事件，并通过所述采样装置中的所述电路记录当前感受野的当前光强值。
  8. 一种仿视网膜中央凹与外周的联合采样装置，其特征在于，所述装置包括：
    获取模块，用于获取当前场景中不同空间位置的光强变化信息和物体纹理信息；
    编码模块，用于根据预设编码方式对所述获取模块获取的所述光强变化信息和所述物体纹理信息进行编码，得到与光信号对应的脉冲信号；以及
    在通过控制模块在动态感知模式和纹理感知模式之间进行自动切换的情况下，使用所述预设编码方式进行编码，并输出与所述脉冲信号对应的异步脉冲信号；
    解码模块，用于对所述编码模块编码出的所述异步脉冲信号进行解码，分离出变化信息编码数据和纹理信息编码数据；
    分析模块，用于根据所述解码模块分离出的所述变化信息编码数据和所述纹理信息编码数据进行视觉任务分析，得到对应的分析结果，所述视觉任务包括感知当前场景变化的任务和重构图像视频的任务。
  9. 一种计算机设备，包括存储器和处理器，所述存储器中存储有计算机可读指令，所述计算机可读指令被所述处理器执行时，使得所述处理器执行如权利要求1至7中任一项权利要求所述联合采样方法的步骤。
  10. 一种存储有计算机可读指令的存储介质，所述计算机可读指令被一个或多个处理器执行时，使得一个或多个处理器执行如权利要求1至7中任一项权利要求所述联合采样方法的步骤。
PCT/CN2021/077573 2021-02-08 2021-02-24 一种仿视网膜中央凹与外周的联合采样方法及装置 WO2022165873A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202110170050.4 2021-02-08
CN202110170050.4A CN113014805B (zh) 2021-02-08 2021-02-08 一种仿视网膜中央凹与外周的联合采样方法及装置

Publications (1)

Publication Number Publication Date
WO2022165873A1 true WO2022165873A1 (zh) 2022-08-11

Family

ID=76384413

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2021/077573 WO2022165873A1 (zh) 2021-02-08 2021-02-24 一种仿视网膜中央凹与外周的联合采样方法及装置

Country Status (2)

Country Link
CN (1) CN113014805B (zh)
WO (1) WO2022165873A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115246559A (zh) * 2022-07-22 2022-10-28 鄂尔多斯市国源矿业开发有限责任公司 一种工业皮带纵撕识别的方法

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015190013A1 (ja) * 2014-06-10 2015-12-17 オリンパス株式会社 画像処理装置、撮像装置、顕微鏡システム、画像処理方法及び画像処理プログラム
CN110427823A (zh) * 2019-06-28 2019-11-08 北京大学 基于视频帧与脉冲阵列信号的联合目标检测方法与装置
CN111008647A (zh) * 2019-11-06 2020-04-14 长安大学 一种样本提取及基于空洞卷积与残差链接的图像分类方法

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9294475B2 (en) * 2013-05-13 2016-03-22 Hoyos Labs Ip, Ltd. System and method for generating a biometric identifier

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2015190013A1 (ja) * 2014-06-10 2015-12-17 オリンパス株式会社 画像処理装置、撮像装置、顕微鏡システム、画像処理方法及び画像処理プログラム
CN110427823A (zh) * 2019-06-28 2019-11-08 北京大学 基于视频帧与脉冲阵列信号的联合目标检测方法与装置
CN111008647A (zh) * 2019-11-06 2020-04-14 长安大学 一种样本提取及基于空洞卷积与残差链接的图像分类方法

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
KONG DELEI , FANG ZHENG: "A Review of Event-based Vision Sensors and Their Applications", INFORMATION AND CONTROL, vol. 50, no. 1, 22 September 2020 (2020-09-22), pages 1 - 19, XP055956411, ISSN: 1002-0411, DOI: 10.13976/j.cnki.xk.2021.0069 *
ZHU LIN; DONG SIWEI; HUANG TIEJUN; TIAN YONGHONG: "A Retina-Inspired Sampling Method for Visual Texture Reconstruction", 2019 IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO (ICME), 8 July 2019 (2019-07-08), pages 1432 - 1437, XP033590394, DOI: 10.1109/ICME.2019.00248 *

Also Published As

Publication number Publication date
CN113014805B (zh) 2022-05-20
CN113014805A (zh) 2021-06-22

Similar Documents

Publication Publication Date Title
CN110889672B (zh) 一种基于深度学习的学生打卡及上课状态的检测系统
CN105681787A (zh) 对时空信号进行编码的方法和装置
Bertalmío Vision models for high dynamic range and wide colour gamut imaging: techniques and applications
US10909424B2 (en) Method and system for object tracking and recognition using low power compressive sensing camera in real-time applications
US11985439B2 (en) Cone-rod dual-modality neuromorphic vision sensor
EP4261778A1 (en) Posture estimation method related apparatus
CN110427823B (zh) 基于视频帧与脉冲阵列信号的联合目标检测方法与装置
CN110880163A (zh) 一种基于深度学习的弱光彩色成像方法
CN110555527A (zh) 延时摄影视频的生成方法及设备
WO2022165873A1 (zh) 一种仿视网膜中央凹与外周的联合采样方法及装置
CN113688741A (zh) 一种基于事件相机和视觉相机协同的运动训练评估系统及方法
CN115484410A (zh) 基于深度学习的事件相机视频重建方法
CN113570689A (zh) 人像卡通化方法、装置、介质和计算设备
US11979660B2 (en) Camera analyzing images on basis of artificial intelligence, and operating method therefor
CN114067435A (zh) 一种基于伪3d卷积网络与注意力机制的睡眠行为检测方法和系统
CN116664694A (zh) 图像亮度获取模型的训练方法、图像获取方法及移动终端
CN116091337B (zh) 一种基于事件信号神经编码方式的图像增强方法及装置
CN115410261B (zh) 面部识别异构数据关联解析系统
KR20200111104A (ko) 인공지능에 기반하여 영상을 분석하는 카메라 및 그것의 동작 방법
US11895308B2 (en) Video encoding and decoding system using contextual video learning
CN105491336B (zh) 一种低功耗图像识别模块
CN110211146B (zh) 视交叉仿真的视频前景分割方法及装置
CN113378923A (zh) 图像生成装置获取方法及图像生成装置
CN115115526A (zh) 图像处理方法及装置、存储介质和图形计算处理器
CN112200816A (zh) 视频图像的区域分割及头发替换方法、装置及设备

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21923883

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21923883

Country of ref document: EP

Kind code of ref document: A1