Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
  • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
  • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
  • G06N3/065—Analogue means

Definitions

• Certain aspects of the present disclosure generally relate to neural system engineering and, more particularly, to a method and apparatus for unsupervised training of input synapses of primary visual cortex cells and other neural circuits.
• Image recognition and motion detection systems can be divided into those based on machine vision (i.e., Artificial Intelligence (AI)) techniques and those utilizing visual cortex techniques (i.e., biologically plausible systems).
• machine vision i.e., Artificial Intelligence (AI)
• visual cortex techniques i.e., biologically plausible systems.
• the machine vision systems have well established methods of training, but poor recognition accuracy. For example, distinguishing a dog from a cat remains a challenging task for machine vision systems with a 50/50 outcome.
• biologically plausible systems use a human visual cortex structure. Methods based on these systems promise to be more accurate than the machine vision systems. However, the training methods for biologically plausible systems that lead to their self-organization are not well developed. This is due to a poor understanding of the visual cortex organization and self-training methods.
• the electrical circuit generally includes a plurality of Retinal Ganglion Cell (RGC) circuits, wherein each of the RGC circuits generates, at an output, a sum of weighted inputs from receptor circuits associated with that RGC circuit, a plurality of primary visual cortex cell (VI) circuits, wherein each of the VI circuits generates another sum of weighted outputs of a subset of the RGC circuits, and a circuit configured to adjust weights applied on the outputs for generating the other sum, wherein the adjustment of one of the weights is based on at least one of one of the outputs on which that weight is applied or the other sum.
• RRC Retinal Ganglion Cell
• VI primary visual cortex cell
• Certain aspects of the present disclosure provide a method for implementing a neural system.
• the method generally includes generating, at an output of each Retinal Ganglion Cell (RGC) circuit of a plurality of RGC circuits in the neural system, a sum of weighted inputs from receptor circuits associated with that RGC circuit, generating, by each primary visual cortex cell (VI) circuit of a plurality of VI circuits in the neural system, another sum of weighted outputs of a subset of the RGC circuits, and adjusting weights applied on the outputs for generating the other sum, wherein the adjustment of one of the weights is based on at least one of one of the outputs on which that weight is applied or the other sum.
• RRC Retinal Ganglion Cell
• the apparatus generally includes means for generating, at an output of each Retinal Ganglion Cell (RGC) circuit of a plurality of RGC circuits in the apparatus, a sum of weighted inputs from receptor circuits associated with that RGC circuit, means for generating, by each primary visual cortex cell (VI) circuit of a plurality of VI circuits in the apparatus, another sum of weighted outputs of a subset of the RGC circuits, and means for adjusting weights applied on the outputs for generating the other sum, wherein the adjustment of one of the weights is based on at least one of one of the outputs on which that weight is applied or the other sum.
• RRC Retinal Ganglion Cell
• FIG. 1 illustrates an example neural system in accordance with certain aspects of the present disclosure.
• FIG. 2 illustrates an example model of receptors connected with different types of Retinal Ganglion (RG) cells in accordance with certain aspects of the present disclosure.
• FIG. 3 illustrates an example model of receptors connected with an RG cell that may be ON-cell or OFF-cell depending on a sign of synapse connecting the RG cell and a primary visual cortex (VI) cell in accordance with certain aspects of the present disclosure.
• RG Retinal Ganglion
• FIG. 4 illustrates an example model of connection between receptors and an RG cell, and an example model of connection between RG cells and a VI cell in accordance with certain aspects of the present disclosure.
• FIG. 5 illustrates example operations that may be performed at a neural system for training of synapse weights between RG cells and a VI cell in accordance with certain aspects of the present disclosure.
• FIG. 5A illustrates example components capable of performing the operations illustrated in FIG. 5.
• FIG. 1 illustrates an example neural system 100 with multiple levels of neurons in accordance with certain aspects of the present disclosure.
• the neural system 100 may comprise a level of neurons 102 connected to another level of neurons 106 though a network of synaptic connections 104.
• a network of synaptic connections 104 For simplicity, only two levels of neurons are illustrated in FIG. 1, although more levels of neurons may exist in a typical neural system.
• each neuron in the level 102 may receive an input signal 108 that may be generated by a plurality of neurons of a previous level (not shown in FIG. 1).
• the signal 108 may represent an input current of the level 102 neuron. This current may be accumulated on the neuron membrane to charge a membrane potential. When the membrane potential reaches its threshold value, the neuron may fire and generate an output spike to be transferred to the next level of neurons (e.g., the level 106).
• the transfer of spikes from one level of neurons to another may be achieved through the network of synaptic connections (or simply "synapses") 104, as illustrated in FIG. 1.
• the synapses 104 may receive output signals (i.e., spikes) from the level 102 neurons, scale those signals according to adjustable synaptic weights wj ' p s a tota j num ber of synaptic connections between the neurons of levels 102 and 106), and combine the scaled signals as an input signal of each neuron in the level 106. Every neuron in the level 106 may generate output spikes 110 based on the corresponding combined input signal. The output spikes 110 may be then transferred to another level of neurons using another network of synaptic connections (not shown in FIG. 1).
• the neural system 100 may be emulated by an electrical circuit and utilized in a large range of applications, such as image and pattern recognition, machine learning, motor control, and alike.
• Each neuron in the neural system 100 may be implemented as a neuron circuit.
• the neuron membrane charged to the threshold value initiating the output spike may be implemented, for example, as a capacitor that integrates an electrical current flowing through it.
• the capacitor may be eliminated as the electrical current integrating device of the neuron circuit, and a smaller memristor element may be used in its place.
• This approach may be applied in neuron circuits, as well as in various other applications where bulky capacitors are utilized as electrical current integrators.
• each of the synapses 104 may be implemented based on a memristor element, wherein synaptic weight changes may relate to changes of the memristor resistance. With nanometer feature-sized memristors, the area of neuron circuit and synapses may be substantially reduced, which may make implementation of a very large-scale neural system hardware implementation practical.
• the present disclosure proposes a simplified structure of primary visual cortex (VI) cells and retinal ganglion cells (RGCs) utilized for color vision, wherein the VI cells and RGCs may be implemented as neuron circuits of the neural system 100 from FIG. 1.
• the RGCs may correspond to the neurons 102
• the VI cells may correspond to the neurons 106.
• VI input synapses e.g., the synapses 104 of the neural system 100
• the present disclosure proposes an efficient method of training connectivity between VI and RGC layers of cells that may lead to an autonomous formation of feature detectors (simple cells) within the VI layer.
• the proposed approach may enable a hardware-efficient and biological-plausible implementation of image recognition and motion detection systems.
• the proposed unsupervised training method may utilize simple neuron models for both RGC and VI layers.
• the model simply adds weighted inputs of each cell, wherein the inputs may have positive or negative values.
• the resulting weighted sums of inputs is called activations, wherein the activations may also be positive or negative.
• the weights of each VI cell may be adjusted depending on a sign of corresponding RGC output and a sign of activation associated with that VI cell in the direction of increasing the absolute value of activation.
• the VI weights may be positive or negative.
• the proposed training method of synapse weights may be suitable for efficient implementation in software and hardware. Furthermore, it may run much faster that the Spike Timing Dependant Plasticity (STDP) training approach.
• STDP Spike Timing Dependant Plasticity
• RG cells may be divided into ON-cells and OFF-cells.
• the ON-cells can distinguish objects that are brighter than a background. For example, peripheral vision is all based on ON-cells, so that humans can better see bright spots against a dark background.
• RG OFF-cells can distinguish objects that are darker than a background. It should be noted that illuminating the entire receptive field comprising both RG ON-cells and RG OFF-cells has a limited effect on an RGC firing rate.
• FIG. 2 illustrates an example model 200a of connection between photoreceptors 202 and an RG ON-cell 204, and an example model 200b of connection between photo-receptors 206 and an RG OFF-cell 208 in accordance with certain aspects of the present disclosure.
• the receptor circuits 202 and 206 may be organized as orthogonal arrays of image pixels. Therefore, receptive fields of ON- and OFF-cells may have a rectangular shape (instead of circular).
• each RGC may receive input from nine receptors, wherein input weights associated with the receptors may form a Laplacian filter (i.e., a Laplacian window function may be applied on signals from the receptors), as illustrated in FIG. 2.
• the weights may depend on whether the receptors are connected to ON- or OFF- RG cells, as illustrated in the models 200a and 200b. It should be noted that the RG cells 204 and 208 illustrated in FIG. 2 may not correspond to magno-ganglion cells, which are being able to receive inputs from a much larger number of receptors.
• each RG cell e.g., an RG cell 304 receiving inputs from receptors 302 in a model 300 illustrated in FIG. 3 may be either ON or OFF cell. This may depend on a sign of weight 306 associated with a synapse 308 connecting the RG cell 304 to a VI simple cell 310.
• an input y of the VI cell 310 may be obtained by applying a weight w on input signals .3 ⁇ 4 from the receptors 302 that may be input into the RG cell 304:
• Equation (1) may be rewritten as:
• a neuron model as the one illustrated in FIG. 4 may be utilized.
• a VI cell 414 may sum its weighted inputs 416 from RGCs 412, i.e.: where each of the weights w ⁇ ⁇ from equation (4) may be bipolar for modeling both ON and OFF RGCs.
• an output 418 (activation signal) of the VI cell 414 may be compared with a threshold. If the summation result 418 is above the threshold, then the VI cell 414 is activated (i.e., fires). On the other hand, if the result 418 is below the threshold, then the VI cell 414 is resting.
• weights of RGC-to-Vl connections may be trained.
• the model 400b from FIG. 4 may utilize the following rule.
• the activation of each VI simple cell may be calculated as the weighted sum of the corresponding RGC outputs. If the VI -cell activation exceeds a threshold, then the VI cell may fire. Otherwise, the VI cell may rest (i.e., the VI cell does not generate any signal) and its input weights may not change.
• the activation of each VI simple cell may be calculated as the weighted sum of the corresponding RGC outputs. Then, the weights of each VI cell may be adjusted depending on a sign of corresponding RGC output and a sign of activation of that VI cell activation. In an aspect, if the sign of RGC output and the sign of activation are same, then a positive increment may be added to a weight applied on the RGC output (i.e., the weight may be increased). In another aspect, if the sign of RGC output and the sign of activation are not same, then a positive increment may be subtracted from the corresponding weight (i.e., the weight may be decreased).
• from equations (5)-(6) may be pre-determined.
• may be in the form of 1/2 N , where N may be an integer. Since the activation value z may be represented as a binary number, the product z ⁇
• FIG. 5 illustrates example operations 500 that may be performed at a neural system for training of synapse weights between RG cells and VI cells in accordance with certain aspects of the present disclosure.
• RGC Retinal Ganglion Cell
• a sum of weighted inputs from receptor circuits associated with that RGC circuit may be generated.
• each primary visual cortex cell (VI) circuit of a plurality of VI circuits in the neural system may generate another sum of weighted outputs of a subset of the RGC circuits.
• weights applied on the outputs for generating the other sum may be adjusted, wherein the adjustment of one of the weights may be based on at least one of one of the outputs on which that weight is applied or the other sum.
• the various operations of methods described above may be performed by any suitable means capable of performing the corresponding functions.
• the means may include various hardware and/or software component(s) and/or module(s), including, but not limited to a circuit, an application specific integrate circuit (ASIC), or processor.
• ASIC application specific integrate circuit
• those operations may have corresponding counterpart means-plus-function components with similar numbering.
• operations 500 illustrated in FIG. 5 correspond to components 500A illustrated in FIG. 5A.
• determining encompasses a wide variety of actions. For example, “determining” may include calculating, computing, processing, deriving, investigating, looking up (e.g., looking up in a table, a database or another data structure), ascertaining and the like. Also, “determining” may include receiving (e.g., receiving information), accessing (e.g., accessing data in a memory) and the like. Also, “determining” may include resolving, selecting, choosing, establishing and the like.
• a phrase referring to "at least one of a list of items refers to any combination of those items, including single members.
• "at least one of: a, b, or c" is intended to cover: a, b, c, a-b, a-c, b-c, and a-b-c.
• DSP digital signal processor
• ASIC application specific integrated circuit
• FPGA field programmable gate array signal
• PLD programmable logic device
• a general purpose processor may be a microprocessor, but in the alternative, the processor may be any commercially available processor, controller, microcontroller or state machine.
• a processor may also be implemented as a combination of computing devices, e.g., a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration.
• a software module may reside in any form of storage medium that is known in the art. Some examples of storage media that may be used include random access memory (RAM), read only memory (ROM), flash memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM and so forth.
• RAM random access memory
• ROM read only memory
• flash memory EPROM memory
• EEPROM memory EEPROM memory
• registers a hard disk, a removable disk, a CD-ROM and so forth.
• a software module may comprise a single instruction, or many instructions, and may be distributed over several different code segments, among different programs, and across multiple storage media.
• a storage medium may be coupled to a processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor.
• the methods disclosed herein comprise one or more steps or actions for achieving the described method.
• the method steps and/or actions may be interchanged with one another without departing from the scope of the claims.
• the order and/or use of specific steps and/or actions may be modified without departing from the scope of the claims.
• Computer-readable media include both computer storage media and communication media including any medium that facilitates transfer of a computer program from one place to another.
• a storage medium may be any available medium that can be accessed by a computer.
• such computer-readable media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer.
• any connection is properly termed a computer-readable medium.
• Disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray ® disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.
• computer-readable media may comprise non-transitory computer-readable media (e.g., tangible media).
• computer-readable media may comprise transitory computer- readable media (e.g., a signal). Combinations of the above should also be included within the scope of computer-readable media.
• certain aspects may comprise a computer program product for performing the operations presented herein.
• a computer program product may comprise a computer readable medium having instructions stored (and/or encoded) thereon, the instructions being executable by one or more processors to perform the operations described herein.
• the computer program product may include packaging material.
• Software or instructions may also be transmitted over a transmission medium.
• a transmission medium For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of transmission medium.
• DSL digital subscriber line
• modules and/or other appropriate means for performing the methods and techniques described herein can be downloaded and/or otherwise obtained by a user terminal and/or base station as applicable.
• a user terminal and/or base station can be coupled to a server to facilitate the transfer of means for performing the methods described herein.
• various methods described herein can be provided via storage means (e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or floppy disk, etc.), such that a user terminal and/or base station can obtain the various methods upon coupling or providing the storage means to the device.
• storage means e.g., RAM, ROM, a physical storage medium such as a compact disc (CD) or floppy disk, etc.
• CD compact disc
• floppy disk etc.
• any other suitable technique for providing the methods and techniques described herein to a device can be utilized.

Landscapes

• Engineering & Computer Science (AREA)
• Physics & Mathematics (AREA)
• Health & Medical Sciences (AREA)
• Life Sciences & Earth Sciences (AREA)
• Biomedical Technology (AREA)
• Biophysics (AREA)
• Theoretical Computer Science (AREA)
• Evolutionary Computation (AREA)
• General Engineering & Computer Science (AREA)
• Data Mining & Analysis (AREA)
• Artificial Intelligence (AREA)
• General Health & Medical Sciences (AREA)
• Molecular Biology (AREA)
• Computing Systems (AREA)
• Computational Linguistics (AREA)
• General Physics & Mathematics (AREA)
• Mathematical Physics (AREA)
• Software Systems (AREA)
• Neurology (AREA)
• Image Analysis (AREA)
• Image Processing (AREA)
• Measuring Or Testing Involving Enzymes Or Micro-Organisms (AREA)
• Heterocyclic Carbon Compounds Containing A Hetero Ring Having Oxygen Or Sulfur (AREA)

Priority Applications (4)

Applications Claiming Priority (2)

Publications (1)

Family

ID=46208849

Family Applications (1)

Country Status (6)

Cited By (1)

Families Citing this family (2)

Citations (1)

Family Cites Families (13)

• 2011
  • 2011-05-25 US US13/115,154 patent/US8583577B2/en active Active
• 2012
  • 2012-05-25 JP JP2014512164A patent/JP6113719B2/ja active Active
  • 2012-05-25 WO PCT/US2012/039704 patent/WO2012162663A1/en not_active Ceased
  • 2012-05-25 CN CN201280024956.4A patent/CN103548042B/zh active Active
  • 2012-05-25 KR KR1020137034213A patent/KR101549767B1/ko not_active Expired - Fee Related
  • 2012-05-25 EP EP12725998.4A patent/EP2715620B1/en active Active
• 2016
  • 2016-02-26 JP JP2016035857A patent/JP6117392B2/ja active Active

Patent Citations (1)

Non-Patent Citations (5)

Also Published As

Similar Documents

Legal Events