KR20170031986A

KR20170031986A - Visual cortex inspired circuit apparatus based on the multi-sensor and object searching system, method using the same

Info

Publication number: KR20170031986A
Application number: KR1020150129639A
Authority: KR
Inventors: 한일송; 김동욱
Original assignee: 한국과학기술원
Priority date: 2015-09-14
Filing date: 2015-09-14
Publication date: 2017-03-22
Also published as: KR101738422B1

Abstract

The present invention relates to a visual neural circuit apparatus based on a multi-sensor and object search system and method using the same, and more particularly, to a visual neural circuit apparatus based on a multi-sensor and object search system and method using the same, in which a high resolution image sensor and a low resolution image sensor are associated. In addition, the present invention provides the object search system and the method thereof, in which the object search system includes: a trans-conductance neural circuit for performing a brain-imitating visual information directional processing on an image inputted from a low resolution image input unit and outputting the processed image; a detection unit for extracting an interest region image from the low resolution image outputted from the trans-conductance neural circuit; and an object search unit for extracting an object corresponding to an interest region detected by the detection unit from an image inputted from a high resolution image input unit.

Description

TECHNICAL FIELD [0001] The present invention relates to an object-oriented visual nerve circuit (OBE)

The present invention relates to an optic neural circuit device and an object search system and method using the same, and more particularly, to an optic neural circuit device in which a high-resolution image sensor and a low-resolution image sensor are connected and an object search system and method using the same.

In general, brain science (Brain Science) is an applied discipline that explores the mysteries of the brain and explores the physical and mental functions of human beings. It is not only a basic science field such as mathematics, physics, chemistry, And cognitive science to reveal the mysteries of the brain and thereby explore in depth the overall physical and mental functions of human beings.

Neural networks or neuromorphic frameworks are circuits designed to resemble human brain or neuron response and are semiconductor chips that have the ability to think like humans, To communicate and to express and memorize information. It is a core technology of artificial intelligence computer. It is mainly used in fields of speech recognition, character recognition, image processing, understanding of natural language, Research is underway.

Brain information processing is accomplished through the physiological activation of networks in which individual cells that perform relatively simple functions called neurons are configured with certain constraints. The brain is made up of cells just like any other organs in the body, but unlike other organs, the network of cells is a major component of function.

System Neuroscience is a field of neuroscience that seeks to understand the spatial and temporal distribution of activation of cell populations, and its representative field is visual neuroscience. Understanding the input and output connection patterns of the retina, lateral geniculate body, superior colliculus, and visual cortex cells of the visual system, receiving signals from cells, The task of visual neuroscience is to identify how the biological vision is achieved by studying the processing, storage, and conversion methods.

To this end, Korean Patent Laid-Open No. 10-2006-0089487 discloses a technique for providing a user with a higher quality image. As such, in recent years, much research has been conducted on a vision device using a living body based vision and visual nerve change. However, in actual use, most of them are lacking in accuracy such as human or animal vision, and are limited due to miniaturization trends and environmental changes.

Korean Patent Publication No. 10-2006-0089487

Some embodiments of the present invention mimic the visual sensing and recognition functions of the cranial nerves to detect and recognize the condition of a vehicle occupant, particularly a driver, including a vehicle, to alert the user to drowsiness or severe fatigue, A high-resolution image that enables effective and timely real-time recognition to alert the driver or to communicate the situation to the safety system in the vehicle, and to enable the operator to identify the driver or object effectively in any environment or complex background, A sensor and a low-resolution image sensor are applied to a neural circuit that imitates a brain visual nerve, and an object search system and method using the same.

In addition, some embodiments of the present invention are directed to applying a neuron response of a visual cortex to a visual stimulus, applying the neuron response to a heterodyne sensor output to a neuron mimetic principle to separate a proximity supervisor, such as a driver, To a distributed processing circuit having a parallel output circuit configuration, thereby realizing an visual nerve circuit similar to a human visual nerve, and an object searching system and method using the same.

Further, in some embodiments of the present invention, the proximity management object / area such as the driver's face is selected, the reference position of the configuration of the object / area is recognized, and the state change is recognized based on the fixed mutual position between the management object- Driver status monitoring that recognizes distraction or drowsiness phenomenon by detecting change of specific part situation on the driver's face such as around the mouth, or object using visual nerve circuit device that realizes real-time change of abnormal state with respect to a specific situation of target screen Search system and method therefor.

According to an aspect of the present invention, there is provided an image processing apparatus including: a low-resolution image input unit receiving an image; And a transconductance neural circuit for outputting the image input from the low-resolution image input unit by performing a brain imitating time information directional processing.

The low-resolution image input unit according to an aspect of the present invention is one of an infrared sensor, a phototransistor, and a photodiode.

According to another aspect of the present invention, the transconductance neural network includes a transconductance unit for converting a current corresponding to an image input through a low-resolution image input unit into a voltage through linear current conversion; And a buffering unit for performing a weighting operation on the neural connection loop and a weighting operation and a buffering operation on the current signal of the low-resolution input unit by amplifying the voltage of the transconductance unit by outputting a current source.

Further, the transconductance part of one aspect of the present invention comprises a linear conductance circuit whose voltage is controlled by a pair of NMOS (N-Metal Oxide Semiconductor) transistors.

According to another aspect of the present invention, the buffering unit includes: a forward buffer for externally providing an output current; And a negative directional buffer that receives the output current from the outside.

According to an aspect of the present invention, the apparatus further includes a weight control unit for applying a weight to the buffering unit through a terminal of the weighting unit of the buffering unit to emphasize an image of a region of interest.

According to another aspect of the present invention, there is provided a transconductance neural circuit for outputting an image input from a low-resolution image input unit by performing a brain imitating visual information directional processing, A detection unit for extracting a region of interest from a low-resolution image output from the transconductance neural circuit; And an object search unit for extracting an object corresponding to the region of interest detected by the detection unit in the image input from the high resolution image input unit.

According to another aspect of the present invention, there is further provided a weight control unit for adjusting a weight through a weight connection terminal of the transconductance neural circuit to emphasize a region of interest.

In another aspect of the present invention, the low-resolution image input unit is one of an infrared sensor, a phototransistor, and a photodiode.

In another aspect of the present invention, the high-resolution image input unit is a CMOS sensor.

According to another aspect of the present invention, the low-resolution image sensor and the high-resolution image sensor are integrated to constitute an integrated input unit.

According to another aspect of the present invention, there is further provided an input control unit for time-division-controlling the low-resolution image sensor and the high-resolution image sensor of the integrated input unit to output the low-resolution image and the high-

According to another aspect of the present invention, there is further provided a situation determination unit for determining an abnormal state of the object extracted by the object search unit.

According to another aspect of the present invention, there is further provided a warning unit that alerts a user when an abnormal state occurs, and the situation determination unit outputs the warning to the warning unit when an abnormal condition occurs.

According to another aspect of the present invention, there is further provided a safety system for causing a vehicle control system to take an action when an abnormal condition occurs, and the situation determination unit outputs the abnormal condition to the safety system when an abnormal condition occurs.

According to still another aspect of the present invention, there is provided a method for controlling a brain, comprising the steps of: (A) processing an image input through a low-resolution image input unit by a transconductance neural circuit; (B) extracting a region of interest when the brain shape visual information directional processing is completed through the transconductance neural circuit; And (C) searching for an object corresponding to a region of interest detected by the detection unit on an image received from the object search unit from the high resolution image sensor.

According to another aspect of the present invention, in the step (A), the weight control unit controls the weight of the transconductance neural circuit to emphasize the region of interest.

According to another aspect of the present invention, (D) the situation determination unit determines whether an object is in an abnormal state by determining the state of the object; And (E) the situation determination unit further includes the step of providing an alarm through the alarm unit or taking a safety measure through the safety system when an indication of an abnormal condition is found.

The present invention aims at implementing and utilizing a large-capacity imitation system of the brain as a circuit device or a miniaturization system, simultaneously distributing the visual neural network structured with a parallel module to integrate the output current source signals, It is possible to minimize the environmental influence of the visual signal itself so that it can be imitated with the accuracy close to that of the visual eye nerve. In order to adapt and respond to the light intensity characteristic of the visual stimulus, It is possible to provide an object search system and method for selectively separating based on sensor proximity objects, such as a driver, which is a recognition target based on recognition capability close to a robust environment response capability of brain visual intelligence.

This is because the driver state monitoring (DSM) technology is limited to the range of the driver's face status information (facial expression or change status) and the noise range when the infrared image is based on the infrared filter. The extracted driver face can analyze various original states, so it can provide various complex state and situation recognition.

FIG. 1 is a diagram showing experimental results of visual cortex of human and animal, which is a principle of the present invention.
FIG. 2A is a diagram for explaining a basic concept of designing a brain visual cortical neuron as a neuromorphic neural network based on the contents of an animal brain clinical experiment of FIG. 1 as a CMOS ASIC electronic circuit. FIG. Circuit simulation result.
3A is an embodiment in which the variable resistor shown in FIG. 2A is implemented as a MOSFET, and FIGS. 3B and 3C are views illustrating an embodiment in which the variable resistor shown in FIG. 2A is implemented as a transconductance.
FIG. 4A is a diagram illustrating a neural connection-based filter operation model of FIG. 2A, and FIG. 4B is a diagram for explaining brain imitation visual recognition based on the model.
FIG. 5A illustrates an optic neural circuit device including a transconductance neural circuit combined with a phototransistor according to an embodiment of the present invention, and FIG. 5B illustrates an optic neural circuit device including a transconductance neural circuit coupled with a photodiode.
Figure 6 is a diagram illustrating an imitating implementation of the cerebral cortical neural network of Figures 2a-3c based on the unit design circuitry of Figures 5a and 5b.
FIG. 7 shows the principle of performance enhancement through hybrid sensor hybrid type through the performance characteristics of a general visible light sensor and an infrared light type infrared-only sensor.
FIG. 8 shows an object search system using a multi-sensor-based visual neural network device according to an embodiment of the present invention.
FIG. 9A shows a process of calculating a mask for selecting a specific management target area by applying a transconductance circuit to the output of the low resolution image sensor. FIG. 9B shows the process of applying a mask to select a specific management target area, Fig.
10 is a block diagram illustrating an object search system according to another embodiment of the present invention.
11 is a conceptual diagram of a search system according to another embodiment of the present invention.
12 is a block diagram illustrating an object search system according to another embodiment of the present invention.
13 is a flowchart of an object searching method using a multi-type sensor-based visual neural network apparatus according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings, which will be readily apparent to those skilled in the art. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the embodiments of the present invention in the drawings, parts not related to the description are omitted, and like parts are denoted by similar reference numerals throughout the specification.

Throughout the specification, when a part is referred to as being "connected" to another part, it includes not only "directly connected" but also "electrically connected" with another part in between . Also, when an element is referred to as "including" an element, it is to be understood that the element may include other elements as well as other elements, And does not preclude the presence or addition of one or more other features, integers, steps, operations, components, parts, or combinations thereof.

FIG. 1 shows the results of a visual cortex experiment of human and animal, which is a principle of the present invention. The upper left side is a light source (a light source having different directional inclination) and the right side is a cat Cats) is a response signal of visual cortex neurons, and Brain Visual Recognition is a drawing that demonstrates the clinical experiments of Nobel Prize winner Hubel and Wiesel who proved that the brain neural network is one of the basic principles of selective reaction to directionality.

Referring to FIG. 1, brain visual recognition is a selective response to the directionality of the brain neural network. The brain visual cortex is composed of neurons that respond to each direction at a specific sensing position (corresponding to a pixel in the image), as revealed by cat or monkey experiments, and the present invention provides a neuron function that selectively responds to such directionality , It realizes robust detection ability against animal environment change and greatly improves vulnerability to brightness fluctuation which is the limit of conventional image recognition technology.

FIG. 2A is a diagram for explaining a basic concept of designing a brain visual cortical neuron as a neuromorphic neural network based on the contents of an animal brain clinical experiment of FIG. 1 as a CMOS ASIC electronic circuit. FIG. 1 is a diagram showing the results of a circuit simulation, in which the characteristics and functions of the brain cortical neurons of Fig. 1 are imitated. Fig.

FIG. 2A illustrates the principle and feasibility of implementing the present invention to imitate a phenomenon occurring in the animal's brain nerve cell of FIG. 1 electronically or by an algorithm. The neuronal morphic neural network functions as a neuron of the visual cortex in a CMOS transistor ASIC electronic Circuit.

And, Figure 2b shows that the performance of electronic circuit neuromorphic neurons can mimic similar to that of animals.

3A is an embodiment in which the variable resistor shown in FIG. 2A is implemented as a MOSFET, and FIGS. 3B and 3C are views illustrating an embodiment in which the variable resistor shown in FIG. 2A is implemented as a transconductance.

As shown in FIG. 3A, the non-saturation operating region of the MOSFET can be used as a resistive or conductive element by changing the drain-source terminal current according to the drain-source terminal voltage. The current-voltage relationship of the non-saturation region MOSFET is as follows.

(1)

Ids = k (W / L) [(Vgs-Vt) Vds-Vds 2/2]

k is a fixed constant physically determined in the MOSFET fabrication process, W is the width of the transistor gate, and L is the length of the transistor gate, which determines the conductivity or resistivity range of the transistor. Vt is the threshold voltage of the transistor, and the transistor used in the experiment of this embodiment is an n-channel depletion mode. Even if the polarity or the threshold voltage is different, the principle of the transconductance circuit of Fig. There is no.

In Equation 1 [Vds ^2/2] entry is the cause of the non-linear resistance or conductive properties, the linear resistor is a non-linear current source next to the current in the equivalent circuit shown in Fig. 3a - in a non-linear characteristic, such as voltage test pictures to a mathematical computation circuit .

FIG. 3B is an implementation example of a variable resistor used in the nerve connecting loop mimic of FIG. 2A as a transconductance circuit, and has the following operational principle of Equation (2).

(2)

Ids = Ids1 + Ids2

= K (W / L) [ (Vgs-Vt) Vds-Vds 2/2] + k (W / L) [(Vds-Vt) Vds-Vds 2/2]

= K (W / L) [ (Vgs-Vt) Vds-Vds 2/2 + (Vds-Vt) Vds-Vds 2/2]

= k (W / L) (Vgs-2Vt) Vds

Ids1 is the drain-source current of the transistor M1, Ids2 is the drain-source current of the transistor M2, and the total current Ids is the second-order nonlinear current component 2 * to produce a [Vds - Vds] wherein the offset (Vds ^2/2)]. The principle of mathematical 2 is confirmed from the characteristic test results showing the current-voltage linearity of FIG. 3B.

FIG. 3C shows a configuration in which the gate voltage circuit to be applied to the transistor M2 is supplemented so that the Vt of the transistors M1 and M2 can be applied to an enhancement mode, not a depletion mode, on the basis of FIG. 3B or Equations 1 and 2 to be.

3C, the circuit has been proposed as a linear conductance circuit whose voltage is controlled by a pair of N metal oxide semiconductor (NMOS) transistors M1 and M2, and the conductance of the MOS transistor is one of the essential components in the design of an analog circuit , These circuits can be applied to a variety of neural networks ranging from nasps to neurons at runtime.

Thus, referring to FIG. 3C, VBIAS is made by M3, such as PMOS (P Metal Oxide Semiconductor), with a bias value to drive the same as a diode.

Thus, the linear resistance or controlled linear transconductance circuit of Figs. 3a to 3c is not only simulated, but also verified by test ASIC chip fabrication and experimentation as shown in the figure. The linear resistance or conductivity of a transistor can be applied to components of the Hodgkin-Huxley formalism modeling the dynamic dynamic characteristics of neuronal link loop weighting operations and the generation of asynchronous spike signals in neurons.

FIG. 4A is a diagram illustrating a neural connection-based filter operation model of FIG. 2A, and FIG. 4B is a diagram for explaining brain imitation visual recognition based on the model.

The directional selectable filter of FIG. 4A can be implemented as a result of rotating according to the directional angle, and can be applied with arbitrary expandability such as 1, 2, 4, and 8 directionality according to applications. FIG. to be.

Based on the principle of FIGS. 2A to 3C, a directional selection filter as shown in FIG. 3A is applied by the directional selection processing of the function shown in FIG. 2B as an example of image recognition showing robust performance even in a limited environment such as darkness, He showed that he recognized bicycle racers in dark tunes as well.

FIG. 5A illustrates an optic neural circuit device including a transconductance neural circuit combined with a phototransistor according to an embodiment of the present invention, and FIG. 5B illustrates an optic neural circuit device including a transconductance neural circuit coupled with a photodiode.

Referring to FIG. 5A, a multi-sensor-based visual nerve circuit device includes a low-resolution image input unit 10 and a transconductance nerve circuit 20.

The low-resolution image input unit 10 receives an image having a lower resolution than a high-resolution image input unit composed of a CMOS image sensor. The low-resolution image input unit 10 may be an infrared sensor, a phototransistor, or the like.

The transconductance neural circuit 20 includes a transconductance unit 21 (M1 and M2) for converting a current corresponding to an image input through the low-resolution image input unit 10 into a voltage through linear current conversion, And a buffering unit 25 (M3 and M4) for performing a weighting operation on the neural connection loop and a weighting operation and a buffering operation on the current signal of the phototransistor.

The transconductance section 21 may be connected in parallel vertically and symmetrically, may be composed of a transconductor, or may be an equivalent circuit of a differential transconductance amplifier.

The transconductance section 21 may be formed of at least one or at least one of a BiCMOS (Bipolar CMOS) transistor, a JFET (junction field-effect transistor), a metal-oxide semiconductor field-effect transistor (MOSFET), and a GaAs MESFET As shown in FIG.

The transconductance section 21 may be constituted by a linear conductance circuit whose voltage is controlled by a pair of NMOS (N-Metal Oxide Semiconductor) transistors M1 and M2. This can be applied to a variety of neural networks ranging from nasps to neurons at runtime.

On the other hand, the buffering unit 25 buffers the output current of the transconductance unit 21 to prevent the feedback current from flowing backward, amplifies the input current for the same frame as the pattern of the object, and applies a weight have.

At this time, the weight value may be given to a pixel of an image or an image if the pixel of the image or image is brighter than the predetermined brightness. For example, if the input of the visual stimulus is greater than a predetermined threshold, weighting may be applied. At this time, the visual stimulus may be a contrast, and the predetermined threshold value may be a contrast of a certain brightness.

In such a multi-sensor-based neural circuit device, the transconductance neural circuit 20 is modeled with a variable resistance or a conductance value as shown in FIGS. 3A to 3C. At this time, a low-resolution image of M * N pixels can be obtained by constructing a plurality of low-resolution image input units 10 and a corresponding conductance neural circuit 20 in the vertical and horizontal directions.

Referring to FIG. 5B, the multi-sensor-based visual neural circuit device includes a low-resolution image input unit 10 'and a transconductance neural circuit 20'.

The low resolution image input unit 10 'receives a low resolution image having a lower resolution than a high resolution image input unit composed of a CMOS image sensor, and a photodiode is displayed in FIG. 5b.

The transconductance neural circuit 20 'includes transconductance sections 21-1' and 21-2 'for converting the current corresponding to the image input through the low-resolution image input section 10' into a voltage through linear current conversion. ) M1, M2, M1A, and M2A, amplifies the output of the current source buffer, and performs weighting operation and weighting operation on the current signal of the photodiode, and buffering units 25-1 'and 25-2 ') (M3 + M4 + M5, M3A + M4A + M5A + M6A + M7A).

The transconductance sections 21-1 'and 21-2' may be connected in parallel in a vertically symmetrical manner, and may be formed of a transconductor or an equivalent circuit of a differential transconductance amplifier.

The transconductance sections 21-1 'and 21-2' may be formed of any one of a BiCMOS transistor, a JFET (junction field-effect transistor), a MOSFET (metal-oxide semiconductor field-effect transistor), a GaAs MESFET FET), or a combination of at least one of them.

Such transconductance sections 21-1 'and 21-2' may be constituted by a pair of NMOS (N-Metal Oxide Semiconductor) transistors M1 and M2 or by a linear conductance circuit whose voltage is controlled by M1A and M2A have. This can be applied to a variety of neural networks ranging from nasps to neurons at runtime.

On the other hand, the buffering unit 25 'buffers the output currents of the transconductance units 21-1' and 21-2 'so that the feedback current does not flow backward, and the input current for the same frame as the pattern of the object is amplified The weight can be applied.

The buffering unit 25 'includes a forward buffer 25-1' (M3 + M4 + M5) for providing an output current to the outside, and a negative direction buffer 25-2 ' (M3A + M4A + M5A + M6A + M7A).

In such a multi-sensor-based neural circuit device, the transconductance neural circuit 20 'is modeled with a variable resistance or a conductance value as shown in FIGS. 3A to 3C. At this time, a low-resolution image of M * N pixels can be obtained by constructing a plurality of low-resolution image input units 10 'and a corresponding conductance nerve circuit 20' in the vertical and horizontal directions.

The transconductance circuits 20 and 20 'process the image inputted through the low-resolution image input unit 10 in the direction of the brain shape visual information and output the processed images.

When the above-described transconductance circuits 20 and 20 'are subjected to a charge processing, an image in which the region of interest is emphasized can be obtained.

In the design details of FIGS. 5A and 5B, there is no limit to the selective application of FIG. 3B and FIG. 3C to the voltage or conductive implementations, and the bi-directionality may be optional and only the forward direction may be selected as needed. This is to provide flexibility and selectivity of the overall system implementation design using the FIG. 4a of multiple arrays of directional selectable filters of FIG.

Figure 6 is a diagram illustrating an imitating implementation of the cerebral cortical neural network of Figures 2a-3c based on the unit design circuitry of Figures 5a and 5b.

The neural connection loop function, which is an electrical model of the neural network, is modeled by a variable resistance or conductance value, and the result of each operation is shown in FIG. 6 by applying a capacitor to the current operation summing unit, Perform cumulative summation and output in the form of voltage.

FIG. 7 shows the principle of performance enhancement through hybrid sensor hybrid type through the performance characteristics of a general visible light sensor and an infrared light type infrared-only sensor.

FIG. 7 is a diagram illustrating a condition (a) of a general high-quality visible light ray sensor and an infrared ray sensor, and a state example (b, c) when wearing sunglasses, (D) can be recognized even if the sunglasses are worn by analyzing the driver based on the general image sensor.

The left side of FIG. 7 (a) is an image of a general visible light sensor, and the right side is an image of an infrared ray-dedicated sensor.

The left common visual ray sensor image shows the driver's face in a realistic and detailed manner, and the image of the infrared ray-dedicated sensor on the right side omits a detailed skin condition and the hair follicle under the skin appears like a whisker.

7 (b) and 7 (c) show similar results in the same bus driver's seat during the daytime of the same weather (B) and infrared-only sensor image (c). In a situation where sunglasses can not transmit infrared rays, the infrared sensor image has almost no image information of the corresponding region, and the general sensor detects the spectacle portion There are various kinds of image information that can be obtained and the face status information is abundant.

On the other hand, infrared-only sensors show little background behind the driver, and general sensors show complex background information. FIG. 7 (d) is a general sensor image in which a driver wearing a sunglass is yawning due to drowsiness. In a simulated action in which a research institute researcher watches and records a recorded video in the event of a bus accident, The directional selection signal processing results are analyzed to show the performance of the system recognizing whether or not yawn.

Such a general sensor-based image has a wide range of use, and the use of an infrared-dedicated sensor can be stable and efficient in processing complex backgrounds as shown in FIG. 7 (d).

FIG. 8 shows an object search system using a multi-sensor-based visual neural network device according to an embodiment of the present invention.

8, an object search system according to an embodiment of the present invention includes a low-resolution image input unit 110, a high-resolution image input unit 115, a transconductance neural network 120, a weight control unit 130, A detection unit 140, and an object search unit 150. The low-resolution image input unit 110 and the transconductance neural circuit 120 have been described above, and a detailed description thereof will be omitted.

Here, the high-resolution image input unit 115 may be a CMOS image sensor or the like and has a higher resolution than the low-resolution image input unit 110, so that a clear image can be obtained.

The weight control unit 130 is connected to the weight connection terminal of the buffering unit 125 of the transconductance neural circuit 120 and adjusts a weight value input to the buffering unit 125. At this time, the weight value may be given to a pixel of an image or an image if the pixel of the image or image is brighter than the predetermined brightness. For example, if the input of the visual stimulus is greater than a predetermined threshold, weighting may be applied. At this time, the visual stimulus may be a contrast, and the predetermined threshold value may be a contrast of a certain brightness.

If the weight control unit 130 applies different weights to the weight terminals of the buffering unit 125, the desired image can be emphasized as shown in FIG. 9A.

In this regard, FIG. 9A shows a mask for extracting a human head from a low-resolution image of a human using an infrared sensor, a phototransistor, or a photodiode by an operation that imitates the brain by adjusting the weight of the weight control unit 130, This is the simulation result of the neural network.

Directional computation, which is the ultimate visual intelligence stage of the brain in the optic nerve circuit device through the nerve connection loop, consists of a relatively small 100 to 200 connection rings that are operated locally, that is, a sensor output of 100 to 200 and a neural connection ring weighting operation It is necessary in pixel units.

In addition, since the directional response of the cerebral visual cortex can be realized with 6 to 8 kinds of connection weights, 6 to 8 kinds of linking weights are calculated for each pixel, It can be configured as a simple structure of buffering.

In each pixel, the weighted link loop operation is a circuit that uses a common portion to avoid duplication or repetition, and replicates the output buffering, that is, the current duplication transistor alone. Through the circuit configuration, it is possible to implement the connection loop weighting operation of 100 ~ 200 times required for each pixel by 6 ~ 8 connection loop weighting operation. The number of weights to be connected here is n, which can be in the range of 6 to 8, depending on the application.

On the other hand, the detection unit 140 detects a desired region as a mask in an output image of the transconductance circuit 120 in which a desired region is emphasized by the weight control unit 130. In this case, the detected region is a bright portion in the second image in FIG. 9A, a bright portion in the second image in FIG. 9B, and a bright portion in the second image in FIG.

When the mask is detected by the detection unit 140, the object search unit 150 extracts a desired image by applying a mask to the image acquired through the high resolution image sensor 115. [

At this time, the image extracted by the object search unit 150 is the fourth image of FIGS. 9B and 9C.

FIG. 9B is a graph showing the relationship between a low-resolution heterodyne sensor (infrared sensor, phototransistor, photodiode, etc.) by applying a directional selectivity of a cerebral cortical neural network identified by a multiplication operator for calculating a nerve link loop and an experiment ) To generate a mask capable of separating only a proximity target area from a sensor such as a driver so as to remove a background part from the image output from the high resolution image sensor and extract a driver's head image .

In order to extract only the objects in a specific area, a stereo camera or near-infrared image processing can be applied. However, the stereo camera has complicated structure operation, and its implementation and stability are limited. Near infrared ray images are used for various states There is a limit to monitoring.

9B is an embodiment of an object search system in which a high-resolution image sensor and a low-resolution image sensor are fused, extracts a person from mask creation of a low-resolution based proximity management region without processing three-dimensional stereo processing in a human image in a complex background, The extracted face image of different high quality detailed face image is composed of simulation operation.

The object search system using the visual neural circuit device simplifies the processing and enhances the separation performance by imitating the visual neural network of the directional calculation in the process of extracting the visual information of the proximity object with the low resolution infrared sensor. Camera surveillance cameras, as well as inside the vehicle, are similar.

10 is a block diagram illustrating an object search system according to another embodiment of the present invention.

10 shows an example in which the integrated input unit 200 integrated with the low resolution image input unit and the high resolution image input unit is used and the driving of the high resolution image input unit 201 and the driving of the low resolution image input unit 205 are performed by the input control unit 210 And is driven in a divided manner.

That is, the input control unit 210 drives the low-resolution image input unit 205 to obtain a low-resolution image. Next, the input control unit 210 drives the high-resolution image input unit 201 to obtain a high-resolution image.

At this time, the input controller 210 controls the infrared LED driver 212 to turn on the infrared LED 211 when the low-resolution image input unit 205 is driven.

Meanwhile, the object search unit 250 receives the mask (interest region) extracted from the transconductance neural circuit 220 and the detection unit 240 by inputting from the low-resolution image input unit 205, And extracts a desired image from the input image.

11 is a conceptual diagram of a search system according to another embodiment of the present invention.

Referring to FIG. 11, the image sensor is made to detect visible light and near-infrared light, and an infrared sensor-based separation extraction mask is generated by an infrared LED light projection operation at every frame in a frame so that the driver can see a high- And extract and extract various states to perform various state recognition and monitoring of the driver.

12 is a configuration diagram of a search system according to another embodiment of the present invention.

12, when the object search unit 150 shown in FIG. 8 searches for an image desired by the search system according to another embodiment of the present invention, the situation determination unit 160 compares the images before and after the searched image, Identify the status.

In other words, the situation determination unit 160 determines whether the driver is in an abnormal state such as sleepiness, severe fatigue state, neglecting the forward gaze, or the like, and recognizes timely real-time when an abnormality is detected. Or provides an alarm to the in-vehicle safety system 180. The in-

Then, the alarm unit 170 issues a warning to the driver, including an alarm, sound, vibration, or time, so that the driver awakens.

Then, the safety system 180 informs the vehicle control system so that it can take necessary measures.

13 is a flowchart of an object searching method using a multi-type sensor-based visual neural network apparatus according to an embodiment of the present invention.

Referring to FIG. 13, in step S100, an object is searched through a low-resolution image input unit according to an object search method using a multi-species sensor-based visual neural circuit device according to an exemplary embodiment of the present invention. Here, the low-resolution image input unit is an infrared sensor, a phototransistor, or a photodiode.

Thereafter, the transconductance circuit processes the input image through the low-resolution image input unit and outputs it in operation S110.

In this process, the transconductance neural circuit converts the current corresponding to the image input through the low-resolution image input unit to a voltage through linear current conversion (transconductance is performed), amplifies the current to output a current source buffer, And a process of weighting and buffering the current signal of the phototransistor (performed by the buffering unit).

Meanwhile, when the brain shape visual information directional processing is completed through the transconductance neural circuit, the detection unit extracts the region of interest image (S120).

At this time, the image extracted by the detection unit may be in the shape of the upper half of the vehicle driver, which is then used as a mask.

Then, the object searching unit searches for an object corresponding to the region of interest detected by the detecting unit on the image received from the high-resolution image sensor (S130).

In this way, since the object search section selects a specific region from the high resolution image sensor, a clear image can be obtained.

Thereafter, the situation determiner determines whether the driver is in an abnormal state such as sleepiness, severe fatigue, neglecting the forward gaze, or the like (S140). If an abnormal condition is detected, And provides an alarm to the driver through the alarm unit or transmits the situation to the in-vehicle safety system (S150).

Then, the alarm unit issues the alarm to the driver, including the alarm, sound, vibration, or time, so that the driver can awaken.

The safety system informs the vehicle control system so that it can take necessary measures.

The present invention is not necessarily limited to these embodiments, as all the constituent elements constituting the embodiment of the present invention are described as being combined or operated in one operation. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them. In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. In addition, such a computer program may be stored in a computer-readable medium such as a USB memory, a CD disk, a flash memory, etc., and read and executed by a computer, thereby implementing embodiments of the present invention. As the storage medium of the computer program, a magnetic recording medium, an optical recording medium, a carrier wave medium, or the like may be included.

Furthermore, all terms including technical or scientific terms have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs, unless otherwise defined in the Detailed Description. Commonly used terms, such as predefined terms, should be interpreted to be consistent with the contextual meanings of the related art, and are not to be construed as ideal or overly formal, unless expressly defined to the contrary.

The foregoing description is merely illustrative of the technical idea of the present invention, and various changes and modifications may be made by those skilled in the art without departing from the essential characteristics of the present invention. In addition, the embodiments disclosed in the present invention are not intended to limit the scope of the present invention but to limit the scope of the technical idea of the present invention. Accordingly, the scope of protection of the present invention should be construed according to the claims, and all technical ideas within the scope of equivalents should be interpreted as being included in the scope of the present invention.

10, 10 ': low resolution image input unit
20, 20 ': transconductance nerve circuit
21, 21-1 ', 21-2': Transconductance part
25 and 25 ': buffering units 25-1' and 25-2 '
110: low resolution image input unit 115: high resolution image input unit
120: transconductance nerve circuit 121: transconductance part
125: buffering unit 130: weighting control unit
140: detecting unit 150: object searching unit
160: situation determination unit 170:
180: Safety system 200: Integrated input
201: High resolution image sensor 205: Low resolution image sensor
210: input control unit 211: infrared LED
212: Infrared LED driver 220: Transconductance nerve circuit
221: transconductance unit 225: buffering unit
230: Weight control unit 240:
250: Object search section

Claims

A low-resolution image input unit receiving an image; And
And a transconductance neural circuit for outputting the image input from the low-resolution image input unit by performing a brain imitating visual information directional processing and outputting the image.

The method according to claim 1,
Wherein the low resolution image input unit is one of an infrared ray sensor, a phototransistor, and a photodiode.

The method according to claim 1,
The transconductance neural circuit
A transconductance unit for converting a current corresponding to an image input through the low-resolution image input unit into a voltage through linear current conversion; And
And a buffering unit for performing a weighting operation on a neural connection loop and a weighting operation and a buffering operation on a current signal of a low-resolution input unit, and a buffering unit for buffering the voltage of the transconductance unit.

The method according to claim 3,
Wherein the transconductance section comprises a linear conductance circuit whose voltage is controlled by a pair of NMOS (N-Metal Oxide Semiconductor) transistors.

The method according to claim 3,
The buffering unit
A forward buffer for providing an output current to the outside; And
A multi-sensor-based visual neural network consisting of a negative directional buffer that receives the output current from the outside.

The method according to claim 3,
And a weight control unit for applying a weight to the buffering unit through a terminal of the neural connection loop weight of the buffering unit to emphasize an image of a region of interest.

A transconductance neural circuit for outputting the image input from the low-resolution image input unit by performing directional processing of brain imitation visual information;
A detection unit for extracting a region of interest from a low-resolution image output from the transconductance neural circuit; And
And an object search unit for extracting an object corresponding to the region of interest detected by the detection unit in the image input from the high resolution image input unit.

The method of claim 7,
And a weight control unit for adjusting the weight through the nerve link loop weight terminals of the transconductance neural circuit to emphasize the region of interest.

The method of claim 7,
Wherein the low resolution image input unit is one of an infrared ray sensor, a phototransistor, and a photodiode.

The method of claim 7,
Wherein the high resolution image input unit is a CMOS sensor.

The method of claim 7,
The transconductance neural circuit
A transconductance unit for converting a current corresponding to an image input through the low-resolution image input unit into a voltage through linear current conversion; And
And a buffering unit for performing a weighting operation on a neural connection loop and a weighting operation and a buffering operation on a current signal of a low resolution input unit by amplifying the voltage of the transconductance unit by outputting a current source.

The method of claim 7,
Wherein the low resolution image sensor and the high resolution image sensor are integrated to constitute an integrated input unit.

The method of claim 12,
Further comprising an input control unit for time-division-controlling the low-resolution image sensor and the high-resolution image sensor of the integrated input unit to output the low-resolution image and the high-resolution image at different times.

The method of claim 7,
And a situation determination unit for determining an abnormal state of the object extracted by the object search unit.

The method of claim 14,
And an alarm unit for alerting the user when an abnormal state occurs,
And the situation determination unit outputs the abnormal condition to the warning unit.

The method of claim 14,
Further comprising a safety system for informing the vehicle control system of an action when an abnormal condition occurs,
And the situation determination unit outputs the abnormal condition to the safety system when an abnormal condition occurs.

(A) processing the image input through the low-resolution image input unit by the transconductance neural circuit to process the brain shape visual information direction and outputting;
(B) extracting a region of interest when the brain shape visual information directional processing is completed through the transconductance neural circuit; And
(C) searching for an object corresponding to a region of interest detected by the detection unit on an image received from the object search unit from the high resolution image sensor and extracting the object.

The method of claim 17,
In the step (A)
And the weight control unit controls the weight of the transconductance neural circuit to emphasize the region of interest.

The method of claim 17,
(D) determining a situation of the object judging unit and determining whether the object is in an abnormal state; And
(E) If the situation determination unit finds an indication of an abnormal state, the method further comprises providing an alarm through the alarm unit or taking a safety measure through the safety system.