KR20240060522A - Neuromorphic device implementing neural network and operation method of the same - Google Patents

Abstract

A neuromorphic device implementing a neural network, comprising: a memory storing at least one program; on-chip memory including crossbar array circuitry; and at least one processor that drives the neural network by executing the at least one program, wherein the at least one processor receives an audio signal and transmits the neural network to the neural network learned based on predetermined training data. An audio signal is input and a voice recognition result is output, and the neural network includes a Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, and a Mel generator that generates a Mel spectrogram from the frequency signal. It may include an output layer that outputs the voice recognition result based on the voice characteristics of the layer and the mel spectrogram.

Description

Neuromorphic device implementing a neural network and its operating method {NEUROMORPHIC DEVICE IMPLEMENTING NEURAL NETWORK AND OPERATION METHOD OF THE SAME}

The present invention relates to a neuromorphic device implementing a neural network and a method of operating the same. More specifically, to a system that performs voice recognition using an edge AI chip without using a cloud server or physical server and a method of operating the same. It's about.

With the development of Internet technology, interaction between people and computer devices is becoming more frequent. In the process of interacting with a computer device through human speech, it has become very important to accurately recognize natural language and determine the user's intention.

However, voice recognition is currently carried out in a cloud environment, and in the case of cloud-based voice recognition systems, real-time voice recognition has limitations because the input voice must be transmitted to the cloud. In particular, if the communication connection is unstable at the time of voice recognition or a problem occurs in the communication connection, there is a problem in which voice recognition itself becomes impossible.

The above-mentioned background technology is technical information that the inventor possessed for deriving the present invention or acquired in the process of deriving the present invention, and cannot necessarily be said to be known art disclosed to the general public before filing the application for the present invention.

The purpose of the present disclosure is to provide a neuromorphic device that implements a neural network and a method of operating the same. The problem to be solved by the present disclosure is not limited to the technical problems mentioned above, and other technical problems not mentioned can be clearly understood by those skilled in the art from the description of the present invention. The present disclosure will be understood more clearly by the examples. In addition, it will be appreciated that the problems and advantages to be solved by the present disclosure can be realized by the means and combinations thereof indicated in the patent claims.

As a means for solving the above-described technical problem, a first aspect of the present disclosure includes: a memory storing at least one program; on-chip memory including crossbar array circuitry; and at least one processor that drives the neural network by executing the at least one program, wherein the at least one processor receives an audio signal and transmits the audio to a neural network learned based on predetermined training data. A voice recognition result is output by inputting a signal, and the neural network includes a Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, and a Mel generation layer that generates a Mel spectrogram from the frequency signal. and an output layer that outputs the voice recognition result based on the voice characteristics of the Mel spectrogram.

A second aspect of the present disclosure includes receiving an audio signal; And inputting the audio signal to a neural network learned based on predetermined learning data to output a voice recognition result, wherein the neural network converts the audio signal from a time band to a frequency band to produce a frequency signal. A new layer that implements a neural network, including a Fourier transform layer that generates a Mel, a Mel generation layer that generates a Mel spectrogram from the frequency signal, and an output layer that outputs the voice recognition result based on the voice characteristics of the Mel spectrogram. A method of operating a lomorphic device may be provided.

A third aspect of the present disclosure may provide a computer-readable recording medium recording a program for executing the method of the second aspect on a computer.

In addition, another method for implementing the present invention, another device, and a computer-readable recording medium recording a program for executing the method may be further provided.

Other aspects, features and advantages in addition to those described above will become apparent from the following drawings, claims and detailed description of the invention.

According to the means for solving the problems of the present disclosure described above, it is possible to provide a neuromorphic device that is less dependent on servers, reduces communication costs, and improves security for personal information.

In addition, according to the problem solving means of the present disclosure, it is possible to provide a neuromorphic device with improved speed by preventing bottlenecks due to data transmission and reception.

The effects of the examples are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the description of the present invention.

Figure 1 is a diagram for explaining a neuromorphic chip structure according to an embodiment.
FIG. 2 is a diagram illustrating the architecture of a fully-connected neural network (FCNN) according to an embodiment.
Figure 3 is a block diagram showing the hardware configuration of a neuromorphic device according to an embodiment.
Figures 4a and 4b are exemplary diagrams for comparing the von Neumann structure and the PIM (Processing-In Memory) structure.
5A to 5B are diagrams for explaining a method of operating a neural network according to an embodiment.
6A to 6B are diagrams for comparing vector-matrix multiplication and operations performed in a neural network according to one embodiment.
FIG. 7 is a diagram for explaining an example of a convolution operation being performed in a neural network according to an embodiment.
Figure 8 is an example of a neural network implementation according to an embodiment.
Figure 9 is an exemplary implementation diagram of a crossbar array circuit according to an embodiment.
Figure 10 is a flowchart of a method of operating a neuromorphic device according to one embodiment.
Figure 11 is a block diagram of a neuromorphic device according to an embodiment of the present invention.

In describing the present invention, if it is judged that a detailed description of related known technology may obscure the gist of the present invention, the detailed description may be omitted, and unless otherwise defined, all terms used in this specification refer to the present invention. It has the same meaning as generally understood by those with ordinary knowledge in the relevant technical field.

Phrases such as “according to one embodiment,” “related to one embodiment,” or “according to an implementation of an embodiment” in this specification do not necessarily all refer to the same embodiment.

Since the embodiments can be subject to various changes and have various forms, some embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the embodiments to a specific disclosed form, and should be understood to include all changes, equivalents, and substitutes included in the spirit and technical scope of the embodiments. The terms used in the specification are merely used to describe the embodiments and are not intended to limit the embodiments.

The terms used in the embodiments are general terms that are currently widely used as much as possible while considering the functions in the embodiments, but this is due to the intention or precedent of technicians working in the technical field to which the embodiments belong, the emergence of new technology, etc. It may vary depending on In addition, in certain cases, there are terms arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the relevant section. Therefore, the terms used in the embodiments should be defined based on the meaning of the term and the overall content of the embodiments, rather than simply the name of the term.

Some embodiments of the present disclosure may be represented by functional block configurations and various processing steps. Some or all of these functional blocks may be implemented in various numbers of hardware and/or software configurations that perform specific functions. For example, the functional blocks of the present disclosure may be implemented by one or more microprocessors, or may be implemented by circuit configurations for certain functions.

Additionally, for example, functional blocks of the present disclosure may be implemented in various programming or scripting languages. Functional blocks may be implemented as algorithms running on one or more processors. Additionally, the present disclosure may employ conventional technologies for electronic environment setup, signal processing, and/or data processing.

Terms such as “database,” “element,” “means,” and “configuration” may be used broadly and are not limited to mechanical or physical configurations. Additionally, terms such as “-unit” and “-module” used in the specification refer to a unit that processes at least one function or operation, which may be implemented as hardware or software, or as a combination of hardware and software.

Additionally, connection lines or connection members between components shown in the drawings merely exemplify functional connections and/or physical or circuit connections. In an actual device, connections between components may be represented by various replaceable or additional functional connections, physical connections, or circuit connections.

Additionally, terms including ordinal numbers such as 'first' or 'second' used in this specification may be used to describe various components, but the components should not be limited by the terms. Terms are used only to distinguish one component from another.

Additionally, some components in the drawing may be depicted with their size or proportions somewhat exaggerated. Additionally, components shown in one drawing may not be shown in other drawings.

Throughout the specification, 'examples' is an arbitrary division for easily explaining the invention in the present disclosure, and each embodiment does not need to be mutually exclusive. For example, configurations disclosed in one embodiment may be applied and/or implemented in another embodiment, and may be applied and/or implemented with changes without departing from the scope of the present disclosure.

Additionally, the terms used in this disclosure are for describing the embodiments and are not intended to limit the embodiments. In this disclosure, the singular form also includes the plural form unless otherwise specified.

Below, with reference to the attached drawings, embodiments of the present disclosure will be described in detail so that those skilled in the art can easily implement them. However, embodiments of the present disclosure may be implemented in various different forms and are not limited to the embodiments described in the present disclosure.

Hereinafter, the present invention will be described in detail with reference to the drawings.

Figure 1 is a diagram for explaining a neuromorphic chip structure according to an embodiment.

Referring to FIG. 1, a neuromorphic chip 1 is hardware that simulates human brain function by creating a circuit that mimics the shape of a neuron. In other words, the neuromorphic chip (1) refers to a computer chip that mimics the structure of the nervous system.

Since the neuromorphic chip (1) consists of only the circuits necessary for neural network calculation, it can achieve hundreds of times more gains in terms of power, area, and speed. Unlike conventional computers, the human brain does not consume a lot of power even when processing a large amount of data. The neuromorphic chip (1) mimics the way the brain operates and has a structure connecting neurons and synapses in parallel. Energy can be saved by switching on and off when data is not being processed.

For example, the von Neumann architecture of the existing computer is excellent for executing precisely written programs because it processes data sequentially when data is input, but has problems such as limitations in power consumption and low efficiency in pattern recognition and real-time recognition. there is.

On the other hand, the neuromorphic chip 1 uses an analog operation in which data gradually changes into various states rather than digital ones such as 0 and 1. In other words, artificial neurons configured in parallel operate in an event-driven manner without clock operation. Therefore, it can efficiently process atypical text, voice, and video that are difficult for existing computers to recognize intuitively. Specifically, data can be processed in parallel by distributing neurons, which are nerve cells, and synapses, which are connecting lines, with silicon transistor circuits and memory devices.

In one embodiment, when input data such as an image, video, or voice is input to the neuromorphic chip 1 of FIG. 1, the input data is processed within the neuromorphic chip 1 to output predetermined output data. It can be. Predetermined output data may include voice/image/video recognition results through feature classification of input data. For example, when a voice signal is input as input data, predetermined output data may include a voice recognition result obtained by binary classification of whether the input data includes a preset keyword.

Meanwhile, data input to the neuromorphic chip 1 is not limited to the above-described images, videos, or voices, and may include various types of data such as text.

FIG. 2 is a diagram illustrating the architecture of a fully-connected neural network (FCNN) according to an embodiment.

Referring to Figure 2, FCNN, a fully connected layer, refers to a convolution neural network in which all neurons in one layer are connected to all neurons in the next layer. This is a layer used to classify data through a flattened matrix in the form of a one-dimensional array.

In FCNN, there is a value for each node and a weight and bias value for each neuron. When moving from one layer to another, the value of each node multiplied by the weight and the bias added becomes the node value of the next layer. At this time, if the calculated value satisfies a specific condition, the output value after going through the activation function is used as the node value of the next layer. It is activated by inputting, and if the calculated value does not meet certain conditions, an activation function is involved to deactivate the node through the activation function.

Therefore, since the output value varies depending on the type of activation function, it is important to use an appropriate activation function as needed. Representative examples include the ReLU function and Softmax function.

FCNN's input can only be 1-dimensional array data, so if the input data is a 3-dimensional array image consisting of vertical, horizontal, and channel (color), it has the disadvantage of having to be flattened into 1-dimensional data and then input it to FCNN. In other words, there is a disadvantage in that the features contained in the shape cannot be extracted because the spatial information of the image is ignored. Therefore, in the case of FCNN, it can be highly useful when inputting data that can be implemented as one-dimensional array data, such as voice data. According to one embodiment, the neural network to be described later may be FCNN.

Figure 3 is a block diagram showing the hardware configuration of a neuromorphic device according to an embodiment.

The neuromorphic device 300 can be implemented with various types of devices such as personal computers (PCs), server devices, mobile devices, and embedded devices, and specific examples include voice recognition, image recognition, and image classification using a neural network. This may apply to smartphones, tablet devices, AR (Augmented Reality) devices, IoT (Internet of Things) devices, self-driving cars, robotics, medical devices, etc., but is not limited thereto. Furthermore, the neuromorphic device 300 may correspond to a dedicated hardware accelerator (HW accelerator) mounted on the above device, and the neuromorphic device 300 is a neural processing unit (NPU), a dedicated module for driving a neural network. ), TPU (Tensor Processing Unit), Neural Engine, etc., but is not limited to hardware accelerators.

Neuromorphic device 300 may include a processor and memory. In the neuromorphic device 300 shown in FIG. 3, only components related to the present embodiments are shown, and in addition to the components shown in FIG. 3, the neuromorphic device 300 includes other general-purpose components. It is obvious to those skilled in the art that it can be included.

The neuromorphic device 300 according to one embodiment may include an input/output interface 310.

The input/output interface 310 according to one embodiment may transmit data from the inside of the neuromorphic device 300 to an external source and receive data from the external source to the inside of the neuromorphic device 300. Signals in the input/output interface 310 may be unidirectional or bidirectional, may be single-ended or differential-mode, and may follow one of the other input/output interface standards.

The input/output interface 310 according to one embodiment may further include an audio receiver. For example, it may further include an audio ADC or DAC. The audio receiver can digitize the voice input from the microphone. Additionally, the audio receiver may include an amplifier and support multiple sampling rates.

In one embodiment, input/output interface 310 may receive audio signals external to neuromorphic device 300. Additionally, the input/output interface 310 may receive the voice recognition result, which is the output value of the neural network 330, and transmit it to the operation circuit 350 or another external unit (not shown).

The input/output interface 310 according to one embodiment includes General-Purpose Input/Output (GPIO), Integrated Interchip Sound (I2S), Inter-IC Sound (I2S), Inter-Integrated Circuit (I2C), Serial Peripheral Interface (SPI), and UART. (Universal asynchronous receiver/transmitter), PWM (Pulse Width Modulation), etc. may be further included.

Neuromorphic device 300 according to one embodiment may include analog devices 320 and 340.

The analog devices 320 and 340 can provide functions for stable power supply and system monitoring and supervision in the Edge AI chip.

Specifically, the analog devices 320 and 340 are devices that operate with parameters that appear according to continuous physical quantities such as voltage, resistance, rotation, pressure, etc. in supervisory control, data collection, and automatic control. The analog devices 320 and 340 may include an analog display device and an analog input/output device, and may further include an analog-to-digital converter and a digital-to-analog converter.

Additionally, the analog devices 320 and 340 may perform power management and supervision functions of the neuromorphic device 300. The analog devices 320 and 340 further include a low voltage detector (LVD), and operate a power-on reset (POR) when the digital power supply (VDD) falls below a safe operating level to control the neuromorphic device. Unstable operation of (300) can be prevented.

The neuromorphic device 300 according to one embodiment may include an arithmetic circuit 350.

The operation circuit 350 may include a Micro Controller Unit (MCU), Direct Memory Access (DMA), etc.

The MCU can play a control and management role to execute efficient algorithms in the PIM structure. The PIM structure has high efficiency in MAC (Multiply And Accumulate) operations, so it may be suitable for neural network calculations according to an embodiment of the present invention. The PIM structure will be described later with reference to FIG. 4B.

DMA can perform high-speed data transfer between peripheral devices and memory, or between memories. Data can be quickly moved to DMA without the CPU and the CPU can be utilized for other tasks. DMA is located inside SRAM and can contribute to accelerating data movement in neural networks.

Neuromorphic device 300 according to one embodiment may be implemented with an Edge AI Chip. Edge AI refers to a technology that runs AI algorithms on hardware devices using edge computing based on data generated from the system. AI processing is mainly performed in cloud-based data centers that require enormous computing capacity and is highly server-dependent. On the other hand, when Edge AI is used, AI algorithm calculations are performed locally, reducing dependence on the cloud (server), thereby reducing communication costs, and protecting privacy as sensitive personal information is not transmitted to the cloud. You can. Therefore, by configuring the neuromorphic device 300 with an Edge AI Chip, not only can costs be reduced and security improved, but calculations can be performed immediately within the same hardware, enabling the implementation of a highly responsive system.

Figures 4a and 4b are exemplary diagrams for comparing the von Neumann structure and the PIM (Processing-In Memory) structure.

Referring to Figure 4a, the von Neumann architecture is a computer architecture proposed by John von Neumann and is a program-embedded computer architecture consisting of a typical three-level structure of a main memory, a central processing unit, and an input/output device.

The von Neumann structure has the advantage of greatly improving versatility because only the software (program) needs to be changed without the need to rearrange the hardware (wires, etc.) when changing from one computing device to another, but the listed commands are performed sequentially and the commands are Because it consists of changing the value of a certain memory location, it causes serious problems in the design of high-speed computers. This is called the von-Neumann bottleneck phenomenon.

In order to solve the von Neumann bottleneck, the Harvard architecture divides memory into where instructions are stored and where data is stored, or the PIM architecture that not only stores data in memory but also performs data operations, and imitates the brain structure of higher animals. Neuromorphic computing, which is an integrated circuit in the form of an artificial neural network that consists of numerous units with integrated calculation and memory functions, connects them in parallel like a mesh, and then operates each unit in an event-driven manner, etc. This is being presented as an alternative.

Referring to Figure 4b, it can be seen that the PIM structure is composed of a processor and a memory with a computing function.

Unlike in the existing von Neumann structure, all data inside the memory is moved to the processor for calculation, in the PIM structure, when the processor's command is delivered, the calculation is performed within the memory and only the result data is sent to the processor, thereby moving a large amount of data. Without this, the aforementioned von Neumann bottleneck can be effectively solved. Additionally, there is an advantage that power consumption is significantly lowered.

Returning to FIG. 3 , the neuromorphic device 300 according to one embodiment may include a neural network 330.

In one embodiment, the neural network 330 may output output data by extracting features for input data using a plurality of layers. For example, the neural network 330 may receive an audio signal and output a voice recognition result. As described above, the neural network 330 filters or classifies the characteristics of the input audio signal to determine the type of the audio signal, such as whether the audio signal contains a specific keyword, and outputs it as a voice recognition result. You can. At this time, the neural network 330 is composed of an orthogonal matrix, and input may be input through the row direction and output may be generated through the column direction.

In one embodiment, the neural network 330 may be a model learned based on predetermined training data. Each layer that makes up the neural network 330 has a weight and/or bias value. Since the number of nodes in one layer is the size of the input tensor, there may be hundreds or thousands of nodes in each layer, and multiple such layers can be stacked. As the number of weights connected to the nodes of the next layer increases, it is difficult to set all weight and/or bias values. Therefore, through learning, you can find the most optimized layer (i.e. weight and/or bias) for what you want to classify on a large amount of data.

For example, deep learning as a learning method for the neural network 330 is a neural network using a combination of three methods: loss function or cost function, optimization, and back propagation. You can find and set the most appropriate weight.

The neural network 330 can perform calculations using only on-chip memory without using external memory. For example, the neural network 330 performs calculations for each layer based on PIM using only on-chip memory without using external memory (e.g., off-chip memory, etc.), thereby processing the memory while processing the audio signal. Operations can be performed without updates. Specifically, the neural network 330 may perform PIM-based calculations in which each memory cell and the processor are directly connected.

However, PIM requires high memory bandwidth, and since memory is sensitive to high temperatures, the power that a computing device can consume may be limited. Therefore, in order to produce high-performance PIM structure chips, a new hardware structure may be required, which may increase manufacturing costs. Therefore, the PIM structure may be advantageous for performing relatively small amounts or simple calculations. On the other hand, the neural network 330 may be configured with a memory capable of multi-bit implementation to overcome the shortcomings of the PIM structure. For example, the neural network 330 may be configured with a memory capable of implementing 7 bits 128 analog memory states. By configuring the neural network 330 with a large capacity, it is possible to process a large amount of data with low power and high performance even when used for a long time, unlike typical PIM chips that have problems such as heat generation or performance degradation.

5A to 5B are diagrams for explaining a method of operating a neural network according to an embodiment.

Referring to FIG. 5A, the neural network may include a plurality of cores, and each core may be implemented as RCA (Resistive Crossbar Memory Arrays). Specifically, each core includes a plurality of presynaptic neurons (510), a plurality of postsynaptic neurons (520), and a plurality of presynaptic neurons (510) and a plurality of postsynaptic neurons (520). It may include a synapse 530 that provides each connection.

In one embodiment, the core of the neural network includes 4 pre-synaptic neurons 510, 4 post-synaptic neurons 520, and 16 synapses 530, but these numbers may vary. The number of pre-synaptic neurons 510 is N (where N is a natural number of 2 or more), and the number of post-synaptic neurons 520 is M (where M is a natural number of 2 or more and may be equal to or different from N). In this case, N*M synapses 530 may be arranged in a matrix form.

Specifically, a wiring 512 connected to each of the plurality of pre-synaptic neurons 510 and extending in the first direction (e.g., horizontal direction), and connected to each of the plurality of post-synaptic neurons 520 and extending in the first direction. A wiring 522 extending in a second direction (eg, vertical direction) that intersects may be provided. Hereinafter, for convenience of explanation, the wiring 512 extending in the first direction will be referred to as a row line, and the wiring 522 extending in the second direction will be referred to as a column line. A plurality of synapses 530 may be disposed at each intersection of the row wiring 512 and the column wiring 522 to connect the corresponding row wiring 512 and the corresponding column wiring 522 to each other.

The pre-synaptic neuron 510 generates a signal, for example, a signal corresponding to specific data and sends it to the row wiring 512, and the post-synaptic neuron 520 transmits the synaptic signal that has passed through the synaptic element 530 to the column wiring. It can perform the role of receiving and processing through (522). The pre-synaptic neuron 510 may correspond to an axon, and the post-synaptic neuron 520 may correspond to a neuron. However, whether a neuron is pre-synaptic or postsynaptic can be determined by its relative relationship to other neurons. For example, when the pre-synaptic neuron 510 receives synaptic signals in a relationship with another neuron, it may function as a post-synaptic neuron. Similarly, a post-synaptic neuron 520 may function as a pre-synaptic neuron when it sends signals in relationships with other neurons. The pre-synaptic neuron 510 and the post-synaptic neuron 520 may be implemented with various circuits such as CMOS.

The connection between the pre-synaptic neuron 510 and the post-synaptic neuron 520 may be made through the synapse 530. Here, the synapse 530 is an element whose electrical conductance or weight changes depending on electrical pulses, such as voltage or current, applied to both ends.

The core of the neural network is ReRAM (Resistive RAM), where the synapse 530 is composed of variable resistance, and FeRAM (Ferroelectric RAM), where the synapse 530 is composed of a ferroelectric. As the synapse 530 is heated, it changes from an amorphous state to a crystalline state. It will be configured using multi-level memory such as PRAM (Phase-change RAM) made of chalcogenide glass, MRAM (Magnetic RAM) whose synapse 530 is made of magnetic elements, and NAND/NOR flash memory. You can.

The synapse 530 may include, for example, a variable resistance element. A variable resistance element is an element that can switch between different resistance states depending on the voltage or current applied to both ends, and is made of various materials that can have multiple resistance states, such as transition metal oxides and perovskite systems. It may have a single-layer structure or a multi-layer structure including metal oxides such as metal oxides, phase change materials such as chalcogenide-based materials, ferroelectric materials, and ferromagnetic materials.

The synapse 530 of the core has no abrupt change in resistance during set and reset operations and exhibits analog behavior in which conductivity gradually changes depending on the number of input electrical pulses. It can be implemented to have various characteristics that distinguish it from a variable resistance element. This is because the characteristics required for the variable resistance element in the memory and the characteristics required for the synapse 530 in the core of the neural network are different from each other.

Specifically, the neuromorphic chip can apply diversified voltages through the synapse 530 analog device of the core. Accordingly, the resistance value or weight of the synapse 530 may gradually change.

The operation of the above neural network is explained with reference to FIG. 5b as follows. For convenience of explanation, the row wiring 512 may be referred to as a first row wiring 512A, a second row wiring 512B, a third row wiring 512C, and a fourth row wiring 512D in order from the top. , the column wiring 522 may be referred to as a first column wiring 522A, a second column wiring 522B, a third column wiring 522C, and a fourth column wiring 522D in that order from the left.

Referring to FIG. 5B, in the initial state, all of the synapses 530 may be in a state of relatively low conductivity, that is, a high resistance state. If at least some of the plurality of synapses 530 are in a low-resistance state, an additional initialization operation may be required to bring them into a high-resistance state. Each of the plurality of synapses 530 may have a predetermined threshold required for change in resistance and/or conductance. More specifically, when a voltage or current smaller than a predetermined threshold is applied to both ends of each synapse 530, the conductivity of the synapse 530 does not change, and a voltage or current larger than the predetermined threshold is applied to the synapse 530. When this happens, the conductivity of the synapse 530 may change.

In this state, in order to perform an operation of outputting specific data as a result of the specific column wiring 522, an input signal corresponding to specific data comes into the row wiring 512 in response to the output of the pre-synaptic circuit 510. You can. For example, the input signal may appear as the application of an electrical pulse to each row wiring 512. Additionally, the column wiring 522 may be driven with an appropriate voltage or current for output.

As another example, the column wiring 522 to output specific data may not be determined. In this case, while applying an electrical pulse corresponding to specific data to the row wiring 512, the current flowing through each of the column wirings 522 is measured and the column wiring 522 that reaches a predetermined threshold current first, for example, the third column wiring. (522C) may be the column wiring 522 that outputs this specific data.

Using the method described above, different data can be output to different column wires 522, respectively.

6A to 6B are diagrams for comparing vector-matrix multiplication and operations performed in a neural network according to one embodiment.

6A to 6B are diagrams for comparing vector-matrix multiplication and operations performed in a neural network according to one embodiment.

First, referring to FIG. 6A, the convolution operation between input data and the kernel can be performed using vector-matrix multiplication. For example, pixel data of input data can be expressed as a matrix X (610), and kernel values can be expressed as a matrix W (611). Pixel data of the output data can be expressed as a matrix Y (612), which is the result of a multiplication operation between the matrix X (610) and the matrix W (611).

Referring to FIG. 6B, a vector multiplication operation can be performed using the core of a neural network. 6A , pixel data of the input data may be received as an input value of the core, and the input value may be a voltage 620. Additionally, kernel values may be stored in the synapse of the core, that is, a memory cell, and the kernel values stored in the memory cell may be conductance 621. Accordingly, the output value of the core can be expressed as a current 622, which is the result of a multiplication operation between the voltage 620 and the conductance 621.

FIG. 7 is a diagram for explaining an example of a convolution operation being performed in a neural network according to an embodiment.

The neural network can receive an audio signal 710, and the core 700 of the neural network can be implemented with RCA (Resistive Crossbar Memory Arrays). At this time, the audio signal 710 is converted from the time band to the frequency band by the Fourier transform layer implemented by the core 700 of the neural network, and into a mel spectrum by the mel generation layer implemented by the core 700 of the neural network. It is converted into a mel-spectrogram, and a convolution operation can be performed on the pixel data corresponding to the mel-spectrogram. Therefore, in the description of FIG. 7 below, pixel data of the audio signal 710 may mean pixel data of a converted frequency band or pixel data of a generated Mel spectrogram.

In one embodiment, when the core 700 is a matrix of size NxM (N and M are natural numbers of 2 or more), the number of pixel data of the audio signal 710 is less than the number of columns (M) of the core 700. It can be the same. Pixel data of the audio signal 710 may be parameters in floating point format or fixed point format.

The neural network can receive pixel data in the form of a digital signal, and can convert the received pixel data into voltage in the form of an analog signal using a DAC (Digital Analog Converter) 720. Pixel data of the audio signal 710 may have various bit resolution values, such as 1-bit, 4-bit, and 8-bit resolution. In one embodiment, the neural network may convert pixel data into voltage using the DAC 720 and then receive the voltage as the input value 701 of the core 700.

Additionally, learned kernel values may be stored in the core 700 of the neural network. Kernel values may be stored in a memory cell of the core, and the kernel values stored in the memory cell may be conductance 702. At this time, the neural network can calculate the output value by performing a vector multiplication operation between the voltage 701 and the conductance 702, and the output value can be expressed as a current 703. That is, the neural network can use the core 700 to output the same result as the result of the convolution operation between the audio signal and the kernel.

Since the current 703 output from the core 700 is an analog signal, the neural network can use the ADC (Analog Digital Converter) 730 to use the current 703 as input data for another core. The neural network can convert the current 703, which is an analog signal, into a digital signal using the ADC 730. In one embodiment, the neural network may use the ADC 730 to convert the current 703 into a digital signal to have the same bit resolution as the pixel data of the audio signal 710. For example, if the pixel data of the audio signal 710 has 1-bit resolution, the neural network can convert the current 703 into a digital signal with 1-bit resolution using the ADC 730.

The neural network can apply an activation function to the digital signal converted in the ADC 730 using the activation unit 740. The Sigmoid function, Tanh function, and ReLU (Rectified Linear Unit) function can be used as activation functions, but the activation function applicable to digital signals is not limited to these.

The digital signal to which the activation function is applied can be used as an input value for another core 750. When a digital signal to which an activation function is applied is used as an input value for another core 750, the above-described process can be applied equally to the other core 750.

Meanwhile, the core 700 and the other core 750 are not physically separated, but each core 700 has a variable resistance element value of the synapse changed according to the weight and/or bias value of each core 700 and 750. , 750).

Figure 8 is an example of a neural network implementation according to an embodiment.

Referring to Figure 8, the neural network includes a Fourier Transform layer 810 that generates a frequency signal by converting the received audio signal from a time band to a frequency band, and a Mel generation layer that generates a Mel spectrogram from the frequency signal. (820), it may include one or more hidden layers (830, 840) that classify features of the audio signal based on the Mel spectrogram, and an output layer (850) that outputs a voice recognition result. The order of each layer may be as shown in FIG. 8, but is not limited thereto.

In one embodiment, the Fourier transform layer 810 may perform Fourier transform on an audio signal in the time band to generate a frequency signal in the frequency band. As described above, the audio signal may be data that the input/output interface (or an audio receiver included in the input/output interface) digitizes the voice input from the microphone, so it may have a time band. Accordingly, the Fourier transform layer 810 can convert data in a frequency band through Fourier operation to be suitable for input to other layers 820 to 850 for classifying audio signals. For example, the Fourier transform layer 810 may generate a frequency signal through Discrete Fourier Transform (DFT), which is a Fourier transform for a discontinuous discrete function.

In one embodiment, the neural network further includes a digital-to-analog converter (DAC) that converts an audio signal corresponding to a digital signal into an analog signal, and the received audio signal is converted to an analog signal through the digital-to-analog converter and then converted to an analog signal. It may be input to the Fourier transform layer 810.

In one embodiment, the Mel generation layer 820 may receive a frequency signal from the Fourier transform layer 810 and generate a Mel spectrogram corresponding to the audio signal.

A spectrogram is a visualization of the spectrum of a voice signal and expressed as a graph. The x-axis of the spectrogram represents time, the y-axis represents frequency, and the value of each time frequency can be expressed in color according to the size of the value. At this time, since the result of performing Fourier transform on the audio signal is a complex value, the absolute value of the complex value can be taken to lose phase information and generate a spectrogram containing only magnitude information.

Meanwhile, the Mel spectrogram is a re-adjustment of the frequency interval of the spectrogram into Mel Scale. The human auditory system is more sensitive to low frequencies than to high frequencies, and the Mel scale reflects this characteristic and expresses the relationship between physical frequencies and frequencies perceived by humans. A Mel spectrogram can be generated by applying a filter bank based on the Mel scale to the spectrogram.

In one embodiment, one or more hidden layers 830 and 840 may receive a Mel spectrogram corresponding to an audio signal from the Mel generation layer 820 and classify voice features of the audio signal or Mel spectrogram.

One or more hidden layers 830 and 840 can classify the Mel spectrogram according to a predetermined standard by using the tensor corresponding to the input Mel spectrogram as a pre-synaptic neuron and reflecting the weight to the pre-synaptic neuron. Therefore, as the number of hidden layers increases, the function of the algorithm can be improved by processing input data more precisely.

In one embodiment, one or more hidden layers 830 and 840 may be located between the mel generation layer 820 and the output layer 850. Accordingly, one or more hidden layers 830 and 840 may receive the Mel spectrogram from the Mel generation layer 820, classify the features, output the classification result, and transmit it to the output layer 850.

In one embodiment, the output layer 850 may receive voice features or a result of classifying voice features from the mel generation layer 820 or one or more hidden layers 830 and 840 and output a voice recognition result. For example, the output layer 850 may output a voice recognition result by determining whether one or more hidden layers 830 and 840 classify voice features and correspond to one of a plurality of preset keywords. . Specifically, the neuromorphic device can store a plurality of preset keywords. At this time, the hidden layers 830 and 840 classify the characteristics of the audio signal, and the output layer 850 receives the classification result to determine whether it corresponds to one of a plurality of preset keywords.

For example, a plurality of keywords may be commands consisting of a predetermined number of syllables, such as “turn on the light,” “turn off the light,” “brighten the light,” “dim the light,” “block blue light,” and “warm light.” Therefore, the input/output interface receives audio signals including living noise and natural language, and classifies the features of the audio signal through the Fourier transform layer 810, the mel generation layer 820, and one or more hidden layers 830 and 840. , the output layer 850 may determine which one of the plurality of preset keywords corresponds to the audio signal received as a classification result. That is, when the received audio signal corresponds to one of a plurality of preset keywords, the neuromorphic device outputs the corresponding keyword as a voice recognition result through the output layer 850, and the received audio signal corresponds to one of the plurality of preset keywords. If it does not correspond to any of the keywords, an empty value (null) can be output, or the next audio signal can be received without output, and the calculation by each layer described above can be performed repeatedly. In one embodiment, the plurality of keywords may be set to around 20 or less than 20 keywords.

In one embodiment, when the audio signal corresponds to a negative signal similar to at least one of a plurality of preset keywords, the output layer 850 outputs an empty value as a voice recognition result or receives the next audio signal without outputting the above-described audio signal. Operations for each layer can be performed repeatedly. A negative signal may mean a keyword that is similar to one of a plurality of preset keywords and is therefore likely to be recognized by the neural network as a similar keyword. For example, when the preset keyword is “auto care,” “what should I do?” may be a negative signal. Therefore, the negative signal may be preset to prevent the neural network from incorrectly recognizing a negative signal similar to a keyword as a keyword.

In one embodiment, the neural network further includes an analog-to-digital converter (ADC) that converts a voice recognition result corresponding to an analog signal into a digital signal, so that the feature classification result or voice recognition result output from the output layer is analog-to-digital. It can be converted into a digital signal through a converter and then output through an input/output interface.

In one embodiment, when it is determined that the audio signal does not correspond to any of a plurality of preset keywords based on the Mel spectrogram output from the Mel generation layer 820, the neural network generates the Mel spectrogram to the hidden layer ( 830, 840), an empty value can be output as a voice recognition result, or the next audio signal can be received without output, and the calculation by each layer described above can be performed repeatedly. In other words, when the neural network receives an audio signal, it performs operations on the data with default values up to the Fourier transform layer 810 and the Mel generation layer 820, but based on the Mel spectrogram, the audio signal is If it is determined that the voice is not a voice or does not correspond to one of a plurality of preset keywords, data may not be transmitted to the hidden layers 830 and 840 and/or the output layer 850. Since there is no need to perform calculations on the hidden layers (830, 840) and/or output layer (850) for all audio signals, the calculation efficiency of the neuromorphic device is increased by reducing the amount of calculations, and when it has a PIM structure, it increases the computational efficiency of the neuromorphic device. It can also prevent fever.

Specifically, the neural network receives an audio signal, and if the input signal received through an inference model learned with predetermined training data is not included in the learned model data, it stops calculating and enters a low-power mode (sleep mode). can be converted to . That is, if the input signal is nothing more than noise including everyday noise from the surrounding environment, unnecessary power consumption can be minimized by not performing calculations through the hidden layers 830 and 840. Conversely, if the received input signal is included in the learned model data, the neural network performs classification and output of speech recognition results by the next hidden layer (830, 840) and output layer (850), and then enters a low-power mode (sleep mode). mode) can be switched.

Meanwhile, each layer 810 to 850 of the neural network may not be physically separated, but may mean a core in which the variable resistance element value of the synapse is changed according to the weight of each layer. Referring again to FIG. 3 for explanation, each layer 810 to 850 of the neural network may be implemented as the neural network 330.

For example, when an audio signal is received through the input/output interface 310, the audio signal converted to an analog signal through the digital-to-analog converter 320 is converted into a variable resistance of the synapse in response to the weight of the Fourier transform layer 810. Element values are input to the set neural network 330, and a frequency signal in which the time band of the audio signal is changed to a frequency band can be generated. Thereafter, the frequency signal can be converted into a digital signal through the analog-to-digital converter 340 and input into the calculation circuit 350. In addition, the frequency signal converted from the output of the operation circuit 350 to an analog signal through the digital-analog converter 320 is connected to a neural network ( 330), and a Mel spectrogram of the frequency signal can be generated. Afterwards, the Mel spectrogram can be converted into a digital signal through the analog-to-digital converter 340 and input into the calculation circuit 350. Likewise, the characteristics of the audio signal are classified based on the neural network 330 in which the variable resistance element value of the synapse is set in response to the weight of each of the one or more hidden layers 830, 840 and the output layer 850, and voice recognition Since the process of outputting the results overlaps with the above-mentioned process, description will be omitted.

In one embodiment, the neural network may be trained with certain training data. For example, predetermined learning data may be audio data generated by combining a plurality of preset keywords and noise sets. In the field of speech recognition, the learning data used to learn a neural network is mainly data mixed with noise such as a plurality of preset keywords and household noise, and by learning multiple times, a neural network with good recognition performance can be built. However, in order to build a neural network with good keyword recognition performance by using keyword voices in a severe noisy environment as learning data, countless learning data and learning processes may be required. On the other hand, first, a neural network is learned using a plurality of preset keyword voices as training data, and then a neural network is learned using a data set generated by a combination of the preset plurality of keyword voices and noise data as learning data. In this case, compared to the above-mentioned learning method, the amount of learning data is small, so it is efficient and the cost for generating learning data can also be reduced.

In one embodiment, one or more hidden layers 830 and 840 may be composed of weights adjusted based on predetermined learning data. The calculation of one or more hidden layers 830 and 840 is performed using the resistance values of each variable resistance element of the synapse of the neural network as weights. At this time, each weight can be adjusted based on the above-mentioned predetermined learning data, that is, audio data generated by combining a plurality of preset keyword voices and noise sets.

Figure 9 is an exemplary implementation diagram of a crossbar array circuit according to an embodiment.

Referring to FIG. 9, a crossbar array circuit 900 implementing a neural network may include a plurality of sub-circuits 910, 920, and 930. The sub-circuits 910, 920, and 930 are circuits composed of a combination of a plurality of cores constituting the crossbar array circuit 900, and each sub-circuit 910, 920, and 930 corresponds to each layer constituting the neural network. The weight of the synapse can be set.

In one embodiment, the crossbar array circuit 900 may include a first sub-circuit 910, a second sub-circuit 920, and a third sub-circuit 930. At this time, the first sub-circuit 910 may implement a Fourier transform layer, the second sub-circuit 920 may implement a Mel generation layer, and the third sub-circuit 930 may implement an output layer.

In one embodiment, weights corresponding to the Fourier transform layer may be stored in the synapse of the first sub-circuit 910. For example, the first sub-circuit 910 receives an audio signal in the time band as input data through a pre-synaptic neuron, and outputs a synaptic signal that has passed through a synapse in which weights corresponding to the Fourier transform layer are stored through a post-synaptic neuron. By transmitting data, a frequency signal can be generated.

In one embodiment, weights corresponding to the mel generation layer may be stored in the synapse of the second sub-circuit 920. For example, the second sub-circuit 920 receives a frequency signal as input data through a pre-synaptic neuron, and transmits a synaptic signal that has passed through a synapse in which the weight corresponding to the mel generation layer is stored as output data through a post-synaptic neuron. By doing so, a Mel spectrogram can be generated.

In one embodiment, weights corresponding to the output layer may be stored in the synapse of the third sub-circuit 930. For example, the third sub-circuit 930 receives the Mel spectrogram or the features of the Mel spectrogram as input data through the pre-synaptic neuron, and sends the synaptic signal through the synapse in which the weight corresponding to the output layer is stored to the post-synaptic neuron. The voice recognition result can be output by transmitting it as output data through .

Likewise, the crossbar array circuit 900 may further include other sub-circuits (not shown) that implement a hidden layer in addition to the first to third sub-circuits 910, 920, and 930. Another sub-circuit (not shown) that implements the hidden layer may be a circuit in which weights corresponding to the hidden layer are stored in the synapse. For example, the sub-circuit (not shown) receives the Mel spectrogram as input data through the pre-synaptic neuron, and transmits the synaptic signal that has passed through the synapse in which the weight corresponding to the hidden layer is stored as output data through the post-synaptic neuron. , the voice features of the mel spectrogram can be classified.

Meanwhile, each sub-circuit (910, 920, 930) may receive input data not through pre-synaptic neurons, but through row wiring or column wiring connected to the sub-circuit implementing the previous layer. Additionally, similarly, each sub-circuit (910, 920, 930) can transmit output data not through a post-synaptic neuron, but through a row wire or column wire connected to a sub-circuit implementing the next layer. The method by which each sub-circuit (910, 920, and 930) transmits and receives input data and output data is not limited to this. In addition, FIG. 9 shows that the crossbar array circuit 900 includes three sub-circuits 910, 920, and 930, but the number of sub-circuits is not limited to this, and the crossbar array circuit 900 includes three sub-circuits 910, 920, and 930. , 920, 930), the area, synapse configuration and location are also not limited thereto.

The crossbar array circuit 900 according to the above-described embodiment is included in an on-chip memory and can implement a neural network through PIM-based calculation. This neuromorphic device not only reduces chip volume, but also achieves high performance relative to power consumption, making it possible to output voice recognition results for a relatively small number of keywords very efficiently.

Figure 10 is a flowchart of a method of operating a neuromorphic device according to one embodiment.

At step 1010, a neuromorphic device (hereinafter “device”) may receive an audio signal.

In step 1020, the device may output a voice recognition result by inputting an audio signal to a neural network learned based on predetermined training data.

In one embodiment, the neural network includes a Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, a Mel generation layer that generates a Mel spectrogram from the frequency signal, and a Mel spectrogram based on speech features of the Mel spectrogram. It may include an output layer that outputs voice recognition results.

In one embodiment, the neural network may further include one or more hidden layers that classify voice features of the mel spectrogram. At this time, one or more hidden layers may be located between the mel generation layer and the output layer.

In one embodiment, the output layer may output a voice recognition result by determining whether the audio signal corresponds to one of a plurality of preset keywords as a result of the hidden layer classifying voice features.

In one embodiment, the predetermined learning data is audio data generated by combining a plurality of preset keywords and noise sets, and one or more hidden layers may be composed of weights adjusted based on the predetermined learning data.

In one embodiment, the neural network may output an empty value (null) as a voice recognition result when the audio signal corresponds to a negative signal similar to at least one of a plurality of preset keywords.

In one embodiment, the device may be one in which a memory cell (or on-chip memory) and a processor are directly connected to perform a processing-in-memory (PIM)-based operation.

In one embodiment, the device may convert an audio signal corresponding to a digital signal into an analog signal, input it to a Fourier transform layer, and convert the voice recognition result corresponding to the analog signal output from the output layer into a digital signal.

In one embodiment, the device may not input the Mel spectrogram to the hidden layer in response to determining that the audio signal does not correspond to any of a plurality of preset keywords based on the Mel spectrogram.

Figure 11 is a block diagram of a neuromorphic device according to an embodiment of the present invention. In the following description, descriptions that overlap with those of FIGS. 1 to 10 described above will be omitted.

The processor serves to control overall functions for executing the neuromorphic device 1100. For example, the processor 1110 generally controls the neuromorphic device 1100 by executing programs stored in the memory 1120 in the neuromorphic device 1100. The processor 1110 may be implemented as a central processing unit (CPU), a graphics processing unit (GPU), an application processor (AP), etc. provided in the neuromorphic device 1100, but is not limited thereto.

The memory 1120 is hardware that stores various data processed within the neuromorphic device 1100. For example, the memory may store data processed and data to be processed in the neuromorphic device 1100. . Additionally, the memory 1120 may store applications, drivers, etc. to be driven by the neuromorphic device 1100. Memory includes RAM (random access memory) such as DRAM (dynamic random access memory), SRAM (static random access memory), ROM (read-only memory), EEPROM (electrically erasable programmable read-only memory), CD-ROM, and blue It may include Ray or other optical disk storage, a hard disk drive (HDD), a solid state drive (SSD), or flash memory.

Unlike the neural network 1 shown in FIG. 1, an actual neural network running in the neuromorphic device 1100 may be implemented with a more complex architecture. Accordingly, the processor 1110 performs operations corresponding to a very large operation count, ranging from hundreds of millions to tens of billions, and the frequency with which the processor 1110 accesses the memory 1130 for operations also dramatically increases. It can be.

Accordingly, memory 1120 may be an on-chip memory. The neuromorphic device 1100 according to one embodiment has the memory 1120 only in the form of an on-chip memory, and can perform calculations without accessing the external memory 1130. For example, the memory 1120 may be SRAM implemented as an on-chip memory. In this case, unlike what was described above, types of memory mainly used as external memory 1130, such as DRAM, ROM, HDD, and SSD, may not be used as memory 1120.

The processor 1110 can read/write neural network data, for example, voice data (audio signal), etc., from the memory 1120, and execute a neural network using the read/written data. When the neural network is executed, the processor 1110 may repeatedly perform operations on the audio signal to generate data related to the output. In other words, an audio signal can be received according to a predetermined period, and the operation related to the above-described layer can be repeatedly performed at the predetermined period.

In one embodiment, neuromorphic device 1100 may be a server. A server may be implemented as a computer device or a plurality of computer devices that communicate over a network to provide commands, codes, files, content, services, etc. The server may receive data necessary for voice recognition and perform voice recognition based on the received data.

Meanwhile, embodiments according to the present invention may be implemented in the form of a computer program that can be executed through various components on a computer, and such a computer program may be recorded on a computer-readable medium. At this time, the media includes magnetic media such as hard disks, floppy disks, and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, and ROM. , RAM, flash memory, etc., may include hardware devices specifically configured to store and execute program instructions.

Meanwhile, the computer program may be designed and configured specifically for the present invention, or may be known and available to those skilled in the art of computer software. Examples of computer programs may include not only machine language code such as that created by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like.

According to one embodiment, methods according to various embodiments of the present disclosure may be included and provided in a computer program product. Computer program products are commodities and can be traded between sellers and buyers. The computer program product may be distributed in the form of a machine-readable storage medium (e.g. compact disc read only memory (CD-ROM)) or through an application store (e.g. Play StoreTM) or between two user devices. It may be distributed in person or online (e.g., downloaded or uploaded). In the case of online distribution, at least a portion of the computer program product may be at least temporarily stored or temporarily created in a machine-readable storage medium, such as the memory of a manufacturer's server, an application store's server, or a relay server.

Unless there is an explicit order or statement to the contrary regarding the steps constituting the method according to the invention, the steps may be performed in any suitable order. The present invention is not necessarily limited by the order of description of the above steps. The use of all examples or illustrative terms in the present invention is simply for explaining the present invention in detail, and the scope of the present invention is not limited by the examples or illustrative terms unless limited by the claims. Additionally, those skilled in the art will recognize that various modifications, combinations and changes may be made depending on design conditions and factors within the scope of the appended claims or their equivalents.

Therefore, the spirit of the present invention should not be limited to the above-described embodiments, and the scope of the patent claims described below as well as all scopes equivalent to or equivalently changed from the scope of the claims are within the scope of the spirit of the present invention. It will be said to belong to

Claims (10)

In a neuromorphic device implementing a neural network,
At least one processor running the neural network; and
An on-chip memory including a crossbar array circuit that receives instructions from the at least one processor and performs in-memory operations,
The at least one processor,
receive an audio signal,
Input the audio signal to the neural network learned based on predetermined learning data and output a voice recognition result,
The neural network is,
A Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, a Mel generation layer that generates a Mel spectrogram from the frequency signal, and a voice recognition result based on the voice characteristics of the Mel spectrogram. Contains an output layer that outputs,
The neural network is,
A neuromorphic device comprising one or more hidden layers that classify the speech features of the Mel spectrogram, wherein the one or more hidden layers are located between the Mel generation layer and the output layer. According to claim 1,
The at least one processor,
A neuromorphic device that does not input the Mel spectrogram to the hidden layer in response to determining that the audio signal does not correspond to any of a plurality of preset keywords based on the Mel spectrogram. According to claim 2,
The at least one processor,
A neuromorphic device that outputs a null value as the voice recognition result in response to determining that the audio signal does not correspond to any of the plurality of preset keywords. According to claim 2,
The at least one processor,
A neuromorphic device, in response to determining that the audio signal does not correspond to any of the plurality of preset keywords, receiving a next audio signal without outputting the voice recognition result. According to claim 2,
The at least one processor,
A neuromorphic device, switching to a low power mode in response to determining that the audio signal does not correspond to any of the plurality of preset keywords. According to claim 1,
The predetermined learning data is,
A neuromorphic device, which is audio data generated based on at least one of a plurality of preset keywords and noise sets. According to claim 6,
The neural network is,
First learning is performed using the audio data of the plurality of preset keywords as learning data, and after the first learning, secondary learning is performed using the audio data set generated by a combination of the plurality of preset keywords and the noise set as learning data. A neuromorphic device. According to claim 1,
The at least one processor,
When the audio signal corresponds to a negative signal similar to at least one of a plurality of preset keywords, a neuromorphic device outputs an empty value (null) as the voice recognition result or receives the next audio signal without output. In a method of operating a neuromorphic device implementing a neural network,
Receiving an audio signal; and
Including the step of inputting the audio signal to a neural network learned based on predetermined learning data and outputting a voice recognition result,
The neural network is,
A Fourier transform layer that converts the audio signal from a time band to a frequency band to generate a frequency signal, a Mel generation layer that generates a Mel spectrogram from the frequency signal, and a voice recognition result based on the voice characteristics of the Mel spectrogram. Includes an output layer that outputs,
The neural network is,
A method comprising one or more hidden layers that classify the speech features of the Mel spectrogram, wherein the one or more hidden layers are located between the Mel generation layer and the output layer. A computer-readable recording medium that records a program for executing the method of claim 9 on a computer.

Images