KR102236458B1 - Method and system for animal sounds classification using interpolation and long short term memory-fully convolutional network - Google Patents

Abstract

Disclosed are a method and a system for classifying companion animal sounds. The sound classification method includes: obtaining sound data of the companion animal through a sound detection sensor attached to the companion animal; Transforming the dimension of the sound data using an interpolation method; And classifying the sound data whose dimensions have been transformed through deep learning.

Description

Method and system for classifying dog sounds with separation anxiety by applying interpolation and LSDTM-FCN techniques

The description below relates to a technique for classifying the sounds of companion animals.

Due to the increase in single-person households, the population raising companion animals such as dogs and cats is increasing.

Accordingly, related researches to grasp or communicate with companion animals are also increasing.

For example, Korean Patent Publication No. 10-1703413 (registration date January 31, 2017) discloses a pet monitoring device, a control terminal, and a method thereof.

Provides a method and system for classifying dog sounds with separation anxiety symptoms using interpolation and LSTM-FCN (Long short-term memory-Fully Convolutional Network) techniques.

Provides a method and system for classifying dog sounds through time-series sound data analysis.

A sound classification method performed by a computer device, the method comprising: acquiring, in at least one processor, sound data of the companion animal through a sound detection sensor attached to the companion animal; Transforming, in the at least one processor, the dimension of the sound data using an interpolation method; And classifying, in the at least one processor, sound data whose dimensions have been transformed through deep learning.

According to an aspect, prior to the converting, the at least one processor may further include labeling the sound data as an event of a predefined sound characteristic.

According to another aspect, the at least one processor may further include normalizing the labeled data.

According to another aspect, the converting may increase the dimension of the sound data to within a predetermined multiple through linear conversion.

According to another aspect, in the classifying step, the sound data may be classified using a long short-term memory -Fully Convolutional Network (LSTM-FCN) model.

According to another aspect, the step of classifying includes a process of reducing the data dimension through a dimension shuffle layer and then learning the reduced dimension data through an LSTM layer, and a plurality of conballs. After extracting features through a convolution layer, the sound data can be classified using an LSTM-FCN model that includes a process of reducing the size of data output from the convolution layer through a pooling layer. have.

A computer program stored in a computer-readable recording medium to execute a sound classification method, the sound classification method comprising: acquiring sound data of the companion animal through a sound detection sensor attached to the companion animal; Transforming the dimension of the sound data using an interpolation method; And it provides a computer program comprising the step of classifying the sound data of the dimension is transformed through deep learning.

A computer-implemented sound classification system, comprising: at least one processor configured to execute computer-readable instructions included in a memory, wherein the at least one processor includes the companion animal through a sound detection sensor attached to the companion animal. A data acquisition unit that acquires sound data of the sound; A data processing unit that converts the dimension of the sound data using an interpolation method; And it provides a sound classification system including a data classification unit for classifying the sound data whose dimensions have been transformed through deep learning.

According to an aspect, the sound detection sensor may be configured as an attachable wireless sensor.

According to another aspect, the data processing unit normalizes the labeled data by labeling the sound data as an event of a predefined sound characteristic, and then increases the dimension of the normalized data to within a predetermined multiple through linear transformation. The data classification unit reduces the data dimension through the dimensional shuffle layer and then learns the reduced-dimensional data through the LSTM layer, and extracts features through a plurality of convolution layers, and then the convolution through the pooling layer. The sound data may be classified using an LSTM-FCN model including a process of reducing the size of data output from the lusion layer.

According to embodiments of the present invention, it is possible to classify the sound of a dog suffering from separation anxiety using an interpolation method and an LSTM-FCN technique.

According to embodiments of the present invention, it is possible to more accurately classify a dog's sound through time-series sound data analysis.

1 is a block diagram showing an example of a computer device according to an embodiment of the present invention.
2 is a diagram illustrating an example of components that may be included in a processor of a computer device according to an embodiment of the present invention.
3 is a flowchart illustrating an example of a sound classification method that can be performed by a computer device according to an embodiment of the present invention.
4 shows an example of normalization of sound data in an embodiment of the present invention.
5 illustrates data comparison before and after applying the interpolation method in an embodiment of the present invention.
6 shows an example of a learning model for sound classification according to an embodiment of the present invention.
7 shows a result of verification of data to which an interpolation method is applied in an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

For dogs, it is important to take measures to identify emotions such as separation anxiety disorder.

Representative sound characteristics of dog separation anxiety symptoms include barking, growing, howling, and whining.

A general sound sensor (for example, a recorder) has a large resource constraint and a large amount of load and battery consumption due to data transmission, and thus a method for supplementing real-time and energy efficiency is required.

Accordingly, in the present invention, an IoT sensor (wireless sensor) is attached to detect a dog's emotion to detect a unit action.

In addition, a sound detection sensor having low power and high data size efficiency has a disadvantage of low accuracy compared to a general sound sensor (recorder).

Accordingly, the present invention proposes a data processing process to compensate for the disadvantages of the sound sensor, and in particular, is intended to compensate for the disadvantages of the sound sensor by applying a deep learning technique.

Embodiments of the present invention relate to a method and system for classifying dog sounds.

In the embodiments including those specifically disclosed in the present specification, sound data of the dog may be obtained using a wireless sensor, and sound data of the dog may be classified using an interpolation method and an LSTM-FCN technique, through which accuracy In terms of efficiency, cost savings, etc., significant advantages can be achieved.

1 is a block diagram showing an example of a computer device according to an embodiment of the present invention.

A sound classification system according to embodiments of the present invention may be implemented through the computer device 100 of FIG. 1. For example, a computer program according to an embodiment may be installed and driven in the computer device 100, and the computer device 100 is a sound classification method according to embodiments of the present invention under control of the driven computer program. You can do it.

The sound classification system may be composed of a PC-based program or an application dedicated to a mobile terminal. The sound classification system according to the present embodiment may be implemented in the form of a program that operates independently, or may be implemented in the form of an in-app form of a specific application so as to be able to operate on the specific application.

The sound classification system through the computer device 100 may be a fixed terminal or a mobile terminal. For example, the computer device 100 may be a device such as a computer (PC), a smart phone, a tablet, and a notebook.

As shown in FIG. 1, the computer device 100 may include a memory 110, a processor 120, a communication interface 130, and an input/output interface 140 as components for executing the sound classification method. .

The memory 110 is a computer-readable recording medium and may include a permanent mass storage device such as a random access memory (RAM), a read only memory (ROM), and a disk drive. Here, a non-destructive large-capacity recording device such as a ROM and a disk drive may be included in the computer device 100 as a separate permanent storage device separate from the memory 110. In addition, an operating system and at least one program code may be stored in the memory 110. These software components may be loaded into the memory 110 from a computer-readable recording medium separate from the memory 110. Such a separate computer-readable recording medium may include a computer-readable recording medium such as a floppy drive, a disk, a tape, a DVD/CD-ROM drive, and a memory card. In another embodiment, software components may be loaded into the memory 110 through a communication interface 130 other than a computer-readable recording medium. For example, software components may be loaded into the memory 110 of the computer device 100 based on a computer program installed by files received through the network 160.

The processor 120 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. Instructions may be provided to the processor 120 by the memory 110 or the communication interface 130. For example, the processor 120 may be configured to execute a command received according to a program code stored in a recording device such as the memory 110.

The communication interface 130 may provide a function for the computer device 100 to communicate with other computer devices (not shown) through the network 160. For example, requests, commands, data, files, etc., generated by the processor 120 of the computer device 100 according to a program code stored in a recording device such as the memory 110, are transmitted through the network ( 160) can be transmitted to other computer devices. Conversely, signals, commands, data, files, etc. from other computer devices may be received by the computer device 100 through the communication interface 130 of the computer device 100 via the network 160. Signals, commands, data, etc. received through the communication interface 130 may be transmitted to the processor 120 or the memory 110, and the file, etc. may be a storage medium (described above) that the computer device 100 may further include. Permanent storage).

The input/output interface 140 may be a means for an interface with the input/output device 150. For example, the input device may include a device such as a microphone, a keyboard, or a mouse, and the output device may include a device such as a display or a speaker. As another example, the input/output interface 140 may be a means for interfacing with a device in which input and output functions are integrated into one, such as a touch screen. The input/output device 150 may be configured with the computer device 100 and one device.

In addition, in other embodiments, the computer device 100 may include more components than those of FIG. 1. However, there is no need to clearly show most of the prior art components. For example, the computer device 100 may be implemented to include at least some of the input/output devices connected to the input/output interface 140 described above, or may further include other components such as a transceiver, a camera, various sensors, and a database. It can also be included.

FIG. 2 is a diagram showing an example of components that can be included in a processor of a computer device according to an embodiment of the present invention, and FIG. 3 is a classification of sounds that can be performed by a computer device according to an embodiment of the present invention. It is a flow chart showing an example of the method.

As shown in FIG. 2, the processor 120 may include a data acquisition unit 201, a data processing unit 202, and a data classification unit 203. Components of the processor 120 may be expressions of different functions performed by the processor 120 according to a control command provided by at least one program code. For example, the data acquisition unit 201 may be used as a functional expression that operates to control the computer device 100 so that the processor 120 acquires sound data.

The processor 120 and components of the processor 120 may perform steps included in the sound classification method of FIG. 3. For example, the processor 120 and the components of the processor 120 may be implemented to execute an instruction according to the code of the operating system included in the memory 320 and the at least one program code described above. Here, at least one program code may correspond to a code of a program implemented to process a sound classification method.

The sound classification method may not occur in the illustrated order, and some of the steps may be omitted or an additional process may be further included.

The processor 120 may load a program code stored in a program file for a sound classification method into the memory 320. For example, a program file for a sound classification method may be stored in the permanent storage device 330 described with reference to FIG. 3, and the processor 120 may use a program file from a program file stored in the permanent storage device 330 through a bus. The computer device 100 may be controlled so that the code is loaded into the memory 320. At this time, each of the data acquisition unit 201, the data processing unit 202, and the data classification unit 203 included in the processor 120 and the processor 120 These may be different functional expressions of the processor 120 for executing the command and then executing the sound classification method. In order to execute the sound classification method, the processor 120 and components of the processor 120 may directly process an operation according to a control command or control the computer device 100.

The data acquisition unit 201 may acquire analog data using a sound sensor. Since the behavioral radius of the dog in the home is not limited to a certain section, a wearable device is required for continuous data acquisition. In order to acquire sound data of a dog, a sound detection sensor composed of a wireless sensor may be attached to the dog's body to collect sound data from the sensor. In this case, the sound detection sensor may use an IoT sensor having a communication function as a type of wearable device. In addition, Wi-Fi can be used for analog data transmission of wireless wearable devices.

The sound data sensed by the sound sensor may be transferred to a workstation, that is, a sound classification system for labeling, various data processing, and storage and classification.

The collected sound data is a series of time series data, which are sounds corresponding to four events, and these events can be defined as barking, growling, crying, and whining, respectively, which are representative sound characteristics of separation anxiety symptoms. have.

The data processing unit 202 may label the sound data acquired by the data acquisition unit 201 with each predefined event (barking sound, growling sound, howling sound, whining sound).

The data processing unit 202 may perform dimensional transformation by applying normalization and linear transformation techniques to the labeled data.

During the data collection process in the sound detection sensor, the range of peak data is not constant.

The data processing unit 202 may perform a normalization (0-1 Normalization) to match the data range and make the distribution similar to compensate for the disadvantage of a sensor having an uneven data value range.

The formula for normalization is as follows.

[Equation 1]

Here, d is a series of time series data sets, and d _i means data input as a subset of d. It can be converted into normalized data by using the maximum and minimum values of the data set d.

4 shows an example of normalized data.

To process one event, sound data has different lengths.

The dimension of sound data is increased through linear transformation. In this case, the data processing unit 202 may perform a preprocessing step for interpolating a small number of data because the number of time series data for each event is different.

The data processing unit 202 may convert the normalized sound data into a constant data length using an interpolation method.

The linear transformation technique can apply bicubic interpolation, and through this, a small number of data can be expanded without significant change.

The formula of the interpolation method is as follows.

[Equation 2]

Interpolation is mainly used in image processing to enlarge or reduce an image using values of surrounding pixels. The bicubic interpolation method is determined using the product of the values of 16 adjacent pixels and the weight according to the distance. This is calculated using differentiation and cross-differentiation, and in Equation 2, x and y denote pixel coordinates of time series data.

In the present invention, the dimension of the data is increased within a predetermined multiple (1 to 3 times).

5 shows comparison of data before (left) and after (right) the interpolation method is applied. Each represents an increase of 1 to 3 times, and it can be seen that there is no significant change in data characteristics.

After increasing the data dimension by applying the interpolation method, four events can be classified by applying the LSTM-FCN technique.

The data classification unit 203 may classify data subjected to normalization and dimensional transformation through the data processing unit 202 into four events using the LSTM-FCN technique.

6 shows a schematic diagram of LSTM-FCN for classifying four events.

Data that has been labeled and preprocessed (normalized and linearly transformed) is input and learned in two processes, and the final output data (Activation map) is formed by connecting the two learned data.

First, dimensional reduction of the entire data is induced through a dimension shuffle layer. This is a step to solve the problem of overfitting, and the reduced-dimensional data is learned through an LSTM layer. At this time, the LSTM model adds a cell-state to the hidden state of Recurrent Neural Networks (RNN), and the gradient gradually decreases through backpropagation, reducing the learning ability. The LSTM model designed to solve the'vanishing gradient) problem' is used.

At the same time, features are extracted through 3 stages of the convolution layer using filter sizes of 128, 256, and 128, ReLU activation function, and batch normalization, respectively, and the pooling layer is global. The size of the final output data (Activation map) output from the convolution layer is reduced by applying a global average pooling technique.

The results of the above two steps are combined and Softmax is applied as the final output.

Four types of time-series sound data learned using the LSTM-FCN model are verified and tested using test data to calculate the final result.

7 shows the verification results shown by applying the interpolation method.

Referring to FIG. 7, after data expansion is performed using an interpolation method, it is possible to confirm an increase in accuracy up to 1 to 3 times, and at 3 times or more, it is possible to confirm a decrease in accuracy due to a problem of overfitting.

As described above, according to embodiments of the present invention, by collecting sound data of the dog using a wireless sensor attachable to the dog, battery consumption can be significantly reduced compared to a method of obtaining sound data through general recording, and data transmission energy efficiency is also improved. Can be improved. In particular, according to embodiments of the present invention, data accuracy may be further improved by classifying sound data using an LSTM-FCN model, which is a deep learning technique, after data expansion is performed using an interpolation method.

In the above, the companion dog has been described as a representative example, but the present invention is not limited thereto, and can be extended to a technology for classifying sounds of all animals such as cats and dogs.

The apparatus described above may be implemented as a hardware component, a software component, and/or a combination of a hardware component and a software component. For example, the devices and components described in the embodiments include a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA), and a programmable gate array (PLU). It may be implemented using one or more general purpose or special purpose computers, such as a logic unit), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications executed on the operating system. Further, the processing device may access, store, manipulate, process, and generate data in response to the execution of software. For the convenience of understanding, although it is sometimes described that one processing device is used, one of ordinary skill in the art, the processing device is a plurality of processing elements and/or a plurality of types of processing elements. It can be seen that it may include. For example, the processing device may include a plurality of processors or one processor and one controller. In addition, other processing configurations are possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of these, configuring the processing unit to behave as desired or processed independently or collectively. You can command the device. Software and/or data may be embodyed in any type of machine, component, physical device, computer storage medium or device to be interpreted by the processing device or to provide instructions or data to the processing device. have. The software may be distributed over networked computer systems and stored or executed in a distributed manner. Software and data may be stored on one or more computer-readable recording media.

The method according to the embodiment may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer-readable medium. In this case, the medium may be one that continuously stores a program executable by a computer, or temporarily stores a program for execution or download. Further, the medium may be a variety of recording means or storage means in a form in which a single or several pieces of hardware are combined, but is not limited to a medium directly connected to a computer system, but may be distributed on a network. Examples of media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, magneto-optical media such as floptical disks, And there may be ones configured to store program instructions, including ROM, RAM, flash memory, and the like. In addition, examples of other media include an app store that distributes applications, a site that supplies or distributes various software, and a recording medium or a storage medium managed by a server.

As described above, although the embodiments have been described by the limited embodiments and drawings, various modifications and variations are possible from the above description to those of ordinary skill in the art. For example, the described techniques are performed in a different order from the described method, and/or components such as systems, structures, devices, circuits, etc. described are combined or combined in a form different from the described method, or other components Alternatively, even if substituted or substituted by an equivalent, an appropriate result can be achieved.

Therefore, other implementations, other embodiments, and those equivalent to the claims also fall within the scope of the claims to be described later.

Claims (10)

In the sound classification method performed in a computer device,
Obtaining, by at least one processor, sound data of the companion animal through a sound detection sensor attached to the companion animal;
Labeling, in the at least one processor, the sound data as an event of a predefined sound characteristic;
Normalizing the labeled data in the at least one processor;
Transforming, in the at least one processor, the dimension of the normalized data using an interpolation method; And
In the at least one processor, classifying the data whose dimensions have been transformed through deep learning
Including,
The sound detection sensor is an attached wireless sensor that is a wearable device and is composed of an IoT sensor having a communication function using WiFi,
The event is defined as barking, growing, howling, and whining,
The normalizing step,
Converting the labeled data into the normalized data through the following equation,

(Where d is the time series data set of each event, d _i is the data input as a subset of d, max(d) is the maximum value of d, and min(d) is the minimum value of d)
The converting step,
As a linear transformation technique, the dimension of the normalized data is increased within a predetermined multiple based on the pixel coordinates of the time series data of each event through a bicubic interpolation using differential and cross-differential,
The classifying step,
The process of learning the reduced dimension data through the LSTM layer after reducing the data dimension through the Dimension Shuffle layer, and a plurality of convolution layers and ReLU activation functions and arrangements LSTM-FCN model including the process of reducing the size of data output from the convolutional layer through a pooling layer using a global average pooling technique after extracting features through batch normalization Classifying the dimensionally transformed data into the event by using
Sound classification method, characterized in that. delete delete delete delete delete In a computer program stored in a computer-readable recording medium to execute a sound classification method,
The sound classification method,
Acquiring sound data of the companion animal through a sound detection sensor attached to the companion animal;
Labeling the sound data as an event of a predefined sound characteristic;
Normalizing the labeled data;
Transforming the dimension of the normalized data using an interpolation method; And
Classifying the dimensionally transformed data through deep learning
Including,
The sound detection sensor is an attached wireless sensor that is a wearable device and is composed of an IoT sensor having a communication function using WiFi,
The event is defined as barking, growing, howling, and whining,
The normalizing step,
Converting the labeled data into the normalized data through the following equation,

(Where d is the time series data set of each event, d _i is the data input as a subset of d, max(d) is the maximum value of d, and min(d) is the minimum value of d)
The converting step,
As a linear transformation technique, the dimension of the normalized data is increased within a predetermined multiple based on the pixel coordinates of the time series data of each event through a bicubic interpolation using differential and cross-differential,
The classifying step,
The process of learning the reduced dimension data through the LSTM layer after reducing the data dimension through the Dimension Shuffle layer, and a plurality of convolution layers and ReLU activation functions and arrangements LSTM-FCN model including a process of reducing the size of data output from the convolutional layer through a pooling layer using a global average pooling technique after extracting features through batch normalization Classifying the dimensionally transformed data into the event by using
Characterized in that, a computer program. In the computer-implemented sound classification system,
At least one processor configured to execute computer readable instructions contained in memory
Including,
The at least one processor,
A data acquisition unit that acquires sound data of the companion animal through a sound detection sensor attached to the companion animal;
A data processing unit that converts the dimension of the sound data using an interpolation method; And
A data classification unit that classifies the data whose dimensions have been transformed through deep learning
Including,
The sound detection sensor is an attached wireless sensor that is a wearable device and is composed of an IoT sensor having a communication function using WiFi,
The data processing unit,
After normalizing the labeled data by labeling the sound data as an event of a predefined sound characteristic, the dimension of the normalized data is increased to within a predetermined multiple through linear transformation,
The event is defined as barking, growing, howling, and whining,
The data processing unit,
Converting the labeled data into the normalized data through the following equation,

(Where d is the time series data set of each event, d _i is the data input as a subset of d, max(d) is the maximum value of d, and min(d) is the minimum value of d)
As a linear transformation technique, the dimension of the normalized data is increased within a predetermined multiple based on the pixel coordinates of the time series data of each event through a bicubic interpolation using differential and cross-differential,
The data classification unit,
The process of learning the reduced dimension data through the LSTM layer after reducing the data dimension through the Dimension Shuffle layer, and a plurality of convolution layers and ReLU activation functions and arrangements LSTM-FCN model including the process of reducing the size of data output from the convolutional layer through a pooling layer using a global average pooling technique after extracting features through batch normalization Classifying the dimensionally transformed data into the event by using
Sound classification system, characterized in that. delete delete

Images