KR102259299B1 - Book sound classification method using machine learning model of a book handling sounds - Google Patents

Abstract

The present invention relates to a book sound classification using a convolutional neural network (CNN) and, more specifically, to a book sound classification method using a machine learning model for book handling sounds to quantify a user's book handling method. To this end, the method comprises: a step (S100) of acquiring and preparing data of book handling sound; a data augmentation step (S110) of changing the data to increase the number of data in order to increase strength of machine learning; a step (S120) of converting the data of the book handling sound and the augmented data of the book handling sound into a format for deep learning; a step of (S130) of generating a CNN model and performing deep learning-based machine learning by using the converted data; a step (S150) of completing the CNN model after the deep learning is completed; a step (S160) of inputting book sound data into the completed CNN model; and a step (S170) of allowing the CNN model to operate the book sound data to classify book sound.

Description

Book sound classification method using machine learning model of a book handling sounds}

The present invention relates to a method for classifying book sounds using a Convolutional Neural Network (CNN), and more particularly, to classifying the fine sounds made when handling books using deep learning based on CNN (Convolutional Neural Network) and analyzing the user's book. It is about a machine learning method that tries to understand the context.

When purchasing a book, consumers tend to scan through several books and look more closely at the books they are interested in. Most of the methods to determine the level of interest of consumers in books are through surveys. If it is possible to quantitatively understand how consumers read books rather than using a subjective survey method, it will be easy to determine which books have attracted attention.

Books are one of the most common media, and are used for the purpose of obtaining information or having fun. Recently, the e-book market is also being revitalized due to the development of technology, but there is still a lot of demand for paper books due to the physical advantages of paper books. People do various actions such as scanning a book or reading it thoroughly. There are various criteria for people to purchase a book, but the reputation of the book, the author's popularity, and whether it is a bestseller affect the decision to buy a book. Thus, books without such fame tend to pay attention to book cover design and text to attract people's attention. However, it is difficult to know whether these factors really attracted the attention of buyers and even led to actual purchase. Although a survey is used to understand this, it is difficult to determine the degree of interest of buyers for a large number of books.

1. Jeong Jae-Yeol and Shin Dong-Hee, "A Study on the Book Purchase Intention and Propensity of E-Book and Paper Book Readers: Focusing on the Situation of Purchasing at Bookstores," Korean HCI Society Conference, pp 1135-1137, 2014. 2. Convolutional neural network to classify urban sounds," in TENCON 2017-2017 IEEE Region 10 Conference, 2017: IEEE, pp 3089-3092 3. J Salamon, C Jacoby, and J P Bello, "A Dataset and Taxonomy for Urban Sound Research," presented at the Proceedings of the 22nd ACM international conference on Multimedia, Orlando, Florida, USA, 2014 4. Dae-Seo Park, Jun-Il Bang, Hwa-Jong Kim, and Young-Jun Ko, "Study on gender and age classification technology for voice data using CNN," Journal of the Korean Society for Information Technology, vol 16, no 11, pp 11-21, 2018, 5. Se-Young Kim, Hyun-Woong Kim, Chan-Ho Park, and Mok-Dong Jeong, "Deep Learning-based Sound Direction and Type Identification System for the Hearing Impaired," Proceedings of the Korean Society for Information Science and Technology, pp 1896-1898, 2017, 6. B McFee et al., “librosa: Audio and music signal analysis in python,” in Proceedings of the 14th python in science conference, 2015, vol 8, 7. L Muda, M Begam, and I Elamvazuthi, “Voice recognition algorithms using mel frequency cepstral coefficient (MFCC) and dynamic time warping (DTW) techniques,” arXiv preprint arXiv: 1003.4083, 2010.

Accordingly, the present invention has been devised to solve the above problems, and the object of the present invention is to quantify how the user handles the book based on the minute sound generated when handling the book. To provide a book sound classification method using a machine learning model of

In addition, another object of the present invention is a book using a machine learning model of book handling sound that can classify and predict a user's book reading operation based on a fine sound using a convolutional neural network (CNN) To provide a sound classification method.

However, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned are clearly to those of ordinary skill in the technical field to which the present invention belongs from the following description. It will be understandable.

In order to achieve the above technical task, obtaining and preparing the data of the sound of the book (S100); A data augmentation step (S110) of modulating data to increase the number of data in order to increase the robustness of the machine learning system (S110); converting the original data and the augmented data into a format for deep learning (S120); generating a convolutional neural network model, and performing deep learning-based machine learning using the converted data (S130); Completing the convolutional neural network model after the deep learning is completed (S150); inputting the book sound data to the convolutional neural network model for which learning is completed (S160); and an inference step (S170) in which the convolutional neural network model classifies the book sound by calculating the book sound data; a book sound classification method using a machine learning model of book handling sound is provided.

In addition, the sound of book handling includes the sound of turning the bottom of the book one by one (C0), the sound of turning one at a time (C1), the sound of pressing the centerline of the book (C2), and the sound of turning pages one by one. The sound of motion (C3), the sound of closing the bookshelf (C4), the sound of flipping a book over (C5), the sound of moving a finger on the bookshelf (C6), the sound of writing with fingers and at least one sound related to the underlining operation C7.

In addition, the data augmentation step (S110) may include at least one of a step of adding background music to the data of sound handling a book, a step of adding white noise, changing the pitch of the sound source, and moving the time axis of the sound source. can

In addition, in the conversion step (S120), the data of the sound handling the book is converted into an mp3 file in the range of 0.5 seconds to 2 seconds.

In addition, the deep learning step (S130) of the neural network model, the first convolution step (S131) using the transformed data; calculating using the first activation function (S132); performing a second convolution using the calculated data (S133); calculating using the second activation function (S134); pooling the calculated data (S135); and dropping out the neural network model (S136).

In addition, the first convolution step ( S131 ) to the dropout step ( S136 ) may be repeatedly performed a plurality of times.

In addition, FC (Fully Connected Layer) step of flattening the neural network model (S137); and a soft max step (S138) of determining the classification of book sounds.

In addition, between the step of deep learning-based machine learning (S130) and the step of completing the convolutional neural network model (S150), input test data to confirm the performance of the completed convolutional neural network model (S140) include more

According to an embodiment of the present invention, it is possible to accurately predict the user's motion only by sound by trying to analyze 8 motions through the book.

In addition, a very objective and systematic analysis is possible by using sound data and a convolutional neural network instead of conducting a survey as in the prior art to analyze the reader's degree of interest in the book.

In addition, if you analyze the various sounds generated in the process of looking at a book, it has the advantage of being able to easily and reliably determine which book attracted people's attention. In other words, if the reader is interested in a book, he will slowly turn through or underline one page at a time, and if he is not interested, he will flip through it quickly or quickly close the bookshelf. If the reader understands how the reader is looking at the book based on the sound made during these actions, it can be used to quantitatively determine the level of interest.

In addition, the proposed convolutional neural network structure is described in the most general form, and performance is improved by using various convolutional neural network technologies such as shortcut connection or skip connection and cross-layer connections. can be derived.

However, the effects obtainable in the present invention are not limited to the above-mentioned effects, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description. I will be able to.

The following drawings attached to this specification illustrate preferred embodiments of the present invention, and serve to further understand the technical spirit of the present invention together with the detailed description of the present invention to be described later, so the present invention is described in such drawings It should not be construed as being limited only to
1 is an exemplary spectrum showing an example used for data augmentation of the present invention;
2 is an example of classifying eight operations that can be done with books in the present invention;
3 is an example of a spectrum for each classification in the present invention,
4a is a flowchart of a book sound classification method using a machine learning model of book handling sound according to an embodiment of the present invention;
Figure 4b is a detailed flowchart of the construction and deep learning step (S130) of the convolutional neural network model in Figure 4a;
5 is a confusion matrix graph showing the performance of a model according to an embodiment of the present invention;
6 is a fusion matrix graph showing the performance of a model using a conventional MFCC algorithm for comparison with the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art may easily implement the present invention. However, since the description of the present invention is merely an embodiment for structural or functional description, the scope of the present invention should not be construed as being limited by the embodiment described in the text. That is, since the embodiments can be variously changed and have various forms, the scope of the present invention should be understood to include equivalents capable of realizing the technical idea. In addition, since the object or effect presented in the present invention does not mean that a specific embodiment should include all or only such effects, the scope of the present invention should not be understood as being limited thereto.

The meaning of the terms described in the present invention should be understood as follows.

Terms such as "first" and "second" are used to distinguish one component from other components, and the scope of rights is not limited by these terms. For example, a first component may be referred to as a second component, and similarly, a second component may be referred to as a first component. When a component is referred to as being "connected" to another component, it should be understood that although it may be directly connected to the other component, another component may exist in the middle. On the other hand, when a component is referred to as being "directly connected" to another component, it should be understood that there is no other component in the middle. On the other hand, other expressions describing the relationship between components, that is, "between" and "just between" or "neighboring to" and "directly neighboring to" should be interpreted as well.

Singular expressions are to be understood as including plural expressions unless the context clearly indicates otherwise, and terms such as "comprises" or "have" refer to the specified features, numbers, steps, actions, components, parts, or these. It is to be understood that it is intended to designate that a combination exists and does not preclude the presence or addition of one or more other features, numbers, steps, actions, components, parts, or combinations thereof.

All terms used herein have the same meaning as commonly understood by one of ordinary skill in the field to which the present invention belongs, unless otherwise defined. Terms defined in commonly used dictionaries should be interpreted as having meanings in the context of related technologies, and cannot be interpreted as having an ideal or excessively formal meaning unless explicitly defined in the present invention.

Hereinafter, the configuration of the preferred embodiment will be described in detail with reference to the accompanying drawings. 4A is a flowchart illustrating a method for classifying book sounds using a machine learning model of book handling sounds according to an embodiment of the present invention. As shown in Fig. 4a, first, the data of the sound of handling a book is obtained and prepared (S100). About 50,000 sound files of 15 seconds in length were used for the sound of book handling. The sound was recorded and collected using a built-in app using a general cell phone recorder/PC microphone.

On the other hand, book handling sounds were defined and classified into eight categories. That is, the sound of turning the bottom of the book one by one (C0), the sound of turning one at a time (C1), the sound of pressing the center line of the book firmly (C2), and the sound of turning the pages one by one (C3). Sound about the sound of closing the bookcase (C4), the sound of turning a book over and flapping it (C5), the sound of moving a finger on the bookshelf (C6), and the sound of underlining text with a finger (C7) ) was classified as a sound related to

Figure 2 is an example of classifying eight operations that can be done with books in the present invention, and Figure 3 is an example of a spectrum for each classification in the present invention.

Next, a data augmentation step of increasing the number of data by changing the acquired sound data for handling a book is performed (S110). 1 is an exemplary spectrum showing an example used for data augmentation of the present invention. As shown in Figure 1, data augmentation is a step of adding background music to the sound data of the book handling, adding white noise, changing the pitch of the sound source, moving the time axis of the sound source (Shifting) step was applied. [Table 1] shows the classification (Class) of the sound of these books and the number of data and data augmentation.

action Class Data Augmentation flip through the bottom of the book Turn C0 2,262 11,310 flip through bookshelves Flip C1 1,630 8,150 press the center line of the book Rub C2 1,602 8,010 Skip through the pages one by one One Paper C3 1,013 5,061 book cover Close C4 909 4,545 flip the book over Flutter C5 874 4,370 move your finger Finger C6 930 4,650 underline the finger Underline C7 870 4,350 Sum 10,090 50,446

Next, the data is converted into a predetermined format in order to perform deep learning-based machine learning with the original data and the augmented data (S120). Individual data is in a wave format of a specific length, and a predetermined format can be selected in the range of 0.5 seconds to 2 seconds. If it is too short, it is difficult to record sound characteristics and extract from the artificial neural network system. If it is too long, unnecessary noise is recorded together. may be, or it may take longer to learn. In addition, various formats such as m4a, mp3, and wav may be applied to the file format. Using a programming library (eg, LibROSA in Python), one-dimensional sound data was used as learning data by making two-dimensional spectrogram data containing information on time, frequency, and intensity.

Then, a convolutional neural network model is generated, and deep learning-based machine learning is performed using the converted data (S130). It consists of 6 convolutional neural network layers, and includes pooling and dropout for each layer. The activation function is a function that converts the sum of input signals to individual neurons of the neural network into an output signal, and a rectified linear unit (ReLU) is used as the activation function. Although various optimization techniques can be used, in this example, the neural network optimization process was performed using the Adam Optimizer. 4B is a detailed flowchart of the construction of the convolutional neural network model and the deep learning step (S130) of FIG. 4A. As shown in FIG. 4B, a first convolution is performed using the transformed data (S131). Then, the operation is performed using the first activation function (S132). As the first activation function, a rectified linear unit (ReLU) was used.

Next, a second convolution is performed using the calculated data (S133). Then, it is calculated using the second activation function (S134). The second activation function also used a rectified linear unit (ReLU).

Then, the calculated data is pooled (S135). Pooling is a process of reducing data, selecting one number as a representative number within the corresponding range and discarding the rest. For pooling, maximum pooling or average pooling may be applied.

Then, the step of dropping out (Dropout) the neural network model (S136) is executed. Dropout is a technique that turns off nodes and their connections that are not involved in learning or have a small weight in the neural network structure, and is more effective than learning by weight.

The above-described first convolution step S131 to the dropout step S136 are repeatedly performed a plurality of times (eg, 3 to 5 times).

Then, the FC (Fully Connected Layer) step (S137) for flattening the neural network model is executed. The FC stage is the process of "flattening" the output of the previous layer and transforming it into a single vector that can be the input of the next stage.

Then, a step (S138) of calculating a soft max (Soft Max) function for determining the classification of book sounds is executed. The soft pulse function is a real number between 0 and 1.0, which can be interpreted as a probability of classification.

Then, after the deep learning is completed, the convolutional neural network model is completed (S150).

Then, the performance of the completed convolutional neural network model is checked by inputting test data (S140). For the test data, an appropriate number is selected from the data of the sound of the initial book.

Then, the actual book sound data is input to the completed convolutional neural network model (S160). Then, the convolutional neural network model calculates inference based on the input actual book sound data and outputs that it corresponds to one of eight book-handling sounds (e.g., a sound about the operation (C0) of turning the bottom of a book one by one) do. Therefore, if a mobile device to which such a convolutional neural network model is applied is operated in a library or bookstore, it is possible to grasp the context of using a book, and representatively, it will be possible to evaluate the reader's interest in the book (S170) .

Experiment result

5 is a confusion matrix graph showing the performance of a model according to an embodiment of the present invention. As shown in FIG. 5 , as a result of the training, it was confirmed that 8 motions were correctly predicted with an accuracy of about 91%. Precision and Recall were used as performance evaluation indicators of the model, and the results are shown in [Table 2]. Average Precision and Recall were used as performance evaluation indicators of the model. Average Precision and Recall were 0.92 and 0.91, respectively.

action Precision Recall F1-score Turn 0.84 1.00 0.92 Flip 0.96 0.84 0.90 Rub 0.97 0.97 0.97 One Paper 0.85 0.84 0.84 Close 0.96 0.94 0.95 Flutter 0.98 0.88 0.93 Finger 0.90 0.95 0.92 Underline 0.93 0.74 0.83 Average 0.92 0.91 0.91

Comparison with MFCC Algorithm

MFCC (Mel-Frequency Cepstral Coefficieiniti) is an algorithm that reflects the way humans hear sound and is widely used in conventional speech recognition. The performance of CNN according to the present invention was compared with that of the MFCC algorithm. The same data shown in [Table 1] was applied to the MFCC algorithm. Through the MFCC-based feature extraction technique, continuous one-dimensional sound data was transformed into a set of feature values. At this time, the feature value may include zero_crossing_rate, spectral_rolloff, spectral_centroid, spectral_contrast, spectral_bandwidth, etc. that can be obtained from data, but since it is a conventional technique, detailed description is omitted. And the validation accuracy was obtained using a random forest classifier. 6 is a fusion matrix graph showing the performance of a model using the conventional MFCC algorithm for comparison with the present invention, and [Table 3] is the experimental result of the MFCC algorithm. As can be seen in Figure 6 and [Table 3], the MFCC algorithm showed a relatively low verification accuracy performance of 84%, which proves that the CNN-based model of the present invention has the advantage of being generalizable.

action Precision Recall F1-score Turn 0.89 0.96 0.92 Flip 0.93 0.75 0.83 Rub 0.88 0.91 0.89 One Paper 0.51 0.70 0.59 Close 0.99 0.97 0.98 Flutter 0.99 0.98 0.98 Finger 0.95 0.96 0.95 Underline 0.65 0.41 0.50 Average 0.86 0.85 0.85

Detailed description of the preferred embodiments of the present invention disclosed as described above has been provided to enable those skilled in the art to implement and practice the present invention. Although the above has been described with reference to preferred embodiments of the present invention, it will be understood by those skilled in the art that various modifications and changes can be made to the present invention without departing from the scope of the present invention. For example, a person skilled in the art can use each configuration described in the above-described embodiments in a way in combination with each other. Accordingly, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The present invention may be embodied in other specific forms without departing from the spirit and essential features of the present invention. Therefore, the detailed description above should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention. The invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In addition, claims that are not explicitly cited in the claims may be combined to form an embodiment or may be included as a new claim by amendment after filing.

Claims (8)

Preparing to obtain and prepare data of the sound of the book (S100);
a data augmentation step (S110) of increasing the number of data by changing the data to increase the robustness of machine learning;
Converting the data of the sound of the book and the augmented data into a format for deep learning (S120);
generating a convolutional neural network model, and performing deep learning-based machine learning using the converted data (S130);
Completing the convolutional neural network model after the deep learning learning is completed (S150);
inputting book sound data into the completed convolutional neural network model (S160); and
The book sound classification method using a machine learning model of book handling sound, characterized in that it comprises; the convolutional neural network model calculating the book sound data to classify the book sound (S170). The method of claim 1,
The sound of the book handling is a sound related to the operation (C0) of turning the bottom of the book one by one (C0), the sound of the operation of turning at once (C1), the sound of pressing the center line of the book (C2), the operation of turning pages The sound of (C3), the sound of closing the bookcase (C4), the sound of flipping a book over (C5), the sound of moving a finger on the bookshelf (C6), the sound of underlining the text with your finger A book sound classification method using a machine learning model of book handling sound, characterized in that it includes at least one of sounds related to the flicking operation (C7). The method of claim 1,
The data augmentation step (S110) is,
A book handling sound machine comprising at least one of adding background music to the sound data of the book, adding white noise, changing the pitch of the sound source, and moving the time axis of the sound source A book sound classification method using a learning model. The method of claim 1,
The conversion step (S120) is,
A book sound classification method using a machine learning model of book handling sound, characterized in that the data of the book handling sound is converted into an audio file set in the range of 0.5 seconds to 2 seconds. The method of claim 1,
The deep learning step (S130) of the neural network model is,
performing a first convolution using the converted data (S131);
calculating using the first activation function (S132);
performing a second convolution using the calculated data (S133);
calculating using the second activation function (S134);
pooling the calculated data (S135); and
Dropout the neural network model (S136); Book sound classification method using a machine learning model of book handling sound, characterized in that it includes. The method of claim 5,
The book sound classification method using a machine learning model of book handling sound, characterized in that the first convolution step (S131) to the dropout step (S136) are repeatedly executed a plurality of times. The method of claim 5,
FC (Fully Connected Layer) step of flattening the neural network model (S137); and
A book sound classification method using a machine learning model of book handling sound, characterized in that it further comprises a soft max step (S138) of determining the classification of the book sound. The method of claim 1,
Between the deep learning-based machine learning step (S130) and the convolutional neural network model completion step (S150),
The book sound classification method using a machine learning model of book handling sound, characterized in that it further comprises the step (S140) of confirming the performance of the completed convolutional neural network model by inputting test data.

Images