KR102647512B1 - Method for generating meeting minutes based on multi channel voice data - Google Patents

Abstract

In one embodiment of the present disclosure for solving the above-described problem, a computer program stored in a computer-readable storage medium executable by one or more processors, wherein the computer program causes the one or more processors to perform the following operations The operations include: receiving first voice data from a first user terminal and receiving second voice data from a second user terminal, integrated voice based on the first voice data and the second voice data. It may include generating data and generating a conversation log using the integrated voice data using a voice recognition model.

Description

{METHOD FOR GENERATING MEETING MINUTES BASED ON MULTI CHANNEL VOICE DATA}

The present disclosure relates to a computing device that provides a conversation record service, and more specifically, to a computing device that provides a conversation record service based on a voice file containing conversation content between a plurality of users.

Today, office workers frequently hold meetings with executives inside and outside the company. Writing minutes of the meeting contents during the meeting process is very helpful in terms of work efficiency for organizations and individuals. However, since a separate stenographer is required to take minutes during a meeting, additional costs may be incurred, and the quality of the minutes may vary depending on the stenographer's capabilities.

Meanwhile, market demand for artificial intelligence has recently been increasing due to issues such as AlphaGo, and artificial intelligence is evolving at a rapid pace as the amount of information that can be processed increases with the development of computer technology. Additionally, as artificial intelligence evolves at a rapid pace, the accuracy of voice recognition using artificial neural networks is increasing.

Accordingly, in Republic of Korea Patent Publication No. 10-2008-0037402 (April 30, 2008), an algorithm for separating a plurality of speakers based on the frequency band for a voice file input through a single channel is used to separate speakers and convert them into text, A system for automatically recording meeting minutes is being launched.

The present disclosure has been made in response to the above-described background technology, and is intended to provide a computing device that provides a conversation record service based on a voice file containing conversation content between a plurality of users.

According to one embodiment of the present disclosure for solving the above-described problem, a computer program stored in a computer-readable medium executable by one or more processors is disclosed. The computer program causes the one or more processors to perform the following operations, which include: receiving first voice data from a first user terminal and receiving second voice data from a second user terminal; The method may include generating integrated voice data based on the first voice data and the second voice data, and generating a conversation record using the integrated voice data using a voice recognition model.

Alternatively, dividing the first voice data into one or more first voice data subdata based on a speech interval, and dividing the second voice data into one or more second voice data subdata based on a speech interval It may further include an operation of identifying a start point and an end point of each of the first voice data subdata and the second voice data subdata.

Alternatively, the operation of identifying the start point and end point of each of the first voice data subdata and the second voice data subdata may include the first voice data through network packet exchange between the first user terminal and the second user terminal. An operation of synchronizing the start and end points of data and the second voice data or the start points of the first voice data and the second voice data based on the similarity of the first voice data sub-data and the second voice data sub-data and may include at least one of the operations of synchronizing the end point.

Alternatively, the method may further include identifying an overlap section based on the start and end points of each of the first voice data subdata and the second voice data subdata.

Alternatively, the operation of generating integrated voice data based on the first voice data and the second voice data includes performing preprocessing on each voice data to increase the recognition rate for the integrated voice data; , the preprocessing may include at least one of standardization or frequency filtering for the signal size included in each of the one or more voice data.

Alternatively, the operation of generating integrated voice data based on the first voice data and the second voice data includes the operation of distinguishing speakers based on the voice signal size of each of the first voice data and the second voice data. Alternatively, it may include at least one of an operation of distinguishing speakers through a statistical acoustic model based on the first voice data and the second voice data.

Alternatively, the operation of generating the integrated voice data by distinguishing a speaker based on the voice signal size of each of the first voice data and the second voice data includes a first voice data frame based on the first voice data and It may include an operation of distinguishing speakers by comparing voice signal sizes of each second voice data frame based on the second voice data.

Alternatively, the operation of generating the integrated speech data by distinguishing speakers through a statistical acoustic model may include distinguishing the speaker of the current frame based on speaker information of adjacent frames, or the operation of distinguishing the speaker of the current frame based on speaker information of adjacent frames, or The voice data may include at least one operation of distinguishing speakers based on the speech time ratio of each speaker.

Alternatively, the operation of distinguishing speakers based on the ratio of each speaker's speech time between the first voice data and the second voice data may include, when identifying an overlapping section, utterance for each speaker based on the speaker in the frame excluding the overlapping section. An operation of distinguishing speakers by calculating a time ratio or, when identifying an overlapped section, may include an operation of distinguishing speakers based on the speech time ratio of each speaker to the previous frame of the overlapped section frame.

Alternatively, when identifying an overlapping section, the operation of distinguishing speakers based on the ratio of speech time for each speaker to the previous frame of the overlapping section frame is performed when the previous frame of the overlapping section frame is greater than or equal to a predetermined ratio of voice data. It can be.

In another embodiment of the present disclosure, a method for providing a conversation record service is disclosed. The method is a method for providing a conversation record service, comprising receiving first voice data from a first user terminal and receiving second voice data from a second user terminal, the first voice data and the second voice It may include a method of generating integrated voice data based on data and a method of generating a conversation record using the integrated voice data using a voice recognition model.

A server that provides a conversation service in another embodiment of the present disclosure, comprising a server processor including one or more cores, a server memory storing program codes executable on the processor, and a server network unit transmitting and receiving data to and from one or more user terminals. And the processor receives first voice data from a first user terminal, receives second voice data from a second user terminal, and integrates voice data based on the first voice data and the second voice data. Data can be generated, and a conversation record can be created using the integrated voice data using a voice recognition model.

The present disclosure can provide a computing device that provides a conversation record service based on a voice file containing conversation content between a plurality of users.

Various aspects will now be described with reference to the drawings, where like reference numerals are used to collectively refer to like elements. In the following examples, for purposes of explanation, numerous specific details are set forth to provide a comprehensive understanding of one or more aspects. However, it will be clear that such aspect(s) may be practiced without these specific details.
Figure 1 shows a conceptual diagram illustrating an overall system for providing a conversation record service related to an embodiment of the present disclosure.
FIG. 2 shows a block diagram of a computing device providing a conversation record service related to an embodiment of the present disclosure.
Figure 3 shows a flowchart for providing a conversation record service related to an embodiment of the present disclosure.
Figure 4 shows an example of a user interface including a user list display screen and a conversation list display screen according to an embodiment of the present disclosure.
Figure 5 shows an example of a process for opening a chat room and a user interface for a chat room including a plurality of users according to an embodiment of the present disclosure.
Figure 6 shows an example of a user interface that provides a conversation log display screen in response to a user terminal's selection input for a specific conversation among one or more conversation lists performed by the user according to an embodiment of the present disclosure.
FIG. 7 illustrates an example of a user interface displaying a screen that reproduces a specific point in time of voice data in response to a voice recognition result and a user's selection input for a keyword according to an embodiment of the present disclosure.
Figure 8 illustrates means for providing a conversation record service related to one embodiment of the present disclosure.
Figure 9 illustrates a module for providing a conversation record service related to an embodiment of the present disclosure.
Figure 10 shows logic for providing a conversation record service related to an embodiment of the present disclosure.
Figure 11 shows a circuit for providing a conversation record service related to an embodiment of the present disclosure.
Figure 12 shows a block diagram of a computing device providing a conversation record service related to an embodiment of the present disclosure.
FIG. 13 illustrates an example of synchronizing the start and end points of first and second voice data according to an embodiment of the present disclosure.
Figure 14 shows an example of a process for distinguishing speakers according to an embodiment of the present disclosure.
Figure 15 shows an example of a process for distinguishing speakers through a statistical acoustic model according to an embodiment of the present disclosure.
Figure 16 shows a brief, general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Various embodiments are now described with reference to the drawings. In this specification, various descriptions are presented to provide an understanding of the disclosure. However, it is clear that these embodiments may be practiced without these specific descriptions.

As used herein, the terms “component,” “module,” “system,” and the like refer to a computer-related entity, hardware, firmware, software, a combination of software and hardware, or an implementation of software. For example, a component may be, but is not limited to, a process running on a processor, a processor, an object, a thread of execution, a program, and/or a computer. For example, both an application running on a computing device and the computing device can be a component. One or more components may reside within a processor and/or thread of execution. A component may be localized within one computer. A component may be distributed between two or more computers. Additionally, these components can execute from various computer-readable media having various data structures stored thereon. Components can transmit signals, for example, with one or more data packets (e.g., data and/or signals from one component interacting with other components in a local system, a distributed system, to other systems and over a network such as the Internet). Depending on the data being transmitted, they may communicate through local and/or remote processes.

Additionally, the term “or” is intended to mean an inclusive “or” and not an exclusive “or.” That is, unless otherwise specified or clear from context, “X utilizes A or B” is intended to mean one of the natural implicit substitutions. That is, either X uses A; X uses B; Or, if X uses both A and B, “X uses A or B” can apply to either of these cases. Additionally, the term “and/or” as used herein should be understood to refer to and include all possible combinations of one or more of the related listed items.

Additionally, the terms “comprise” and/or “comprising” should be understood to mean that the corresponding feature and/or element is present. However, the terms “comprise” and/or “comprising” should be understood as not excluding the presence or addition of one or more other features, elements and/or groups thereof. Additionally, unless otherwise specified or the context is clear to indicate a singular form, the singular terms herein and in the claims should generally be construed to mean “one or more.”

Those skilled in the art will additionally recognize that the various illustrative logical blocks, components, modules, circuits, means, logic, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or a combination of both. It must be recognized that it can be implemented with To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or software will depend on the specific application and design constraints imposed on the overall system. A skilled technician can implement the described functionality in a variety of ways for each specific application. However, such implementation decisions should not be construed as causing a departure from the scope of the present disclosure.

Figure 1 shows a conceptual diagram illustrating an overall system for providing a conversation record service related to an embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device 100 and one or more user terminals 10 may transmit and receive data through interconnection through wireless and/or wired. The computing device of the present disclosure may include all types of computing devices capable of calculating data in electronic form, for example, general computing devices such as personal computers and server computers, and mobile terminals (smartphones, It may include computing devices with limited computing capabilities, such as tablets.

According to an embodiment of the present disclosure, the computing device 100 may provide a conversation record service based on voice data including conversation content between a plurality of users. Specifically, the computing device 100 receives one or more voice data from each of one or more user terminals 10, and converts each of the one or more voice data into text through a voice recognition model, thereby You can create a transcript of the conversation content. At this time, the one or more user terminals 10 may include a personal computer (PC), a notebook (note book), a mobile terminal, a smart phone, a tablet PC, etc. /Can include all types of terminals that can access a wireless network.

Additionally, the computing device 100 may provide a text-based conversation log based on one or more voice data received from each of the one or more user terminals 10. In this case, the textual conversation log is a text conversion of voice data of a plurality of users who participated in the conversation. The voice data of the conversation in which a plurality of users participated may be converted into text and listed separately by time and speaker. Accordingly, when conducting a conversation (or meeting) between multiple users, multiple users do not need to write a conversation record through a separate stenographer or write the conversation record themselves, but simply voice data containing the contents of the conversation between multiple users. You can receive text conversation logs containing conversations between multiple users.

Additionally, the computing device 100 may provide a user interface including a conversation log to one or more user terminals. In this case, the user interface provides information about each of one or more users participating in the conversation, a conversation log in which the conversation contents of one or more users are categorized by time and speaker and recorded in the form of a chat window, and the utility of editing, management, and search of the conversation records. Convenience can be provided to a plurality of users using the computing device 100 of the present disclosure, including various UI (User Interfaces)/UX (User Experience) to improve.

In the present disclosure, a detailed description of the process by which the computing device 100 provides a conversation record service to one or more user terminals 10 and the user interface including the conversation record will be described later with reference to the drawings below.

FIG. 2 shows a block diagram of a computing device providing a conversation record service related to an embodiment of the present disclosure.

The components of the computing device 100 that provides the conversation record service shown in FIG. 2 are exemplary. Only some of the components shown in FIG. 2 may constitute the computing device 100 that provides the conversation record service, and additional component(s) in addition to the components shown in FIG. 2 may be included in the computing device 100 that provides the conversation record service. may be included.

As shown in FIG. 2, the computing device 100 that provides a conversation service may include a processor 110, a memory 120, and a network unit 130.

According to one embodiment of the present disclosure, the processor 110 may receive voice data from each of one or more user terminals 10. In this case, the voice data that the processor 110 receives is received from each of one or more user terminals 10, and there may be more than one. For example, when three users are having a conversation, three pieces of voice data can be received from each of three user terminals owned by each of the three users. That is, the processor 110 may receive one or more voice data from one or more user terminals 10 owned by each of a plurality of users, rather than receiving one voice data through one user terminal. The specific description of the number of users conducting the above conversation and the number of user terminals corresponding thereto is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may perform preprocessing on one or more voice data received from each of the one or more user terminals 10. The processor 110 may perform preprocessing to increase the recognition rate of voice data. At this time, the preprocessing that the processor 110 performs on each of the one or more voice data may include preprocessing for noise removal and preprocessing for separating speakers based on the speech size in the speech section.

According to an embodiment of the present disclosure, the processor 110 may remove noise from each of one or more voice data received from each of one or more user terminals 10. Removing noise from each of the one or more voice data may be to remove noise included in the one or more voice data. More specifically, the processor 110 may standardize the size of the signal included in each of the one or more voice data based on a comparison between the size of the signal included in each of the one or more voice data and the size of a predetermined reference signal. If the size of the signal included in each voice data received from one or more user terminals 10 is less than a predetermined reference signal, the processor 110 adjusts the size of the corresponding signal to be large, and then receives the signal from the one or more user terminals 10. If the size of the signal included in each received voice data is greater than or equal to a predetermined reference signal, Audio Signal Compression Normalization can be performed to adjust the size of the signal to be smaller (i.e., to prevent clipping). The noise removal operation described above is only an example and the present disclosure is not limited thereto.

Additionally, the processor 110 may analyze the waveform of the voice included in each piece of one or more voice data and remove sounds other than speech. More specifically, the processor 110 may analyze the frequencies of voice data and remove at least one specific frequency. Each of the one or more voice data received by the processor 110 may include white noise with a uniform frequency, random frequency noise with an irregular frequency, or various noises generated depending on the recording device or signal processing process. That is, the processor 110 may perform preprocessing on noise with various frequencies included in each of one or more voice data. Specifically, the processor 110 classifies each of the one or more voice data using a machine learning algorithm such as SVM (Supporting Vector Machine) to determine the type of noise contained in each of the noises containing different frequencies. Noise can be removed through a corresponding noise removal algorithm. The noise removal algorithm described above is only an example and the present disclosure is not limited thereto.

Additionally, the processor 110 may identify one or more speech sections for each of one or more voice data. Processor 110 may identify one or more utterances included in voice data. Specifically, the processor 110 may identify the user's speech section based on one or more silent sections present in voice data. For example, the processor 110 may identify the period between the first and second silent sections included in the voice data as the user's first speech section, and between the second and third silent sections may be identified as the user's first speech section. It can be identified as the second ignition section. That is, in voice data containing the conversation content of one or more users, identify the point in time when the voice (i.e., speech sound) of the one or more users is not recognized (for example, the point in time when only sounds below a certain level are detected, etc.) , One or more user's speech sections can be identified based on the identified time point (i.e., silent section). The detailed description of the user's speech section described above is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may identify the voice of one or more users in each of one or more speech sections included in voice data. Additionally, the processor 110 may determine an utterance whose voice level is greater than a predetermined level among the voices of each of the identified one or more users as a specific speaker corresponding to the voice data. More specifically, if the volume of a specific user's voice among the voices of one or more users included in the voice data is greater than or equal to a predetermined volume, the processor 110 may determine a specific speaker corresponding to the voice data. For example, when voice data including the voices of two users (user A and user B) is received from the first user terminal, the processor 110 determines that the volume of user A's voice is greater than a predetermined volume and , User A may be determined as a specific speaker corresponding to voice data received from the first user terminal. That is, in the present disclosure, the user terminal may be a terminal deployed to each user in order to collect the voice of each user, so the processor 110 may detect an utterance of a certain size or more in the received voice data by the user for whom the voice data was collected. It can be determined that this is an utterance from the user of the terminal. Additionally, the processor 110 may determine a specific speaker corresponding to the voice data based on a comparison of the volume of each user's voice included in the voice data. For example, when voice data including the voices of two users (user A and user B) is received from the first user terminal, the processor 110 selects user B's voice among the voices of the two users included in the voice data. By determining that the volume of voice is greater than that of user A, user B can be determined as a specific speaker corresponding to voice data received from the first user terminal. The above-described specific description of one or more users and the volume of each user's voice is merely an example, and the present disclosure is not limited thereto.

Additionally, the processor 110 may remove voices other than those of a specific speaker from voice data received from the user terminal 10. More specifically, when the processor 110 determines a specific speaker corresponding to voice data, the processor 110 may remove voices of other users other than the voice of the specific speaker from the voice data. For example, if voice data includes the voices of user A and user B, and the user A is determined to be a specific speaker corresponding to the voice data, the processor 110 may remove the voice of user B from the voice data. You can. That is, the processor 110 can more clearly separate the voice of a specific speaker (eg, user A) by removing the voice of another user (eg, user B) from the voice data. The specific description of the user corresponding to the above-described voice data is only an example, and the present disclosure is not limited thereto. Accordingly, when creating a conversation record based on voice data in the future, the separation of each speaker can be more accurate, and a conversation record with higher accuracy can be generated.

In addition, the processor 110 determines a specific speaker corresponding to each of one or more voice data, and when voices other than the voice of the determined specific speaker are removed from each voice data, the processor 110 identifies the specific speaker in one or more user terminals. It can be saved by matching each piece of information. For example, user A, determined as a specific speaker corresponding to the first voice data, may be matched with identification information of user terminal A, and user B determined as a specific speaker corresponding to the second voice data may be matched with identification information of user terminal B. You can save it. Accordingly, by determining the speaker corresponding to each of the one or more voice data received from each of the one or more user terminals and matching each speaker to each user terminal, it is possible to more accurately separate each of the plurality of speakers who conducted the conversation. . The detailed description of the above-described voice data and the corresponding user terminal is only an example, and the present disclosure is not limited thereto.

Additionally, the processor 110 may divide the voice data into one or more voice data sub-data based on the speech section. For example, when first voice data is received through user terminal A, the processor 110 divides the first voice data into one or more voice data subdata based on one or more speech sections identified in the first voice data. It can be divided. For a more specific example, when there are five speech sections identified in the first voice data, the processor 110 divides the first voice data into five voice data sub-data corresponding to each of the five identified speech sections. can do. The specific description of the number of one or more voice data subdata of the voice data described above is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may assign a unified viewpoint to each of one or more voice data received from each of one or more user terminals 10.

Specifically, the processor 110 may tag each of one or more voice data received from each of one or more user terminals 10 with global time information. In this case, the global time information may be time information applied to each of a plurality of voice data received from each of a plurality of user terminals. More specifically, global time information may be time information that serves as a standard provided to clarify the sequential relationship between each of a plurality of voice data. For example, global time information may be time information determined based on time information of a computing device. The method of determining global time information is only an example and the present disclosure is not limited thereto. Local time information may be time information applied to each voice data. That is, the local time information may be information about the elapsed time from the start of voice data (for example, in the case of voice data for 8 minutes, the 32-second point, the 40-second point, etc.). The foregoing description of local time information is merely an example, and the present disclosure is not limited thereto. That is, when receiving a plurality of voice data from each of the plurality of user terminals 10, the processor 110 matches and stores global time information to each of the plurality of voice data, according to the speech section in each of the plurality of voice data. The sequential relationship of one or more divided voice data subdata can be clearly determined. For example, when two pieces of voice data (first voice data and second voice data) are received through two user terminals (user terminal A and user terminal B), each utterance included in the two voice data It may be difficult to determine the precedence relationship using only the local time information of each voice data. That is, if the recording start time of each voice data is different, it may be difficult for the processor 110 to clearly determine the sequential relationship of each utterance included in the plurality of voice data using only local time information. For example, the first voice data received from user terminal A may be voice data for the period from 11:10 to 11:20 (i.e., voice data for 10 minutes), and the first voice data received through user terminal B may be voice data for 11:10 to 11:20. The second voice data may be voice data for 11:12 to 11:20 (i.e., voice data for 8 minutes), so that the start time of each voice data is different from each other, and the same local time information is used. Utterances from different viewpoints may actually be utterances from different viewpoints. Accordingly, since the sequential relationship of the voice data sub-data included in each voice data is not clearly identified, which may cause problems in generating conversation records through each voice data, the processor 110 may use the first voice data and the second voice data. 2 By matching the global time information to each voice data (i.e., displaying the current time corresponding to each speech section) and unifying the timing of each data, each of the first voice data and the second voice data is divided according to the speech section. The sequential relationship of one or more voice data subdata can be clearly distinguished. The above-described specific description of the number of user terminals, the number of voice data, and the timing of each voice data is only an example, and the present disclosure is not limited thereto.

Additionally, the processor 110 may determine the sequential relationship of utterances included in the voice data based on the elapsed time of each of the plurality of voice data. The processor 110 may determine a reference point applied to all of the plurality of voice data, and determine the precedence relationship of the utterances included in the plurality of voice data based on the elapsed time from the reference point. For example, the processor 110 determines the start time of the voice data that started recording first among the plurality of voice data as the reference point, and determines the elapsed time based on the reference point in other voice data, thereby determining the plurality of voices. The sequential relationship of multiple utterances included in the data can be determined. Specifically, the processor 110 records the time when each of the one or more voice data received from each of the one or more user terminals 10 is first generated, and records each voice data based on the first time recorded in each of the one or more voice data. By identifying the progress time of , a unified viewpoint can be provided to determine the sequential relationship of one or more voice data sub-data included in each of one or more voice data.

More specifically, the processor 110 adds a silence section to the voice data that started recording later by the difference between the recording start time of the voice data that started recording first and the recording start time of the voice data that started recording later. It can be added before starting. That is, the processor 110 matches the reference point applied to the plurality of voice data (in this example, the recording start time of the voice data where recording first started, etc.) and calculates the difference between each voice data from the reference point to each voice data. You can determine the sequence of utterances by reflecting them. For example, if recording of the second voice data begins 5 seconds after the start of recording of the first voice data, the processor 110 records the recording at the reference time (e.g., the time of the computing device) for the generation of the voice data. This can be determined based on this, and by providing a silence period of 5 seconds before the start of recording the second voice data, the progress time of the first voice data and the second voice data can be made to match the actual speech time. The foregoing description is merely an example, and the present disclosure is not limited thereto.

In addition, the processor 110 adds the difference between the recording start time of the voice data that started recording first and the recording start time of the voice data that started recording later, to the progress time of the voice data that started recording later, and determines the sequential relationship of the utterance. can be judged. For example, if recording of the second voice data starts 5 seconds after the start of recording of the first voice data, the utterance located 10 seconds from the start of recording of the second voice data is 15 seconds from the start of recording of the first voice data. It may be an utterance located at the beginning. Accordingly, the processor 110 can determine the precedence relationship of each utterance included in each voice data by adding a time equal to the difference between the first voice data and the start time to the second voice data to the progress time. The foregoing description is merely an example, and the present disclosure is not limited thereto.

As described above, the processor 110 can determine the precedence relationship of a plurality of utterances included in the voice data by matching and storing global time information to voice data or matching and storing information about elapsed time from a reference point. there is.

The above-described specific description of the number of user terminals, the number of voice data, and the timing of each voice data is only an example, and the present disclosure is not limited thereto.

In the description of the voice data described below, the start time and end time are based on at least one of global time information and reference time information of the voice data, and the sequential relationship between each of the one or more voice data sub-data included in each of the one or more voice data. It is based on a unified viewpoint assigned to each of one or more voice data to distinguish.

According to an embodiment of the present disclosure, the processor 110 may identify the start point and end point of each of one or more voice data subdata. Specifically, the processor 110 may identify the start point and end point of each of one or more voice data sub-data divided from voice data based on the speech section included in the voice data. For example, when the voice data is divided into two voice data sub-data (voice data first sub-data and voice data second sub-data) corresponding to two speech sections, the processor 110 divides the voice data into the first voice data sub-data. The start and end points of the data can be identified as 3 seconds and 32 seconds, respectively, and the start and end points of the second sub-data of the voice data can be identified as 1 minute 10 seconds and 1 minute 52 seconds, respectively. The detailed description of the start and end points of the voice data sub-data described above is only an example, and the present disclosure is not limited thereto.

Additionally, the processor 110 may synchronize the start and end times of the first voice data and the second voice data through network packet exchange between the first user terminal and the second user terminal. Specifically, the processor 110 may measure and correct errors in the start and end times of the first user terminal and the second user terminal through network packet exchange between the first user terminal and the second user terminal. The times of the first user terminal and the second user terminal can be basically synchronized with the NETWORK TIME PROTOCOL server. For more accurate time synchronization of the first user terminal and the second user terminal, the start time and end time information of each terminal is processed through a statistical analysis filter to synchronize the times of the first user terminal and the second user terminal. You can. The statistical analysis filter can calculate the time difference on the network based on the comparison result of the start time and end time. The statistical analysis filter can perform the time synchronization described above based on the calculated time difference on the network.

FIG. 13 illustrates an example of synchronizing the start and end points of first voice data and second voice data according to an embodiment of the present disclosure.

In addition, according to an embodiment of the present disclosure, the processor 110 determines the start and end points of the first voice data and the second voice data based on the similarity between the first voice data sub-data and the second voice data sub-data. You can synchronize. Specifically, the time delay of each voice data subdata based on the first voice data 1310 and the second voice data 1320 can be measured and corrected. Search for similar signals (1311, 1321) between the first voice data sub-data and the second voice data sub-data, and measure the time delay based on the similar signals to determine the start and end points of the first voice data and the second voice data. can be synchronized. At this time, various feature extraction algorithms such as FFT (FAST FOURIER TRANSFORM) and MFCC (MEL-FREQUENCY CEPSTRAL COEFFICIENTS) can be used to analyze voice signals.

According to an embodiment of the present disclosure, the processor 110 may generate a conversation log by converting integrated voice data into voice recognition results through a voice recognition model. In this case, integrated voice data may be generated based on the first voice data and second voice data received from each user terminal. Additionally, the processor 110 divides the first voice data into one or more first voice data subdata based on the speech section, and divides the second voice data subdata into one or more second voice data subdata based on the speech section. It can be divided into For example, when there are three speech sections identified in the first voice data, the processor 110 divides the first voice data into first voice data first sub-data and first voice data second based on the three speech sections. It can be divided into sub data, first voice data, and third sub data, respectively. For another example, when there are two speech sections identified in the second voice data, the processor 110 divides the second voice data into second voice data first sub-data and second voice data first based on the two speech sections. Each can be divided into 2 sub data. The description of specific values of speech sections included in the above-described first voice data and second voice data is only an example, and the present disclosure is not limited thereto.

Figure 14 shows an example of separating speakers based on voice signal size according to an embodiment of the present disclosure.

*According to an embodiment of the present disclosure, the processor 110 may generate integrated voice data based on the first voice data and the second voice data. Specifically, the processor 110 may distinguish speakers based on the audio signal size of each of the first voice data and the second voice data and generate integrated voice data based on the separated speaker information. The processor 110 may calculate the voice signal strength for each frame of the first voice data and the second voice data and distinguish the speaker by comparing the voice signal strength of the first voice data frame and the second voice data frame. For example, the processor 110 may compare the voice signal sizes of each of the first voice data frame and the second voice data frame 1421 in the period of 0 seconds to 20 seconds. The processor 110 may distinguish the second voice data frame 1421, which has a larger voice signal size, as the speaker. The processor 110 may determine the speaker in the 0 to 25 second period as the second user. The processor 110 may compare the voice signal size of each of the first voice data frame 1411 and the second voice data frame in the period of 25 seconds to 63 seconds. The processor 110 may classify the first voice data frame 1411, which has a larger voice signal, into speakers and determine the speaker in the 25 to 63 second period as the first user. The processor 110 compares the voice signal sizes of each of the first voice data frame and the second voice data frame 1422 in the period of 63 seconds to 80 seconds and selects the second voice data frame 1422, which has a larger voice signal size, at number 63. It can be classified into speakers ranging from 1 to 80 seconds. The processor 110 may determine the speaker from 63 seconds to 80 seconds as the second user. The processor 110 may generate integrated voice data 1430 by distinguishing speakers for each section. In the present disclosure, a voice data frame may be a predetermined time interval of voice data for determining a speaker in the voice data. The description of generating integrated voice data by distinguishing speakers based on the above-described voice data is only an example and the present disclosure is not limited thereto.

Figure 15 shows an example of a process for distinguishing speakers through a statistical acoustic model according to an embodiment of the present disclosure.

According to an embodiment of the present disclosure, the processor 110 may distinguish speakers through a statistical acoustic model based on the first voice data and the second voice data.

More specifically, the processor 110 may distinguish the speaker of the current frame based on speaker information of adjacent frames. For example, when distinguishing speaker information of the current viewpoint frame 1520, if the speaker of the previous viewpoint frame 1510 is determined to be speaker A, the speaker of the current viewpoint frame 1520 is the speaker of the previous viewpoint frame 1510. The probability that it will be speaker A may be higher. Therefore, by reflecting the speaker information of the previous viewpoint frame 1510 in the statistical acoustic model, the speaker of the current viewpoint frame 1520 can be determined to generate integrated voice data. In addition, when the speaker information of the frame 1530 after the current point is determined, the processor 110 reflects the speaker information of the frame 1530 after the current point in the statistical acoustic model to identify the speaker in the frame 1520 of the current point. By deciding, you can create integrated voice data. Additionally, the processor 110 may determine the speaker of the current frame by comparing the speaker in the adjacent frame with the speaker determined based on the size of the voice signal in the current frame. For example, the processor 110 determines that the speaker in the previous frame 1510 is A, the speaker determined based on the size of the voice signal in the current frame 1520 is B, and the speaker in the frame 1530 after the current frame is B. If the speaker is A, based on statistical information, the speaker at the current frame 1520 is A, and B's large voice signal may be treated as noise. The description of distinguishing speakers based on adjacent frames described above is only an example and the present disclosure is not limited thereto.

Additionally, the processor 110 may distinguish speakers based on the ratio of each speaker's speech time between the first voice data and the second voice data. More specifically, when the processor 110 identifies an overlapping section based on the start and end time of each of one or more first voice data subdata and one or more second voice data subdata, the processor 110 uses the overlapping section based on the speaker of the frame excluding the overlapping section. Thus, speakers can be distinguished by calculating the speech time ratio for each speaker. The processor 110 may calculate the ratio of the speech time of the first voice data excluding the overlapping section to the speech time of the second voice data excluding the overlapping section and determine voice data with a large speech time ratio. The processor 110 may determine that the speaker of the voice data determined to have a large speech time ratio has a high probability of being the speaker of the overlapping section frame. The processor 110 may distinguish speakers in the overlapping section based on the speech time ratio of one or more voice data calculated through a statistical acoustic model. For example, the processor 110 may have a ratio of speech time other than the overlapping section of the first voice data to speech time other than the overlapping section of the second voice data of 7 to 3. The processor 110 may determine the speaker of the overlapping section frame as the speaker of the first voice data by reflecting the corresponding ratio using a statistical acoustic model. The example of distinguishing speakers in the above-described overlapping section is only an example and is not limited to the present disclosure.

Additionally, the processor 110 may distinguish speakers based on the ratio of speech time for each speaker to the previous frame of the overlapping section frame. More specifically, if the identified overlap section is a frame after a predetermined time ratio, the processor 110 can distinguish speakers in the overlap section frame through a statistical acoustic model based on the rate of speech time for each speaker up to the point before the overlap section. You can. For example, the processor 110 may set the start point of the overlapping section frame at 14 minutes and 10 seconds in a conversation between the first user terminal and the second user terminal for 30 minutes. Assuming that the predetermined time ratio is 20 percent, the overlapping interval frame may be a point in time after 6 minutes corresponding to the predetermined time ratio. The speech time ratio of each voice data of the first user terminal and the second user terminal up to 14 minutes and 10 seconds before the overlap section frame may be 7 to 3. The processor 110 may determine the speaker of the first user terminal as the speaker of the overlapping section frame through a statistical acoustic model based on the speech rate up to the point before the overlapping section frame. The detailed description of the speaker determination method described above is only an example and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may generate a conversation record using integrated voice data using a voice recognition model. The processor 110 may generate integrated voice data based on first voice data received from the first user terminal and second voice data received from the second user terminal. The processor 110 may generate a conversation record using the generated integrated voice data using a voice recognition model. Specifically, the processor 110 may mix down the first voice data and the second voice data to generate integrated voice data. When mixing down the first voice data and the second voice data, the processor 110 may determine the speaker of the frame of the integrated voice data based on speaker information for each frame of the first voice data and the second voice data. . Additionally, the processor 110 can generate a conversation log in text form by calculating integrated voice data through a voice recognition model. In this case, the voice recognition model may be a model that converts the voice language spoken by the user into text data. A speech recognition model may include one or more network functions pre-trained through training data.

According to an embodiment of the present disclosure, the processor 110 may use each of one or more voice data sub-data as input to a voice recognition model and output one or more voice recognition sub-results corresponding to each of the one or more voice data sub-data. . In this case, the voice recognition model is a model that converts the voice language spoken by the user into text data, and may include one or more network functions pre-trained through training data. In addition, the processor 110 may determine the order of each of the one or more voice recognition sub-results in the conversation log based on the start and end points identified in each voice data sub-data that is the basis of the one or more voice recognition sub-results. . For a specific example, the processor 110 may input first voice data sub-data and second voice data sub-data into a voice recognition model and output a first voice recognition sub-result and a second voice recognition sub-result. The processor 110 identifies the start and end times of the first sub-data of voice data based on the output of the voice recognition first sub-result as 3 seconds and 32 seconds, respectively, and the voice data based on the output of the voice recognition second sub-result. The start and end times of the second sub data can be identified as 1 minute 10 seconds and 1 minute 52 seconds, respectively. The processor 110 may determine the order of the first voice recognition sub-result and the second voice recognition sub-result in the conversation record based on the determined viewpoint information. That is, the processor 110 identifies a start point and an end point of voice data sub-data corresponding to each of the one or more voice recognition sub-results, and at least one of the identified start points and end points. Based on this, the order in which each of one or more voice recognition sub-results is displayed in the conversation log can be determined. That is, in the conversation log, one or more voice recognition sub-results may be arranged in order according to the start time and end time. For example, if the start time of the first sub-result of speech recognition is earlier than the start time of the second sub-result of speech recognition, the processor 110 displays the first sub-result of speech recognition before the second sub-result of speech recognition in the conversation log. You can decide the order as much as possible. For another example, if the end time of the second voice recognition sub-result is earlier than the end time of the first voice recognition result, the processor 110 displays the second voice recognition sub-result before the first voice recognition sub-result in the conversation log. You can also decide the order if possible. The specific description of the start and end points corresponding to the above-described first sub-result of voice recognition and second sub-result of voice recognition is only an example, and the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the processor 110 may generate a conversation record by converting each of one or more voice data into one or more voice recognition results through a voice recognition model. In this case, the one or more voice data may include first voice data and second voice data received from each user terminal. Additionally, the processor 110 divides the first voice data into one or more first voice data subdata based on the speech section, and divides the second voice data subdata into one or more second voice data subdata based on the speech section. It can be divided into For example, when there are three speech sections identified in the first voice data, the processor 110 divides the first voice data into first voice data first sub-data and first voice data second based on the three speech sections. It can be divided into sub data, first voice data, and third sub data, respectively. For another example, when there are two speech sections identified in the second voice data, the processor 110 divides the second voice data into second voice data first sub-data and second voice data first based on the two speech sections. Each can be divided into 2 sub data. The description of specific values of speech sections included in the above-described first voice data and second voice data is only an example, and the present disclosure is not limited thereto.

Additionally, the processor 110 may identify the start time and end time of each of one or more first voice data subdata and one or more second voice data subdata. For example, the first voice data is divided into first voice data first sub-data, first voice data second sub-data, and first voice data third sub-data based on three speech sections, and the second voice data When is divided into second voice data first sub-data and second voice data second sub-data based on two speech sections, the processor 110 sets the start time of the five voice data sub-data to 10 seconds each ( 1st voice data 1st sub data), 21 seconds (1st voice data 2nd sub data), 32 seconds (1st voice data 3rd sub data), 17 seconds (2nd voice data 1st sub data) and 25 seconds. It can be identified by seconds (second voice data, second sub data). The detailed description of the start point of the voice data sub-data described above is only an example, and the present disclosure is not limited thereto.

In addition, the processor 110 uses one or more first voice data sub-data and one or more second voice data sub-data as input to the voice recognition model, and generates one or more first voice recognition sub-results corresponding to each sub-data and one or more first voice recognition sub-data and one or more first voice recognition sub-data corresponding to each sub-data. The above second voice recognition sub-results can be output. For example, when first voice data first sub-data is input to the voice recognition model, the first voice recognition first sub-result may be output, and when second voice data first sub-data is input, the second voice recognition model may output the first sub-data. The voice recognition second sub result can be output.

In addition, the processor 110 may perform one or more first voice recognition sub-results in the conversation record based on the start and end points identified in each voice data sub-data that is the basis of one or more first voice recognition sub-results and one or more second voice recognition sub-results. The order in which each of the first voice recognition sub-result and one or more second voice recognition sub-results are displayed may be determined. For a specific example, in the above example, the first sub-result of voice recognition, the second sub-result of first voice recognition, the third sub-result of first voice recognition, the first sub-result of second voice recognition, and the second sub-result of voice recognition. Each of the two sub results is based on the first voice data first subdata, the first voice data second subdata, the first voice data third subdata, the second voice data first subdata, and the second voice data subdata. can be created. The processor 110 sets the start time of the five voice data sub-data, which are the basis for generating the five voice recognition sub-results, to 10 seconds (first voice data first sub-data) and 21 seconds (first voice data first sub-data), respectively. 2 subdata), 32 seconds (first voice data, third subdata), 17 seconds (second voice data, first subdata), and 25 seconds (second voice data, second subdata). Additionally, the processor 110 may determine the order in which the five voice recognition sub-results are displayed in the conversation log based on the identified start point. That is, the conversation log contains the first sub-result of voice recognition (10 seconds), the first sub-result of second voice recognition (17 seconds), the second sub-result of first voice recognition (21 seconds), and the second sub-result of second voice recognition. It may be arranged in the following order: result (25 seconds), first voice recognition third sub-result (32 seconds).

Additionally, the processor 110 may generate a conversation record based on the determined order of each of the one or more first voice recognition sub-results and one or more second voice recognition sub-results and the identification information of each of the one or more user terminals 10. For example, the processor 110 may identify the speaker based on the identification information of the first user terminal matched to each of one or more first voice recognition sub-results. Additionally, the processor 110 may identify the speaker based on the identification information of the second user terminal matched to each of one or more second voice recognition sub-results. Accordingly, each voice recognition sub-result can be recorded in the conversation log in the form of a chat window based on the identification information of each user terminal. That is, the processor 110 may generate a conversation log in which each speaker (eg, a first user using a first user terminal and a second user using a second user terminal) is separated.

According to an embodiment of the present disclosure, the processor 110 creates overlapping sections of utterances of a plurality of users based on the start and end points of each of one or more first voice data subdata and one or more second voice data subdata. can be identified. At this time, the first voice data subdata is included in the first voice data received through the first user terminal, and the second voice data subdata is included in the second voice data received through the second user terminal, Each may be generated based on voice data received from different user terminals. Specifically, the processor 110 divides the first voice data and the second voice data received from different terminals into one or more first voice data sub-data and one or more second voice data sub-data divided based on each speech section. It is possible to identify whether the first voice data sub-data and the second voice data sub-data overlap in time through the start and end points of each data. That is, the processor 110 can identify a section in which utterances of a plurality of utterances from a plurality of users overlap in each voice data. For example, the start and end of each of the first voice data first subdata, the first voice data second subdata, and the first voice data third subdata included in the first voice data received through the first user terminal. The time points are 10 to 15 seconds, 21 to 28 seconds, and 32 to 35 seconds, respectively, and are included in the second voice data received through the second user terminal. Second voice data first subdata, second voice data second subdata When each start and end time is 17 to 20 seconds and 25 to 31 seconds, respectively, the processor 110 identifies that the first voice data second sub data and the second voice data second sub data overlap, and the second voice data second sub data overlaps from 25 seconds to 28 seconds. Up to second can be identified as an overlapping section. The detailed description of the start and end points of each of the above-described first and second voice recognition sub-results is merely an example, and the present disclosure is not limited thereto.

Additionally, when an overlapping section is identified, the processor 110 may determine the order of voice recognition sub-results corresponding to the overlapping section. In the following example, the first voice recognition sub-result is a voice recognition sub-result for one of the sub-data of the first voice data included in the first voice data received from the first user terminal, and the second voice recognition sub-result is the first voice recognition sub-result. 2 It may be a voice recognition sub-result for one of the sub-data of the second voice data included in the second voice data received from the user terminal. The processor 110 may identify that the first voice recognition sub-result and the second voice recognition sub-result are utterances of overlapping times based on time information of the first voice data sub-data and the second voice data sub-data. In this case, the processor 110 may determine the order of the first voice recognition sub-result and the second voice recognition sub-result in the conversation log according to a predetermined standard. For example, if the start time of the first voice data sub-data that is the basis of the first voice recognition sub-result is earlier than the start time of the second voice data sub-data, the processor 110 may select the first voice recognition sub-result in the conversation log. can be placed before the second voice recognition sub-result.

Additionally, the processor 110 may divide at least one of the voice recognition sub-results corresponding to the overlapping section. Specifically, the processor 110 is based on the start and end points of each of the first voice data subset and the second voice data subset, which are based on the output of each of the first voice recognition sub-result and the second voice recognition sub-result. Thus, when an overlap in the utterance is identified, it may be determined to split at least one of the first voice recognition sub-result and the second voice recognition sub-result. More specifically, if the second voice data sub-data starts between the start and end times of the first voice data sub-data that is the basis of the first voice recognition sub-result, the processor 110 generates the first voice recognition sub-result. It can be divided based on the start time of the second voice data subdata. For example, if the first voice data subdata is data including a 10-second utterance, it can be assumed that the utterance of the second voice data subdata starts 5 seconds after the start of the utterance of the first voice data subdata. there is. In this case, the processor 110 may divide the first voice recognition sub-result, which is a voice recognition result for a 10-second utterance of the first voice data subdata, into segments based on 5 seconds and place them on the conversation log. That is, in this example, the processor 110 selects the first text in the interval between 0 and 5 seconds of the first voice recognition sub-result, the text of the second voice recognition sub-result, and 5 seconds of the first voice recognition sub-result in the conversation log. The second text in the section between 10 seconds can be arranged in order. The description regarding the time and division of voice data sub-data described above is only an example and the present disclosure is not limited thereto.

Additionally, the processor 110 may divide each voice recognition sub-result based on meaning when an overlapping section is identified. The processor 110 may determine a division point of the voice recognition sub-result based on natural language processing of the voice recognition sub-result. In addition, the processor 110 analyzes the combination probability of words included in each of the first and second voice recognition sub-results corresponding to the overlapping section based on a statistical language model, thereby The end point in the sentence for the voice recognition sub result can be determined. At this time, the statistical language model may be a model that can calculate the probability of a sentence, or calculate the probability of the next word appearing when previous words are input, how appropriate the combination of words is, or how long the sentence is. It may be a model that can determine whether it is suitable. Additionally, the processor 110 may determine whether to divide the voice recognition sub-results based on the end point determined for each voice recognition sub-result. For example, if the start and end points of the first voice data sub-data are identified as 5 seconds and 20 seconds, respectively, and the start and end points of the second voice data sub-data are identified as 12 seconds and 18 seconds, respectively (i.e. , when an overlapping section of 6 seconds of speech between 12 and 18 seconds is identified), the processor 110 uses the first voice recognition sub-result as an input to the statistical language model and By outputting the degree of suitability for the words and multiple combinations of each word, the ending point in the sentence for the first voice recognition sub-result can be determined. Additionally, when an end point is determined in the first voice recognition server result, the processor 110 may determine to divide the first voice recognition sub-result. For a more specific example, an utterance of voice data containing the above-mentioned overlapping section would be, "I came to see you because I wanted to consult about copyright (copyright-related?). Yes, that's right." Assume: In the above-described assumption, the utterance within the parentheses may be from the second speaker (i.e., the utterance recognized from the second speech data), and the utterance outside the parentheses may be from the first speaker (i.e., the utterance recognized from the first speech data). In this case, the statistical language model said that each of the words included in the first speech recognition sub-result was 'copyright', 'related', 'consultation', 'receiving', 'want', 'find', 'see you', and ' ', 'Yes', and 'That's correct' can be recognized, and the suitability of the combination of the words 'It has been done' and 'Yes' can be output at a lower level. Accordingly, the processor 110 can set 'It has been' at the end of the sentence. It can be judged by point. Additionally, the processor 110 may determine to divide the first voice recognition sub-result based on the ending point in the sentence determined in the first voice recognition sub-result. The above-described specific description of the start and end points of each of the first voice data subdata and the second voice data subdata is only an example, and the present disclosure is not limited thereto.

Additionally, when the processor 110 determines division of at least one of the first voice recognition sub-result and the second voice recognition sub-result, the processor 110 may divide the voice recognition sub-result corresponding to the overlapping section. For a specific example, the start and end points of the first voice data sub-data are identified as 5 seconds and 20 seconds, respectively, and the start and end points of the second voice data sub-data are identified as 12 seconds and 18 seconds, respectively, creating an overlapping section. The existence was identified, and each of the words included in the first voice recognition sub-result were 'copyright', 'related', 'consultation', 'receiving', 'want', 'find', 'see', and ' ', 'Yes', 'That's right.', and if each of the words included in the second voice recognition sub-result is 'Copyright' and 'Related?', the processor 110 uses a combination of words output from the statistical language model. Based on the degree of suitability, 'was' of the first voice recognition sub-result may be determined as the end point in the sentence for the first voice recognition sub-result. In addition, the processor 110 may divide the first voice recognition sub-result into 'I came to see you because I wanted to receive copyright-related consultation' and 'Yes, that's right' based on the end point determined in the first voice recognition sub-result. . In other words, the processor 110 identifies that a second voice recognition sub-result output based on another user's utterance exists between the start and end points corresponding to the first voice recognition sub-result (i.e., an overlapping section). identification), and by identifying the meaning of words included in the voice recognition sub-results in the overlapping section, the voice recognition sub-results can be divided so that the precedence relationship of each voice recognition sub-result is clearly displayed. That is, the processor 110 categorizes the first voice recognition sub-result as “copyright-related” so that the words included in the second voice recognition sub-result (i.e., “Copyright-related?”) are placed between the first voice recognition sub-results. It can be divided into “I came to see you because I wanted to get counseling” and “Yes, that’s right.” The specific description of the start and end points of each of the above-described first voice data sub-data and second voice data sub-data and the specific description of the first voice recognition sub-result and the second voice recognition sub-result are only examples. The disclosure is not limited thereto.

As described above, the present disclosure is to receive two voice data from each user terminal, determine the order of one or more voice recognition results output from the two voice data, and determine the order of the two user terminals that received the two voice data. A conversation record can be generated based on identification information, but it will be apparent to those skilled in the art that a conversation record can also be generated by receiving two or more pieces of voice data through the processes of the present disclosure.

According to one embodiment of the present disclosure, the processor 110 may determine a keyword corresponding to the conversation log. Specifically, the processor 110 may determine one or more keywords for the conversation record based on at least one of a semantic analysis result of one or more voice recognition results, the number of repetitions for each word included in the one or more voice recognition results, and a predetermined conversation topic. there is.

The processor 110 may determine one or more keywords for the conversation log based on a semantic analysis result of one or more voice recognition results. More specifically, the processor 110 may determine one or more keywords based on the importance of the meaning of one or more words included in each of one or more voice recognition results. For example, if each word included in the voice recognition result is 'copyright', 'related', 'consultation', 'receiving', 'want', 'find', 'meet', and 'was', the processor (110) determines that the meaning of 'copyright' or 'consultation' is relatively high in importance, so 'copyright' or 'consultation' can be decided as the keyword. The specific description of one or more words included in the above-described voice recognition result and the importance of each of the one or more words is merely an example, and the present disclosure is not limited thereto.

Additionally, the processor 110 may determine one or more keywords for the conversation log based on the number of repetitions for each word included in each of one or more voice recognition results. Specifically, the processor 110 may identify the number of repetitions (i.e., number of repetitions) of each of one or more words included in the voice recognition result, and determine a word with a high number of repetitions as a keyword for the conversation log.

Additionally, processor 110 may determine keywords based on predetermined conversation topics. More specifically, the processor 110 may determine the topic of conversation from semantic analysis of the voice recognition result, or may receive the topic of conversation from the user terminal 10, and determine keywords in response to the received topic of conversation. there is. For example, when receiving 'copyright' as a conversation topic from the user terminal 10, the processor 110 uses keywords such as 'infringement', 'complaint', and 'crime' related to the topic (i.e., copyright). You can decide. In this case, one or more keywords may be pre-stored in the memory 120 to correspond to each of the plurality of conversation topics. The specific description of the above-described conversation topics and keywords corresponding to the conversation topics are only examples, and the present disclosure is not limited thereto.

According to one embodiment of the present disclosure, the processor 110 may identify the location of one or more keywords in one or more voice recognition results. Specifically, the processor 110 may identify at least one of a location in the conversation record and a location in the voice data of one or more keywords among one or more voice recognition results included in the conversation record. Additionally, the processor 110 may identify the location of each of one or more keywords in one or more voice recognition results included in the conversation log and reproduce voice data sub-data corresponding to each of the one or more keywords. In detail, the processor 110 uses each of one or more voice data as input to a voice recognition model, identifies keywords in the conversation log generated through one or more voice recognition results output, and uses the identified keywords in response to the identified keywords. Voice data subdata containing keywords can be played. That is, the processor 110 can identify only the part of the voice data received from the user terminal 10 that corresponds to a specific keyword from the conversation log and provide it as a voice recognition result, and the voice data sub-subset corresponding to the identified voice recognition result. Data sections can be regenerated, providing convenience to users. In other words, the user can be provided with only certain important parts of the conversation log, and can be provided with a playback screen that allows the user to play the section corresponding to that specific part in the recorded data (i.e. voice data), so they can read the entire conversation log or record it. Instead of having to listen to all of the data, you can search only certain important parts (i.e. keywords) as needed, providing convenience in understanding the gist.

According to an embodiment of the present disclosure, the processor 110 may provide a user interface including a conversation log generated based on voice data received from each of the one or more user terminals 10 to one or more user terminals.

A detailed description of the user interface provided by the processor 110 to one or more user terminals 10 will be described later with reference to FIGS. 4 to 7.

Figure 4 shows an example of a user interface including a user list display screen and a conversation list display screen according to an embodiment of the present disclosure.

The user interface may be displayed on the display means of the user terminal 10.

As shown in (a) of FIG. 4, the user interface generated by the processor 110 and provided to one or more user terminals 10 includes profile information 310 of a user using the chat book service, other users registered by the user, A user list display screen 340 may be provided that displays a list of users, connection display information 320 about the connection status of other registered users, and a registration request button 330 for additional registration of other users. However, since the information displayed on the user list display screen is not essential elements, additional information may be included in the user list display information.

The user list display screen 340 included in the user interface may display the user's profile information 310. For example, the user's profile information displayed on the user interface may include information related to the user's name, company name, position, contact information, etc. Additionally, the user list display screen 340 included in the user interface may display a list of one or more other users registered as friends by the user, and may display the connection status of one or more other users corresponding to each of the one or more other users' list. Connection indication information 320 may be displayed. For example, as shown in Figure 4, the user interface may display another user named 'Wonhyuk Choi' registered by the user on the user list display screen, and may display connection display information indicating the current connection status of 'Wonhyuk Choi'. can be displayed together. For a more specific example, if the user terminal corresponding to 'Choi Won-hyuk' among one or more other users is not connected to the computing device 100, the user interface may display connection indication information for that user, such as reference number 321. When user terminals corresponding to 'Taebin Jang' and 'Seonhyeong Kim' respectively access the computing device 100, the user interface may display connection display information such as reference numerals 322 and 323. The specific description of the user list display screen included in the user interface is only an example, and the present disclosure is not limited thereto.

In addition, the user list display screen 340 included in the user interface may additionally display an add friend button 330 for registering other users, and the processor 110 may display the add friend button from the user terminal 10 ( When receiving a selection input for 330), it may be determined to transmit a control signal to the user terminal to provide an ID input window where the unique ID of another user who wishes to conduct a conversation or meeting can be input. In addition, when receiving information about another user's ID through the corresponding ID input window from the user terminal 10, the processor 110 provides the other user's information (e.g., ID ) may be determined to be transmitted to the user terminal 10 to add and display the user list display screen 340.

Additionally, the user interface may include a conversation list display screen 350 that displays one or more conversations that the user has conducted with other users. Specifically, the user interface may provide a screen that displays the name of each of one or more conversations conducted with other users based on the user's selection input on the conversation list display screen 350. At this time, the conversation list display screen 350 included in the user interface may include one or more conversation items 370 performed by the user, a conversation content search button 360, and a chat room opening button 390 for opening a chat room. You can.

The conversation list display screen 350 included in the user interface can sort and display the names of one or more conversations that the user has conducted with each of one or more other users. In this case, the list of one or more conversations includes IR meeting (371), weekly meeting (372), OO company meeting (373), OOO interview (374), OOO consultation (375), and It may be displayed aligned with the general meeting 376. Additionally, the name for each conversation may be generated based on a selection input received from the user terminal when opening a chat room, generated as a result of semantic analysis for each conversation, or based on keywords determined in each conversation. It may have been created.

Additionally, the conversation list display screen 350 included in the user interface may display one or more viewpoint information corresponding to each of one or more conversations performed by the user. For example, as shown in FIG. 4, if the name of the conversation performed by the user is IR meeting 371, '2018.10.03. Time information such as '14:00' can be displayed. Additionally, the names of each of one or more conversations displayed on the conversation list display screen may be sorted in descending or ascending order based on viewpoint information.

Additionally, the conversation list display screen 350 included in the user interface may display a conversation search button 360 for searching a specific conversation among one or more conversations performed by the user. When receiving a selection input for a conversation search button from the user terminal 10 through the conversation list display screen 350, the processor 110 sends a control signal to the user terminal ( 10) You can decide what to transmit. In addition, when information about numbers or text is received in the search window from the user terminal 10, the processor 110 searches for and provides a conversation corresponding to the information about the numbers or text received from the user terminal 10. It may be decided to transmit a control signal to the user terminal 10. At this time, the selection input that the processor 110 receives from the user terminal 10 may be information about at least one of the name of the conversation, information on the time of the conversation, and information about one or more users included in the conversation. That is, the processor 110 specifies a specific message among one or more conversations based on the name of the conversation, information on the time of the conversation, and information about one or more users included in the conversation (e.g., ID information for each of the one or more users who performed the conversation, etc.). You can perform a search for conversations.

In addition, when receiving a selection input from the user terminal for a specific conversation name among one or more conversation names displayed on the conversation list display screen 350 included in the user interface, the processor 110 operates based on the selection input from the user terminal. It may be determined to transmit a control signal to the user terminal 10 to provide a conversation log display screen corresponding to the conversation name. A detailed description of the conversation log display screen provided to the user terminal based on the user terminal's selection input for a specific conversation name among one or more conversation names performed by the user will be described later with reference to FIG. 6.

Additionally, the conversation list display screen 350 included in the user interface may include a chat room opening button that receives a selection input from the user terminal 10 in order for the user to open a chat room with one or more other users. In addition, when receiving a selection input for the chat room opening button from the user terminal 10, the processor 110 sends a control signal to the user terminal 10 to provide a user selection input screen for including one or more other users in the chat room. ) can be decided to transmit. A detailed description of the user selection input screen included in the user interface and the chat room established based on the user terminal's selection input to the user selection input screen will be described later with reference to FIG. 5.

Figure 5 shows an example of a process for opening a chat room and a user interface for a chat room including a plurality of users according to an embodiment of the present disclosure.

According to one embodiment of the present disclosure, the processor 110 selects the user to include one or more other users in the chat room in response to the selection input of the user terminal 10 on the chat room opening button (reference number 390 in FIG. 4). It may be determined to transmit a control signal for providing an input screen to the user terminal 10. At this time, the user selection input screen may display a list of one or more other users registered by the user, as shown in FIG. 5, and a selection button for selecting the one or more other users corresponding to each of the one or more other users. may include. For a more specific example, referring to Figure 5, when a user wants to open a chat room that includes other users with the IDs of 'Taebin Jang', 'Seonhyeong Kim', and 'Kihoon Byun' among one or more other users registered by the user. , you can select and enter the selection buttons (reference numbers 422, 423, and 424) corresponding to 'Taebin Jang', 'Seonhyeong Kim', and 'Kihoon Byeon' respectively, and select and enter the button to complete the selection. In other words, the processor 110 receives a selection input for each of one or more other users through a user selection input screen from the user terminal (reference numbers 422, 423, 444) and a selection input for the confirmation button 430 for selection completion. In response, it may be determined to transmit a control signal for opening a chat room to the user terminal 10. The foregoing description is merely an example and the present disclosure is not limited thereto.

Additionally, the processor 110 may determine to transmit a control signal for providing a chat room display screen to the user terminal when a new chat room between the user and one or more other users is opened. For example, when the processor 110 receives a selection input from a user terminal for one or more other users ('Taebin Jang', 'Seonhyeong Kim', and 'Kihoon Byeon') as shown in (a) of FIG. 5, the processor 110 selects A chat room display screen such as (b) can be provided. In this case, as shown in (b) of FIG. 5, the chat room display screen may display information on when the chat room was opened, and information on users included in the chat room (e.g., user ID information ('Taebin Jang') ', 'Kim Seon-hyung' and 'Byun Ki-hoon')) can be displayed. The foregoing description is merely an example and the present disclosure is not limited thereto.

Additionally, the chat room display screen included in the user interface may include a recording start button 440 for receiving one or more voice data from each of one or more users included in the chat room. In addition, when receiving a selection input for the recording start button displayed on the chat room display screen from the user terminal, the processor 110 may receive one or more voice data from each of one or more user terminals based on the time of receiving the selection input. You can.

Additionally, the chat room display screen included in the user interface may display one or more voice recognition results generated based on one or more voice data received from each of one or more users included in the chat room, sorted by time and speaker. The configuration of sorting and displaying one or more voice recognition results generated based on one or more voice data received from each user by time and speaker has been specifically explained with reference to FIG. 2, and redundant explanation will be omitted. The user interface displays the voice recognition results output based on voice data received from the user's own terminal on one side of the chat room display screen, and the voice output based on voice data received from one or more other user terminals. The recognition results may be displayed on another side of the chat room display screen. For example, as shown in (b) of FIG. 5, the voice recognition result output through voice data of a specific user 450 is placed on the left side of the chat room display screen corresponding to the specific user 450 (451). ), and the voice recognition result output through the voice data of one or more other users (460) excluding the specific user can be placed (461) on the right side of the chat room display screen. In other words, the user interface can provide convenience to the user by ensuring readability of the text by displaying the conversation history between multiple users in the form of a chat window so that the user can easily understand the content of the conversation.

In addition, when receiving a selection input for a specific conversation name among one or more conversation names displayed on the conversation list display screen 350 from the user terminal, the processor 110 may transmit a control signal to provide a conversation list display screen to the user terminal. can decide A detailed description of the conversation log display screen will be described later with reference to FIG. 6.

Figure 6 shows an example of a user interface that provides a conversation log display screen in response to a user terminal's selection input for a specific conversation among one or more conversation lists performed by the user according to an embodiment of the present disclosure.

When receiving a selection input from the user terminal for a specific conversation name among one or more conversation names displayed on the conversation list display screen 350 included in the user interface, the processor 110 selects the corresponding conversation name based on the selection input from the user terminal. It may be determined to transmit a control signal to the user terminal to provide a chat log display screen corresponding to . For a specific example, as shown in (a) of FIG. 6, a selection input 510 for a conversation name corresponding to 'service planning meeting' among one or more conversation names displayed on the conversation list display screen is entered at the user terminal 10. ), the processor 110 may transmit a control signal to the user terminal 10 to provide a conversation log display screen corresponding to the 'service planning meeting'. In this case, the chat log display screen may include items such as (b) in FIG. 6. More specifically, the chat record display screen includes information 530 about each of one or more users who conducted a conversation, information 540 about one or more keywords related to the chat room, and one or more voice recognition results listed by time and speaker. It may include information 550 and a voice playback screen 560 that plays a voice section corresponding to the voice recognition result. The foregoing description is merely an example and the present disclosure is not limited thereto.

In addition, when the processor 110 receives a selection input for at least one of one or more keywords and one or more voice recognition results displayed on the chat log display screen 350 from the user terminal 10, the processor 110 generates voice data in response to the selection input. can be played.

A detailed description of how the processor 110 reproduces voice data based on the selection input of one or more keywords and one or more voice recognition results from the user terminal 10 will be described later with reference to FIG. 7.

FIG. 7 illustrates an example of a user interface displaying a screen that reproduces a specific point in time of voice data in response to a voice recognition result and a user's selection input for a keyword according to an embodiment of the present disclosure.

The user interface may display one or more voice recognition results on a conversation log display screen, and the processor 110 may display the corresponding voice recognition result in response to a selection input from the user terminal for at least one voice recognition result among the one or more voice recognition results. It may be determined to transmit a control signal to the user terminal 10 to provide a voice playback screen 630 that plays voice data at the time of creation. For a specific example, as shown in FIG. 7, the conversation log display screen may display one or more voice recognition results as indicated by reference numbers 611, 612, and 613. In this case, when receiving a selection input for reference number 612 from the user terminal 10, the processor 110 reproduces voice data sub-data starting from the speech section of the voice data corresponding to the voice recognition result. It may be determined to transmit a control signal for providing the playback screen 630 to the user terminal 10. That is, the processor 110 may provide a conversation record including one or more voice recognition results, and play the corresponding voice data sub-data in response to a selection input for a specific voice recognition result through the user terminal 10 in the conversation record. You can. Accordingly, a user can receive a conversation conducted with one or more other users in both text and voice form. In addition, because it is difficult to identify the content of a specific section in a voice file, the user can quickly identify a specific section in the conversation through one or more voice recognition results (i.e. text) displayed in the conversation log and provide voice data corresponding to that specific section. Data can be reproduced, making it easier to understand the content of the conversation being conducted.

In addition, the processor 110 is configured to reproduce voice data sub-data at the time of generating one or more voice recognition results matching the keyword in response to a selection input for a specific keyword among one or more keywords from the user terminal 10. It may be determined to transmit a control signal for providing the screen 630 to the user terminal 10. In this case, the location of one or more keywords may be identified in each of the one or more voice recognition results, and a specific keyword may be identified in one or more locations among the one or more voice recognition results included in the conversation log. Accordingly, the processor 110 may display one or more locations of the identified keywords on the voice playback screen 630. For a more specific example, as shown in FIG. 7, one or more keywords displayed on the chat log display screen include 'law', 'counseling', and 'copyright', and selection of a specific 'copyright' from the user terminal 10 When receiving an input, the processor 110 may provide a control signal to the user terminal 10 to provide a voice playback screen 630 that plays voice data subdata of a section corresponding to the keyword, The audio playback screen 630 may display one or more sections corresponding to 'copyright' (i.e., a keyword selected by the user) in the audio data. That is, the processor 110 may display each section (i.e., voice data subdata) in which a keyword exists on the voice playback screen 630 so that the user can easily search for a section corresponding to a specific keyword. Accordingly, the user can easily reproduce a section corresponding to a specific keyword in voice data including the content of a conversation conducted with another user, and thus can easily search for specific content.

According to an embodiment of the present disclosure, the user interface may include a text editing screen that receives a request for modification of one or more voice recognition results included in the conversation log from the user terminal. In addition, when the processor 110 receives a request to modify a specific voice recognition result among one or more voice recognition results included in the conversation log from the user terminal 10, the voice recognition model generates a voice recognition result corresponding to the voice recognition result. The voice recognition model can be corrected to output . At this time, the voice recognition model may utilize the STT (Speech to Text) algorithm to output a voice recognition result corresponding to each of one or more voice data subdata. That is, the processor 110 can improve the performance of the STT algorithm by reflecting feedback corresponding to a request for correction of the voice recognition result received from the user terminal 10 into the STT algorithm.

According to an embodiment of the present disclosure, the memory 120 includes a program code executable on the processor 110, identification information for each of one or more user terminals, and a voice recognition model that outputs a voice recognition result by inputting voice data. Information related to natural language processing and algorithms for natural language processing can be stored. For example, information about a modification request for modifying one or more voice recognition results received from a user terminal through a user interface may be stored.

According to an embodiment of the present disclosure, the memory 120 is a flash memory type, hard disk type, multimedia card micro type, or card type memory (e.g. (e.g. SD or -Only Memory), and may include at least one type of storage medium among magnetic memory, magnetic disk, and optical disk. The computing device 100 may operate in connection with web storage that performs the storage function of the memory on the Internet. The description of the memory described above is merely an example, and the present disclosure is not limited thereto.

According to one embodiment of the present disclosure, the network unit 130 can transmit and receive information to and from one or more user terminals 10. More specifically, the network unit 130 may include a wired/wireless Internet module for network connection. Wireless Internet technologies include Wireless LAN (WLAN) (Wi-Fi), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), and High Speed Downlink Packet Access (HSDPA). Wired Internet technologies include Digital Subscriber Line (XDSL), Fibers to the home (FTTH), and Power Line Communication (PLC).

In addition, the network unit 130 includes a short-range communication module, is located relatively close to the user terminal, and can transmit and receive data with an electronic device including a short-range communication module. Short range communication technologies include Bluetooth, Radio Frequency Identification (RFID), infrared data association (IrDA), Ultra Wideband (UWB), and ZigBee.

Figure 12 shows a flowchart for providing a conversation record service related to an embodiment of the present disclosure.

The computing device 100 may receive first voice data from the first user terminal and receive second voice data from the second user terminal (1210). The computing device 100 may perform preprocessing on each voice data to increase the recognition rate of the integrated voice data. Preprocessing may include at least one of standardization or frequency filtering for the signal size included in each of the one or more voice data.

The computing device 100 may divide the first voice data into one or more first voice data subdata based on the speech section, and divide the second voice data into one or more second voice data subdata based on the speech section. It can be divided. Additionally, the computing device 100 may identify the start and end points of each of the first voice data subdata and the second voice data subdata.

The computing device 100 may synchronize the start and end times of the first voice data and the second voice data through network packet exchange between the first user terminal and the second user terminal. Alternatively, the computing device 100 may synchronize the start and end points of the first voice data and the second voice data based on the similarity between the first voice data sub-data and the second voice data sub-data.

The computing device 100 may identify an overlap section based on the start and end points of each of the first voice data subdata and the second voice data subdata.

The computing device 100 may distinguish speakers based on the audio signal size of each of the first voice data and the second voice data. Alternatively, the computing device 100 may distinguish speakers through a statistical acoustic model based on the first voice data and the second voice data. The computing device 100 may distinguish speakers through comparison of voice signal sizes of a first voice data frame based on the first voice data and a second voice data frame based on the second voice data.

The computing device 100 may distinguish the speaker of the current frame based on speaker information of adjacent frames. The computing device 100 may include at least one of classifying speakers based on the ratio of the speaker's speech time for each of the first voice data and the second voice data.

When identifying an overlapping section, the computing device 100 may distinguish speakers by calculating the speech time ratio for each speaker based on the speakers in the frame excluding the overlapping section. Alternatively, when the overlapping section is a frame after a predetermined time ratio, the computing device 100 may distinguish speakers based on the speech time ratio of each speaker with respect to the previous frame of the overlapping section frame when identifying the overlapping section. That is, when an overlapping section is identified, the computing device 100 may perform speaker separation for the overlapping section by referring to the speaker separation result for the previous frame when the previous frame of the overlapping section is more than a predetermined rate of voice data. there is. When the previous frame of the overlapping section is more than a predetermined ratio of voice data, the computing device 100 performs speaker separation for the overlapping section by referring to the speaker separation results for the previous frames, thereby providing information on the previous frames of the overlapping section. This can be used when the speaker separation results have statistical significance.

The computing device 100 may generate integrated voice data based on the first voice data and the second voice data (1220).

The computing device 100 may generate a conversation log using integrated voice data using a voice recognition model (1230).

Figure 3 shows a flowchart for providing a conversation record service related to an embodiment of the present disclosure.

According to one embodiment of the present disclosure, computing device 100 may receive one or more voice data, the voice data including one or more utterances for each user, from each of one or more user terminals (210). . In this case, the voice data received by the computing device 100 is received from each of one or more user terminals 10, and there may be more than one.

According to an embodiment of the present disclosure, the computing device 100 may generate a conversation record by converting each of one or more voice data into one or more voice recognition results through a voice recognition model (220).

The computing device 100 may identify one or more speech sections for each of one or more voice data. Computing device 100 may identify one or more utterances included in voice data. Specifically, the computing device 100 may identify the user's speech section based on one or more silent sections present in voice data. Additionally, the computing device 100 may divide voice data into one or more voice data subdata based on the speech section.

Additionally, the computing device 100 may identify the voice of one or more users in each of one or more speech sections included in voice data. Additionally, the computing device 100 may determine an utterance whose voice level is greater than or equal to a predetermined level among the voices of each of the identified one or more users as a specific speaker corresponding to the voice data. Additionally, the computing device 100 may store a specific speaker corresponding to voice data by matching it with identification information of the user terminal.

Additionally, the computing device 100 may assign a unified viewpoint to each of the one or more voice data received from each of the one or more user terminals 10.

Additionally, the computing device 100 may identify the start time and end time of each of one or more voice data subdata. Specifically, the computing device 100 may identify the start time and end time of each of one or more voice data sub-data that is divided into voice data based on an utterance section included in the voice data.

Additionally, the computing device 100 may use each of the one or more voice data sub-data as input to a voice recognition model and output one or more voice recognition sub-results corresponding to each of the one or more voice data sub-data. In this case, the voice recognition model is a model that converts the voice language spoken by the user into text data, and may include one or more network functions pre-trained through training data. Additionally, the computing device 100 may determine the order of each of the one or more voice recognition sub-results in the conversation log based on the start and end points identified in each voice data sub-data that is the basis of the one or more voice recognition sub-results. there is.

Additionally, the computing device 100 may generate a conversation log based on the order of each of one or more voice recognition sub-results.

According to an embodiment of the present disclosure, the computing device 100 may generate a user interface including a conversation log and provide the user interface to one or more user terminals (230).

Figure 8 illustrates means for providing a conversation record service related to one embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device 100 collects one or more voice data from each of one or more user terminals, where the voice data includes one or more utterances for each user, in order to provide a conversation record service. means for receiving (710), means (720) for generating a conversation record by converting each of the one or more voice data into one or more voice recognition results through a voice recognition model, and generating a user interface including the conversation record to generate the one or more voice data. It may include means 730 for providing the information to the user terminal.

Alternatively, means for standardizing the magnitude of a signal included in each of the one or more voice data based on a comparison of the magnitude of the signal included in each of the one or more voice data with the magnitude of a predetermined reference signal, and means for standardizing the magnitude of the signal included in each of the one or more voice data, and It may further include means for analyzing the frequencies included in each voice data and removing at least one specific frequency.

Alternatively, means for identifying one or more speech sections based on one or more silent sections included in the speech data and separating speakers based on the loudness of speech in the one or more speech sections included in the speech data. Additional means may be included.

Alternatively, the means for separating speakers based on the utterance size in one or more utterance sections included in each of the one or more speech data may include the size of the speech voice of a predetermined size or more among the one or more utterances included in the speech data. Means for determining an utterance having a specific speaker corresponding to the voice data, means for removing voices other than the voice of the determined specific speaker from the voice data, and a user terminal that transmitted the voice data as the specific speaker. It may include means for matching and storing identification information.

Alternatively, the means for generating a conversation record by converting each of the one or more voice data into one or more voice recognition results through a voice recognition model may include dividing the voice data into one or more voice data sub-data based on an utterance section. means for identifying a start point and an end point of each of the one or more voice data subdata, and corresponding to each of the one or more voice data subdata by using each of the one or more voice data subdata as an input to the voice recognition model. means for outputting one or more voice recognition sub-results, and each of the one or more voice recognition sub-results based on a start point and an end point identified in each voice data sub-data on which the one or more voice recognition sub-results are based. It may include means for determining the order in the conversation log.

Alternatively, the one or more voice data includes first voice data and second voice data received from each user terminal, and each of the one or more voice data is converted into one or more voice recognition results through a voice recognition model. The means for generating a conversation record divides the first voice data into one or more first voice data sub-data based on the speech section, and divides the second voice data into one or more second voice data sub-data based on the speech section. means for dividing into data, means for identifying a start point and an end point of each of the first voice data subdata and the second voice data subdata, the first voice data subdata and the second voice data subdata means for outputting one or more first voice recognition sub-results and one or more second voice recognition sub-results corresponding to each sub-data as input to the voice recognition model, the one or more first voice recognition sub-results and the Each of the one or more first voice recognition sub-results and the one or more second voice recognition sub-results based on start and end points identified in each voice data sub-data that is the basis of the one or more second voice recognition sub-results It may include means for determining an order in the conversation log and means for generating a conversation log based on the determined order and identification information of each of the one or more user terminals.

Alternatively, the first voice data sub-data and the second voice data sub-data are input to the voice recognition model, and one or more first voice recognition sub-results and one or more second voice recognition sub-results corresponding to each sub-data are generated. The means for outputting means for identifying an overlapping section based on the start time and end time of each of the first voice data subdata and the second voice data subdata, and when the overlapping section is identified, the overlap Means for determining whether to divide at least one of the first voice recognition sub-result and the second voice recognition sub-result corresponding to the section, and when the division is determined, dividing the voice recognition sub-result corresponding to the overlapping section It may include means for

Alternatively, when the overlapping section is identified, means for determining whether to divide at least one of the first voice recognition sub-result and the second voice recognition sub-result corresponding to the overlapping section includes: It may include means for determining whether to divide the voice recognition result through a statistical language model for the corresponding voice recognition result.

Alternatively, determining one or more keywords for the conversation record based on at least one of a result of semantic analysis of the one or more voice recognition results, a repetition count for each word included in the one or more voice recognition results, and a predetermined conversation topic. It may further include means for identifying the location of each of the one or more keywords in the one or more voice recognition results.

Alternatively, the user interface may include a user interface that displays a user's profile, a list of one or more other users registered by the user, a registration request button for the user to register additional users, and whether the user has access to each of the one or more other users. It includes a list display screen, and information about the name of each of one or more conversations in which the user participates, a conversation search button for searching a specific conversation among the one or more conversations, and a chat room opening button for opening a chat room with one or more other users. It may include a conversation list display screen that displays.

Alternatively, when receiving a selection input for the chat room opening button from the user terminal, the processor sends a control signal to the user terminal to provide a user selection input screen for including the one or more other users in the chat room. It may further include means for determining what to transmit.

Alternatively, when receiving a selection input for a specific chat room from the list of one or more chat rooms in which the user participates from the user terminal, the processor provides a chat log display screen including a chat log corresponding to the specific chat room and one or more keywords. It may further include means for determining whether to transmit a control signal to the user terminal.

Alternatively, when receiving a selection input for a specific keyword among the one or more keywords from the user terminal, the processor searches for a voice recognition result corresponding to the keyword, and based on the generation of the searched voice recognition result, It may further include means for determining whether to transmit a control signal to the user terminal to provide a keyword playback screen for reproducing an utterance section of voice data.

Alternatively, when a new chat room is opened, the user interface may include a recording start button for receiving voice data from each of the one or more users included in the chat room, and one or more buttons to receive voice data from each of the one or more users included in the chat room. It may include a conversation log display screen that displays one or more voice recognition results generated based on voice data.

Alternatively, the user interface includes a text editing screen that receives a request for modification of one or more voice recognition results included in the conversation log from a user terminal, and the processor displays the text editing screen from the user terminal. When receiving a request for modification of one or more voice recognition results included in the conversation log, the device may further include means for calibrating the voice recognition model so that the voice recognition model outputs a voice recognition result corresponding to the modification request. You can.

Figure 9 illustrates a module for providing a conversation record service related to an embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device 100 collects one or more voice data from each of one or more user terminals, where the voice data includes one or more utterances for each user, in order to provide a conversation record service. A module 810 for receiving, a module 820 for generating a conversation record by converting each of the one or more voice data into one or more voice recognition results through a voice recognition model, and a user interface including the conversation record to generate the one It may include a module 830 for providing to the above user terminals.

Figure 10 shows logic for providing a conversation record service related to an embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device 100 collects one or more voice data from each of one or more user terminals, where the voice data includes one or more utterances for each user, in order to provide a conversation record service. Logic 910 for receiving, logic 920 for generating a conversation record by converting each of the one or more voice data into one or more voice recognition results through a voice recognition model, and generating a user interface including the conversation record to It may include logic 930 for providing to the above user terminals.

Figure 11 shows a circuit for providing a conversation record service related to an embodiment of the present disclosure.

According to one embodiment of the present disclosure, the computing device 100 collects one or more voice data from each of one or more user terminals, where the voice data includes one or more utterances for each user, in order to provide a conversation record service. A circuit 1010 for receiving, a circuit 1020 for generating a conversation log by converting each of the one or more voice data into one or more voice recognition results through a voice recognition model, and a user interface including the conversation record to generate the one It may include a circuit 1030 for providing information to the above user terminal.

Those skilled in the art will additionally recognize that the various illustrative logical blocks, components, modules, circuits, means, logic, and algorithm steps described in connection with the embodiments disclosed herein may be implemented using electronic hardware, computer software, or a combination of both. It must be recognized that it can be implemented with To clearly illustrate the interchangeability of hardware and software, various illustrative components, blocks, configurations, means, logics, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented in hardware or software will depend on the specific application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

Figure 16 shows a brief, general schematic diagram of an example computing environment in which embodiments of the present disclosure may be implemented.

Although the disclosure has been described generally in the context of computer-executable instructions that can be executed on one or more computers, those skilled in the art will appreciate that the disclosure can be combined with other program modules and/or implemented as a combination of hardware and software. will be.

Typically, program modules include routines, programs, components, data structures, etc. that perform specific tasks or implement specific abstract data types. Additionally, those skilled in the art will understand that the methods of the present disclosure are applicable to single-processor or multiprocessor computer systems, minicomputers, mainframe computers, as well as personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, and the like (each of which may be It will be appreciated that the system may be implemented with other computer system configurations, including those capable of operating in conjunction with one or more associated devices.

The described embodiments of the disclosure can also be practiced in distributed computing environments where certain tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote memory storage devices.

Computers typically include a variety of computer-readable media. Computer-readable media can be any medium that can be accessed by a computer, and such computer-readable media includes volatile and non-volatile media, transitory and non-transitory media, removable and non-transitory media. Includes removable media. By way of example, and not limitation, computer-readable media may include computer-readable storage media and computer-readable transmission media. Computer-readable storage media refers to volatile and non-volatile media, transient and non-transitory media, removable and non-removable, implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Includes media. Computer storage media includes RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital video disk (DVD) or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices; or any other medium that can be accessed by a computer and used to store desired information.

A computer-readable transmission medium typically implements computer-readable instructions, data structures, program modules, or other data on a modulated data signal, such as a carrier wave or other transport mechanism. Includes all information delivery media. The term modulated data signal refers to a signal in which one or more of the characteristics of the signal have been set or changed to encode information within the signal. By way of example, and not limitation, computer-readable transmission media includes wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of any of the above are also intended to be included within the scope of computer-readable transmission media.

An example environment 1100 is shown that implements various aspects of the present disclosure, including a computer 1102, which includes a processing unit 1104, a system memory 1106, and a system bus 1108. do. System bus 1108 couples system components, including but not limited to system memory 1106, to processing unit 1104. Processing unit 1104 may be any of a variety of commercially available processors. Dual processors and other multiprocessor architectures may also be used as processing unit 1104.

System bus 1108 may be any of several types of bus structures that may further be interconnected to a memory bus, peripheral bus, and local bus using any of a variety of commercial bus architectures. System memory 1106 includes read only memory (ROM) 2110 and random access memory (RAM) 2112. The basic input/output system (BIOS) is stored in non-volatile memory 2110, such as ROM, EPROM, and EEPROM, and is a basic input/output system that helps transfer information between components within the computer 1102, such as during startup. Contains routines. RAM 2112 may also include high-speed RAM, such as static RAM, for caching data.

Computer 1102 may also include an internal hard disk drive (HDD) 2114 (e.g., EIDE, SATA). Internal hard disk drive 2114 may also be configured for external use within a suitable chassis (not shown). ), a magnetic floppy disk drive (FDD) 2116 (e.g., for reading from or writing to a removable diskette 2118), and an optical disk drive 1120 (e.g., a CD-ROM disk) (1122) or for reading from or writing to other high-capacity optical media such as DVDs). Hard disk drive 2114, magnetic disk drive 2116, and optical disk drive 1120 are connected to system bus 1108 by hard disk drive interface 1124, magnetic disk drive interface 1126, and optical drive interface 1128, respectively. ) can be connected to. The interface 1124 for implementing an external drive includes at least one or both of Universal Serial Bus (USB) and IEEE 1394 interface technologies.

These drives and their associated computer-readable media provide non-volatile storage of data, data structures, computer-executable instructions, and the like. For computer 1102, drive and media correspond to storing any data in a suitable digital format. Although the description of computer-readable media above refers to removable optical media such as HDDs, removable magnetic disks, and CDs or DVDs, those skilled in the art will also recognize removable optical media such as zip drives, magnetic cassettes, flash memory cards, cartridges, etc. It will be appreciated that other types of computer-readable media, such as the like, may also be used in the example operating environment and that any such media may contain computer-executable instructions for performing the methods of the present disclosure.

A number of program modules may be stored in the drive and RAM 2112, including an operating system 2130, one or more application programs 2132, other program modules 2134, and program data 2136. All or portions of the operating system, applications, modules and/or data may also be cached in RAM 2112. It will be appreciated that the present disclosure may be implemented on various commercially available operating systems or combinations of operating systems.

A user may enter commands and information into computer 1102 through one or more wired/wireless input devices, such as a keyboard 2138 and a pointing device such as mouse 1140. Other input devices (not shown) may include microphones, IR remote controls, joysticks, game pads, stylus pens, touch screens, etc. These and other input devices are connected to the processing unit 1104 through an input device interface 1142, which is often connected to the system bus 1108, but may also include a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, It can be connected by other interfaces, etc.

A monitor 1144 or other type of display device is also connected to system bus 1108 through an interface, such as a video adapter 1146. In addition to monitor 1144, computers typically include other peripheral output devices (not shown) such as speakers, printers, etc.

Computer 1102 may operate in a networked environment using logical connections to one or more remote computers, such as remote computer(s) 1148, via wired and/or wireless communications. Remote computer(s) 1148 may be a workstation, computing device computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer device, or other conventional network node, and is generally connected to computer 1102. For simplicity, only memory storage device 1150 is shown, although it includes many or all of the components described. The logical connections depicted include wired/wireless connections to a local area network (LAN) 1152 and/or a larger network, such as a wide area network (WAN) 1154. These LAN and WAN networking environments are common in offices and companies and facilitate enterprise-wide computer networks, such as intranets, all of which can be connected to a worldwide computer network, such as the Internet.

When used in a LAN networking environment, computer 1102 is connected to local network 1152 through wired and/or wireless communication network interfaces or adapters 1156. Adapter 1156 may facilitate wired or wireless communication to LAN 1152, which also includes a wireless access point installed thereon for communicating with wireless adapter 1156. When used in a WAN networking environment, the computer 1102 may include a modem 1158 or be connected to a communicating computing device on the WAN 1154 or to establish communications over the WAN 1154, such as over the Internet. Have other means. Modem 1158, which may be internal or external and a wired or wireless device, is coupled to system bus 1108 via serial port interface 1142. In a networked environment, program modules described for computer 1102, or portions thereof, may be stored in remote memory/storage device 1150. It will be appreciated that the network connections shown are exemplary and that other means of establishing a communications link between computers may be used.

Computer 1102 may be associated with any wireless device or object deployed and operating in wireless communications, such as a printer, scanner, desktop and/or portable computer, portable data assistant (PDA), communications satellite, wirelessly detectable tag. Performs actions to communicate with any device or location and telephone. This includes at least Wi-Fi and Bluetooth wireless technologies. Accordingly, communication may be a predefined structure as in a conventional network or may simply be ad hoc communication between at least two devices.

Wi-Fi (Wireless Fidelity) allows connection to the Internet, etc. without wires. Wi-Fi is a wireless technology, like cell phones, that allows these devices, such as computers, to send and receive data indoors and outdoors, anywhere within the coverage area of a base station. Wi-Fi networks use wireless technology called IEEE 802.11 (a,b,g, etc.) to provide secure, reliable, and high-speed wireless connections. Wi-Fi can be used to connect computers to each other, the Internet, and wired networks (using IEEE 802.3 or Ethernet). Wi-Fi networks can operate in the unlicensed 2.4 and 5 GHz wireless bands, for example, at data rates of 11 Mbps (802.11a) or 54 Mbps (802.11b), or in products that include both bands (dual band). there is.

Those skilled in the art will understand that information and signals may be represented using any of a variety of different technologies and techniques. For example, data, instructions, commands, information, signals, bits, symbols and chips that may be referenced in the above description include voltages, currents, electromagnetic waves, magnetic fields or particles, optical fields. It can be expressed by particles or particles, or any combination thereof.

Those skilled in the art will understand that the various illustrative logical blocks, modules, processors, means, circuits and algorithm steps described in connection with the embodiments disclosed herein may be used in electronic hardware, (for convenience) It will be understood that the implementation may be implemented by various forms of program or design code (referred to herein as “software”) or a combination of both. To clearly illustrate this interoperability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been described above generally with respect to their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. A person skilled in the art of this disclosure may implement the described functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of this disclosure.

The various embodiments presented herein may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term “article of manufacture” includes a computer program, carrier, or media accessible from any computer-readable device. For example, computer-readable media include magnetic storage devices (e.g., hard disks, floppy disks, magnetic strips, etc.), optical disks (e.g., CDs, DVDs, etc.), smart cards, and flash memory. Includes, but is not limited to, devices (e.g., EEPROM, cards, sticks, key drives, etc.). Additionally, various storage media presented herein include one or more devices and/or other machine-readable media for storing information.

It is to be understood that the specific order or hierarchy of steps in the processes presented is an example of illustrative approaches. It is to be understood that the specific order or hierarchy of steps in processes may be rearranged within the scope of the present disclosure, based on design priorities. The appended method claims present elements of the various steps in a sample order but are not meant to be limited to the particular order or hierarchy presented.

The description of the presented embodiments is provided to enable any person skilled in the art to make or use the present disclosure. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the scope of the disclosure. Thus, the present disclosure is not limited to the embodiments presented herein but is to be interpreted in the broadest scope consistent with the principles and novel features presented herein.

Claims (11)

A computer program stored on a computer-readable storage medium executable by one or more processors, the computer program causing the one or more processors to perform the following operations, the operations comprising:
Obtaining first voice data based on a first user terminal and acquiring second voice data based on a second user terminal;
dividing the first voice data into one or more first voice data subdata, and dividing the second voice data into one or more second voice data subdata;
identifying a start point and an end point of each of the first voice data subdata and the second voice data subdata;
Synchronizing the start and end points of the first voice data and the second voice data based on the identified start and end points of each of the first voice data sub-data and the second voice data sub-data; and
generating integrated voice data based on the synchronized first voice data and the synchronized second voice data;
Including,
The operation of synchronizing the start and end points of the first voice data and the second voice data based on the identified start and end points of each of the first voice data sub-data and the second voice data sub-data. ,
Comprising the operation of synchronizing the start and end points of the first voice data and the second voice data based on the similarity of the first voice data subdata and the second voice data subdata,
A computer program stored on a computer-readable storage medium.
According to claim 1,
The operation of dividing the first voice data into one or more first voice data subdata and dividing the second voice data into one or more second voice data subdata includes:
dividing the first voice data into one or more first voice data subdata based on a speech section, and dividing the second voice data into one or more second voice data subdata based on a speech section;
Including,
A computer program stored on a computer-readable storage medium.
According to claim 1,
The operation of synchronizing the start and end points of the first voice data and the second voice data based on the start and end points of each of the identified first voice data sub-data and the second voice data sub-data,
Synchronizing the start and end points of the first voice data and the second voice data based on network packet exchange between the first user terminal and the second user terminal;
Including,
A computer program stored on a computer-readable storage medium.
According to claim 1,
The operation of identifying the start and end points of each of the first voice data subdata and the second voice data subdata includes:
identifying an overlapping section based on the start and end points of each of the first voice data subdata and the second voice data subdata;
containing more
A computer program stored on a computer-readable storage medium.
According to claim 1,
The operation of acquiring first voice data based on the first user terminal and acquiring second voice data based on the second user terminal,
An operation of performing preprocessing on each voice data to increase the recognition rate of the integrated voice data;
Contains and
The preprocessing is,
Comprising at least one of normalization or frequency filtering for the signal size included in each of the one or more voice data,
A computer program stored on a computer-readable storage medium.
According to claim 1,
The operation of generating integrated voice data based on the synchronized first voice data and the synchronized second voice data includes:
distinguishing speakers based on audio signal sizes of each of the synchronized first voice data and the synchronized second voice data; or
distinguishing speakers through a statistical acoustic model based on the synchronized first voice data and the synchronized second voice data;
Containing at least one of
A computer program stored on a computer-readable storage medium.
According to claim 6,
The operation of generating the integrated voice data by distinguishing speakers based on the voice signal size of each of the synchronized first voice data and the synchronized second voice data includes:
An operation of distinguishing speakers by comparing voice signal sizes of a first voice data frame based on the synchronized first voice data and a second voice data frame based on the synchronized second voice data;
Including,
A computer program stored on a computer-readable storage medium.
According to claim 6,
The operation of generating the integrated voice data by distinguishing speakers through the statistical acoustic model includes:
An operation of distinguishing speakers of the current frame based on speaker information of adjacent frames; or
An operation of distinguishing speakers based on a ratio of speech times of each speaker of the synchronized first voice data and the synchronized second voice data;
Containing at least one of
A computer program stored on a computer-readable storage medium.
According to claim 8,
The operation of distinguishing speakers based on the speech time ratio of each speaker of the synchronized first voice data and the synchronized second voice data includes,
When identifying an overlapping section, calculating the speech time ratio for each speaker based on the speakers in the frame excluding the overlapping section and distinguishing the speakers for the overlapping section; or
When identifying an overlapping section, an operation of distinguishing speakers in the overlapping section based on the ratio of speech time for each speaker to the previous frame of the overlapping section frame;
contains at least one of, and
When identifying the overlapping section, the operation of distinguishing speakers based on the ratio of speech time for each speaker to the previous frame of the overlapping section frame is,
Performed when the previous frame of the overlapping section frame is more than a predetermined ratio of voice data,
A computer program stored on a computer-readable storage medium.
As a method for providing an integrated voice data acquisition service,
A method of obtaining first voice data based on a first user terminal and obtaining second voice data based on a second user terminal;
a method of dividing the first voice data into one or more first voice data subdata, and dividing the second voice data into one or more second voice data subdata;
a method for identifying start and end points of each of the first voice data subdata and the second voice data subdata;
A method of synchronizing start and end points of the first voice data and the second voice data based on the identified start and end points of each of the first voice data sub-data and the second voice data sub-data; and
a method of generating integrated voice data based on the synchronized first voice data and the synchronized second voice data;
Including,
A method of synchronizing the start and end points of the first voice data and the second voice data based on the start and end points of each of the identified first voice data sub-data and the second voice data sub-data. ,
Comprising a method of synchronizing the start and end points of the first voice data and the second voice data based on the similarity of the first voice data subdata and the second voice data subdata,
Method for providing integrated voice data acquisition service.
A server for providing an integrated voice data acquisition service,
A server processor including one or more cores;
a server memory storing program codes executable on the server processor; and
A server network unit that transmits and receives data to and from one or more user terminals;
Contains and
The server processor is,
Obtain first voice data based on the first user terminal, and obtain second voice data based on the second user terminal;
dividing the first voice data into one or more first voice data subdata, and dividing the second voice data into one or more second voice data subdata;
identify start and end points of each of the first voice data subdata and the second voice data subdata;
synchronize the start and end points of the first voice data and the second voice data based on the identified start and end points of each of the first voice data sub-data and the second voice data sub-data; and
Generating integrated voice data based on the synchronized first voice data and the synchronized second voice data,
The process of synchronizing the start and end points of the first voice data and the second voice data based on the identified start and end points of each of the first voice data sub-data and the second voice data sub-data. ,
Comprising the process of synchronizing the start and end points of the first voice data and the second voice data based on the similarity between the first voice data subdata and the second voice data subdata,
A server that provides integrated voice data acquisition services.