KR20230123295A - Apparatus of reconstructing the voice, method of reconstructing the voice and method of verification of the voice - Google Patents

Abstract

It relates to a voice restoration device, a voice restoration method, and a voice verification method, wherein the voice processing device includes sensitive information processing voice data obtained by hash redaction processing of sensitive information with respect to voice data, voice conversion parameters, and hash redaction result parameters. Obtain parameter information including at least one, but the hash redaction result parameter is obtained by performing hash redaction processing on the voice conversion parameter, and the part corresponding to the voice conversion parameter among the input unit and sensitive information processing voice data is voice. and a processor for restoring a part corresponding to the hash redaction result parameter as it is or replacing the beep sound by using the conversion parameter.

Description

Voice restoration apparatus, voice restoration method, and voice verification method

It relates to a voice restoration device, a voice restoration method, and a voice verification method.\

Recently, with the development of information and communication technology, smartphones are widely distributed. In general, smart phones provide a recording function capable of storing offline conversations or calls, and accordingly, users can more easily record conversations between conversations anytime and anywhere using a smart phone. Users often record voice conversations with others for short-term or long-term recording of conversations, and these recorded data (voice) can be presented as important evidence in the event of a dispute between others, and its importance is increasing more and more. It is a trend that However, such recorded data has a very high risk of invading the privacy of others. Specifically, the recorded contents may contain sensitive information of the interlocutor(s), such as name, phone number, address, credit card number or password. It may cause great social or economic damage. In addition, the recently developed voiceprint recognition technology makes it possible to identify the speaker of the voice based on the recorded voice, making it impossible even if the other party wants to maintain anonymity. In addition, recorded audio data can be easily counterfeited and tampered with. Therefore, when recorded audio data is presented as evidence in the process of collecting evidence in a civil or criminal action, the court must conduct a forensic test in advance to identify the authenticity of the recorded data before adopting it as evidence. however. Although these forensic tests require a lot of time, sometimes they do not accurately determine the originality of the data, raising many questions about the reliability of the evidence.

Therefore, various voice processing technologies have been studied and introduced to solve these problems. For example, technologies for first detecting sensitive information and then deleting or anonymizing it have been developed to protect personal information. However, these technologies are not only vulnerable to forgery or alteration of voice data, but also have a problem in that they cannot simultaneously protect personal information and prevent forgery/falsification. This was because data (signatures, etc.) for authenticity identification had to be removed in the process of detecting and deleting (or anonymizing) sensitive information. In addition, techniques for securing and determining authenticity (originality) of voice include acoustic fingerprinting, audio watermarking, and perceptual audio hashing. However, these technologies generally do not have high reliability for the results, have poor personal information protection performance, and cannot clearly distinguish between voices from which sensitive information has been legitimately deleted and voices that have been forged or tampered with without permission. Due to its poor performance or processing speed, it was unsuitable for sufficiently solving the above problems. In addition, since these technologies cannot perform voice processing in real time during the recording process, there is a problem in that additional processing time is required for forgery and alteration prevention or personal information protection.

An object to be solved is to provide a voice processing device, a voice restoration device, a voice processing system, a voice processing method, a voice restoration method, and a voice verification method that simultaneously enable privacy protection and reliability verification of recorded voice.

In order to solve the above problems, a voice processing device, a voice restoration device, a voice processing system, a voice processing method, a voice restoration method, and a voice verification method are provided.

The voice processing device performs hash redaction processing on all or some segments of the voice data to obtain a hash redaction processing result, combines the hash redaction processing results to generate summary information, and among the voice data It may include a processor for obtaining sensitive information processing voice data by combining hash redaction processing results for portions corresponding to sensitive information, and a storage unit for storing the summary information and the sensitive information processing voice data.

The processor may arbitrarily determine a voice conversion parameter and convert the voice data using the voice conversion parameter.

The processor may perform hash redaction processing on the voice conversion parameter to obtain a hash redaction result parameter, and generate parameter information using at least one of the voice conversion parameter and the hash redaction result parameter. .

The processor may convert the voice using a double factor warping function.

The processor encodes the converted voice data into an Advanced Audio Coding (AAC) format, and performs hash redaction on a portion corresponding to the sensitive information among voice data encoded in the Advanced Audio Coding format. It is also possible to obtain the sensitive information processing voice data by combining.

The voice restoration apparatus obtains sensitive information processing voice data obtained by hash redaction processing of sensitive information with respect to the voice data, and parameter information including at least one of a voice conversion parameter and a hash redaction result parameter. The action result parameter is an input unit obtained by performing hash redaction processing on the voice conversion parameter and the part corresponding to the voice conversion parameter among the sensitive information processing voice data is restored using the voice conversion parameter, The part corresponding to the hash redaction result parameter may include a processor that restores the part as it is or replaces the beep sound.

The input unit further obtains summary information, wherein the summary information is obtained by performing hash redaction processing on all or some segments of the original voice data to obtain a hash redaction processing result, and combining the hash redaction acquisition results. The processor performs hash redaction processing on the sensitive information processing voice data to obtain a hash redaction processing result for verification, and verifies the voice data by comparing the hash redaction processing result for verification and summary information. can also be performed.

The voice processing system performs hash redaction processing on all or some segments of the voice data to obtain a hash redaction processing result, generates summary information by combining the hash redaction obtained results, and generates sensitive information among the voice data. A terminal device for recording that obtains sensitive information processing voice data by combining hash redaction processing results for a portion corresponding to information, receives the sensitive information processing voice data and the summary information, and receives the sensitive information processing voice data It may include a playback terminal device that obtains a result of hash redaction processing for verification by performing hash redaction processing, and performs verification of the voice data by comparing the result of hash redaction processing for verification and summary information.

The voice processing system converts the voice data by applying voice conversion parameters to the voice data, performs hash redaction processing on sensitive information of the converted voice data, obtains sensitive information processing voice data, and hashes the voice conversion parameters. Performing redaction processing to obtain a hash redaction result parameter, a recording terminal device for obtaining parameter information based on at least one of the voice conversion parameter and hash redaction result parameter, and acquiring the parameter information, Of the information processing voice data, the part corresponding to the voice conversion parameter is restored using the voice conversion parameter, and the part corresponding to the hash redaction result parameter is restored as it is or restored by replacing the beep sound, thereby processing the sensitive information It may also include a terminal device for reproduction that restores voice data.

The voice processing method includes obtaining a hash redaction processing result by performing hash redaction processing on all or some segments of voice data, generating summary information by combining the hash redaction processing results, and the voice processing method. The method may include obtaining sensitive information processing voice data by combining hash redaction processing results for portions corresponding to sensitive information among data.

The voice processing method may further include arbitrarily determining voice conversion parameters and converting the voice data using the voice conversion parameters.

The voice processing method includes obtaining a hash redaction result parameter by performing hash redaction processing on the voice conversion parameter, and generating parameter information using at least one of the voice conversion parameter and the hash redaction result parameter. It may include more steps.

The voice processing method may further include encoding the converted voice data into an Advanced Audio Coding (AAC) format, and the acquiring of the sensitive information processing voice data may include the advanced audio coding format. Acquiring the sensitive information-processed voice data by combining a result of hash redaction processing on a portion corresponding to the sensitive information among voice data encoded as .

The voice restoration method includes obtaining sensitive information processing voice data obtained by hash redaction processing of sensitive information on voice data, obtaining parameter information including at least one of a voice conversion parameter and a hash redaction result parameter, The hash redaction result parameter is obtained by performing hash redaction processing on the voice conversion parameter, and the part corresponding to the voice conversion parameter among the sensitive information processing voice data is restored using the voice conversion parameter. and restoring the part corresponding to the hash redaction result parameter as it is or replacing the beep sound.

The voice verification method includes the steps of obtaining sensitive information-processed voice data obtained by hash redaction processing of sensitive information on voice data, obtaining summary information, wherein the summary information is for all or some segments of original voice data. Performing hash redaction processing to obtain a hash redaction processing result and obtaining a result of hash redaction processing by combining the obtained hash redaction processing results It may also include obtaining and verifying the voice data by comparing a result of hash redaction processing for verification and summary information.

According to the above-described voice processing device, voice restoration device, voice processing system, voice processing method, voice restoration method, and voice verification method, it is possible to obtain the effect of simultaneously enabling privacy protection and reliability certification for recorded voice. .

According to the above-described voice processing device, voice restoration device, voice processing system, voice processing method, voice restoration method, and voice verification method, the problem of leaking personal information of a talker can be solved by deleting or masking sensitive information in a recorded voice. do.

According to the above-described voice processing device, voice restoration device, voice processing system, voice processing method, voice restoration method, and voice verification method, the quality of voice data can be maintained the same as or close to the original even after sensitive information is removed. can also be obtained

According to the above-described voice processing device, voice restoration device, voice processing system, voice processing method, voice restoration method, and voice verification method, the original recorded voice data and the voice data obtained by unauthorized manipulation based on this are quickly and Since it can be easily distinguished, the reliability of voice data can be improved, and accordingly, an advantage of being able to block false statements based on manipulated recordings in advance can be obtained in the event of a dispute.

1 is a block diagram of an embodiment of a voice processing apparatus.
2 is a diagram for explaining an embodiment of segment separation.
3 is a diagram for explaining an example of voice conversion and hash redaction processing.
4 is a diagram for explaining an example of a process of removing sensitive information.
5 is a diagram for explaining an example of an encoding process.
6 is a block diagram of an embodiment of a voice restoration apparatus.
7 is a diagram for explaining an example of an operation of a verification unit.
8 is a diagram for explaining an example of an operation of a restoration unit.
9 is a diagram for explaining another example of an operation of a restoration unit.
10 is a diagram of one embodiment of a voice processing system.
11 is a flowchart of an embodiment of a voice processing method.
12 is a flow chart of one embodiment of a voice verification method.
13 is a flowchart of an embodiment of a voice restoration method.

In the entire specification below, the same reference numerals refer to the same components unless otherwise specified. A term with an added 'unit' used below may be implemented in software and/or hardware, and depending on an embodiment, one 'unit' may be implemented as one physical or logical component, or a plurality of 'units' may be implemented as one physical or logical component. It is possible to implement one physical or logical component, or one 'unit' to implement a plurality of physical or logical components. When a part is said to be connected to another part throughout the specification, this may mean that a part and another part are physically connected to each other and/or electrically connected. In addition, when a part includes another part, this means that it does not exclude another part other than the other part unless otherwise stated, and may further include another part according to the designer's choice. do. Expressions such as the first to Nth (N is a natural number of 1 or more) are for distinguishing at least one part (s) from other part (s), and do not necessarily mean that they are sequential unless otherwise specified. In addition, singular expressions may include plural expressions, unless there is a clear exception from the context.

Hereinafter, an embodiment of the audio processing apparatus 100 will be described with reference to FIGS. 1 to 5 .

1 is a block diagram of an embodiment of a voice processing apparatus.

Referring to FIG. 1 , the voice processing device 100 may include a processor 200 provided for voice processing in one embodiment, and may further include a voice input unit 101 according to an embodiment. can Also, if necessary, the audio processing device 100 may further include at least one of an input unit 103, a communication unit 105, and a storage unit 107.

The voice input unit 101 is a voice processing device by receiving a sound wave (voice) uttered by at least one person or a sound wave generated from other surrounding objects and outputting an analog or digital electrical signal corresponding to the received sound wave. 100 may obtain acoustic data.

Here, the sound data may include data on the uttered voice (hereinafter referred to as original voice data), and the original voice data may include a person who wants to record a voice by manipulating the voice processing device 100. It may include voice data according to utterance or voice data according to utterances of multiple people (hereinafter referred to as talkers), such as a user and a conversation partner.

The voice input unit 101 may be implemented using a condenser microphone, a piezoelectric element microphone, and/or a microelectromechanical systems (MEMS) microphone.

The input unit 103 receives various data, instructions, or programs (which can be referred to as apps, applications, or software) for operation of the voice processing device 100, and stores the input data, instructions, or programs, etc. ) or the processor 200.

For example, the input unit 103 receives raw voice data recorded from another device (for example, an external memory device, etc.), or receives a command from a user or the like on whether or not to start recording or process the recorded voice. Alternatively, in the segment separation process, information on whether the corresponding segment belongs to the user or at least one of the other conversationalists may be received from the user and transmitted to the segment separation unit 220 of the processor 200. there is.

In addition, the input unit 103 inputs sensitive information (e.g., the name, phone number, address, resident registration number, social security number, passport number, password, contents corresponding to business secrets, personal It is also possible to receive a selection of a conversation partner (user or conversation partner) for at least a part corresponding to information related to a person's private life or other information that is not known to the outside world. In addition, the input unit 103 can receive a command such as deletion or transmission of recorded original voice data or sensitive information processing voice data (p-audio).

The input unit 103 may be integrally provided with the audio processing device 100 or may be physically separated, depending on the embodiment. The input unit 103 may include, for example, a keyboard, a mouse, a tablet, a touch screen, a touch pad, a track ball, a track pad, a scanner device, an image capturing module, an ultrasonic scanner, a motion detection sensor, a vibration sensor, a light receiving sensor, and a pressure-sensitive sensor. , a proximity sensor and/or a data input/output terminal, but is not limited thereto.

The communication unit 105 is connected to a wired or wireless communication network and communicates with another external device to transfer data, instructions, programs, etc. to the other device, or to receive them from the other device and store the received data. It may be transmitted to the unit 107 or the processor 200.

Here, other devices may include, for example, the voice restoration device 300 or the signature processing device 500, but are not limited thereto.

In one embodiment, the communication unit 105 transmits at least one piece of summary information (p-digest) obtained by the processor 200 to the signature processing device 500, and from the signature processing device 500 to at least one piece of summary information (p-digest). Receives at least one piece of signed digest information (signed p-digest, hereafter referred to as sp-digest) corresponding to the digest information (p-digest), transmits it to the storage unit 107, and stores the digest in the storage unit 107. In addition to or in place of the information (p-digest), it is also possible to store signed summary information (sp-digest).

In addition, the communication unit 105 may include at least one of the sensitive information processing voice data (p-audio) from which sensitive information is removed by the processor 200 and at least one parameter information (p-profile) obtained by the processor 200. It is also possible to pass one to the voice restoration device 300.

Depending on the embodiment, the communication unit 105 may receive original voice data recorded through another information processing device from another external information processing device (eg, a smart phone or a recorder). The communication unit 105 may be implemented using, for example, at least one communication module (for example, a LAN card, a short-distance communication module, or a mobile communication module).

The storage unit 107 temporarily or non-temporarily stores at least one data, instruction or program (algorithm) necessary for the operation of the voice processing device 100 or data obtained during or as a result of the processing of the processor 200. can be saved

For example, the storage unit 107 stores at least one of parameter information (p-profile) obtained by the processor 200, sensitive information processing voice data (p-audio), and summary information (p-digest), as will be described later. It can be stored and transmitted to the communication unit 105 or the processor 200 as needed. Also, the storage unit 107 may store signed summary information (sp-digest).

The program stored in the storage unit 107 may be directly written or modified by a designer such as a programmer and then stored in the storage unit 107, or may be stored in the storage unit 107 from another physical recording medium (such as an external memory device or a compact disk (CD)). It may be received and stored, and/or obtained or updated through an electronic software distribution network accessible through a wired/wireless communication network.

The storage unit 107 may include at least one of a main memory device and an auxiliary memory device. The main memory device may be implemented using a semiconductor storage medium such as ROM and/or RAM, and the auxiliary memory device may be a flash memory device (solid state drive (SSD)). ), etc.), SD (Secure Digital) card, hard disk drive (HDD, Hard Disc Drive), compact disk, DVD (DVD) or laser disk, etc. can be implemented based on a device that can store data permanently or semi-permanently. .

If necessary, the audio processing device 100 may further include an output unit (not shown). The output unit, for example, visually or aurally outputs at least one of sensitive information processing voice data (p-audio), parameter information (p-profile), and summary information (p-digest) to the outside to inform the user, etc. can provide

In addition, the output unit may provide a user interface (for example, a graphical user interface (GUI)) for receiving information such as selection of a speaker corresponding to a segment, selection of sensitive information, or start of recording to a user. . Accordingly, the user or the conversation partner can check the recorded voice and select sensitive information or a speaker of a specific segment within the recorded voice. The output unit may include, for example, a display, a speaker device, a printer device, an image output terminal, an audio output terminal, and/or a data input/output terminal.

According to one embodiment, the processor 200 may perform voice conversion, process sensitive information within raw voice data, and/or data necessary for voice verification (e.g., summary information (p-digest)). Alternatively, an operation of generating parameter information (p-profile, etc.) may be performed.

In addition, the processor 200 may control the overall operation of the voice processing device 100 or may further perform necessary arithmetic processing for this purpose. The processor 200 may include, for example, a central processing unit (CPU), a graphic processing unit (GPU), a micro controller unit (MCU), an application processor (AP), ), an electronic control unit (ECU), and/or at least one electronic device capable of performing other various calculation and control processes. These devices may be implemented using, for example, one or more semiconductor chips, circuits, or related components alone or in combination.

Referring to FIG. 1 , the processor 200, in one embodiment, includes a pre-processor 210, a segment separator 220, a parameter acquisition unit 230, a parameter information acquisition unit 231, a voice It may include a conversion unit 240, a hash redaction processing unit 250, a sensitive information processing unit 255, a summary information generation unit 260, and an encoding unit 270. At least one of these may be omitted by a user or designer's arbitrary selection. Also, at least two of them may be logically separated or physically separated according to embodiments. For example, the processor 200 may be implemented using one physical device or may be implemented by combining multiple devices (eg, a central processing unit and a graphic processing unit).

The pre-processing unit 210 may receive and obtain an electrical signal (ie, original audio data) output from the audio input unit 101 and perform pre-processing on it. For example, the pre-processor 210 may convert an analog signal into a digital signal based on a predetermined sampling rate, may perform Fourier transform (or fast Fourier transform) on the analog signal or digital signal, and /or noise of the original speech data may be removed. Here, Fourier transform (or fast Fourier transform) may be performed in units of frames. The frame may be a slice obtained by dividing the original voice data into equal time units (eg, 20 to 30 microseconds). A frame may be used as a minimum unit in voice processing of the processor 100 . Depending on embodiments, the processor 100 may independently or dependently perform conversion or sensitive information processing for each frame.

2 is a diagram for explaining an embodiment of segment separation.

As shown in FIG. 2, the segment separation unit 220, at least the original audio data input through the audio input unit 101 or the audio data transmitted from the pre-processor 210 (hereinafter, the audio data 90 to be processed) It may be divided into segments (S1 to S4, which may further include silent segments (b1 to b3) according to embodiments. Here, each segment (S1 to S4) is a specific person (user or conversation partner). ) may include a part corresponding to the utterance of

Specifically, a conversation is configured by sequentially listing each utterance of a plurality of talkers, and each segment S1 to S4 may be divided and acquired in correspondence to each talker's respective utterance. Each of the segments S1 to S4 may include a plurality of sentence(s) or word(s) according to circumstances.

According to another embodiment, the segment separator 220, when the target audio data 90 is input, takes portions in which audio is generally connected and has almost no sound, and areas in which there is no or very little audio. At least one segment S1 to S4 may be separately obtained by dividing based on (b1 to b3).

In this case, the segment separator 220, when the target voice data 90 is input, if voice exists in the data over time (for example, when the data value is greater than a predefined value), it is Acquire and determine that it corresponds to a specific segment (for example, the first segment (S1)), and if a zone without voice (for example, b1) appears, the previously acquired data is determined to be a specific segment (for example, the first segment (for example, the first segment (S1)). By determining S1)), at least one segment S1 to S4 may be obtained.

Meanwhile, parts b1 to b3 without voice may also be treated as separate segments (silent segments). At least one segment S1 to S4 may be simultaneously or sequentially transmitted to the voice converter 240, and the silent segments b1 to b3 may also be transmitted to the voice converter 240 according to an embodiment.

In addition, simultaneously or sequentially, information (for example, the number of segments, etc.) on at least one segment (S1 to S4 and/or b1 to b3) may be transmitted to the parameter acquisition unit 230 to obtain parameters.

That is, the segment separator 220 sets parts having a size smaller than a predetermined standard through analysis of the target audio data 90 as silent zones b1 to b3 in which no or very little audio is present. Then, at least one segment is acquired by dividing the voice data 90 to be processed based on the silent area.

Meanwhile, since a plurality of short silent sections may exist in one segment according to a speaker's speech habit, the segment separator 220 may not set the silent section as a silent section if the length of the silent section is equal to or less than a preset standard.

3 is a diagram for explaining an example of a voice conversion and hash redaction process, and FIG. 4 is a diagram for explaining an example of a process for removing sensitive information.

The parameter acquisition unit 230 determines at least one parameter to be used by the voice conversion unit 240, for example, at least one pair of voice conversion parameters ((α1, β1) to (αt, βt)), and converts them into voice conversion It can be delivered to unit 240.

The voice conversion parameters (α1, β1) to (αt, βt) are values used for sufficient conversion of voice, and may have values within a logically or experimentally determined range. For example, the audio conversion parameters (α1, β1) to (αt, βt) may have values larger than the separation frequency ω_s described later and smaller than pi (π).

The parameter acquisition unit 230 may determine voice conversion parameters (α1, β1) to (αt, βt) for each segment (S1 to St, where t is a natural number greater than or equal to 1). In this case, all of the voice conversion parameters ((α1, β1) to (αt, βt)) corresponding to the respective segments (S1 to St) may be the same, all may be different, or some may be the same and others may be the same. may be different. Also, each voice conversion parameter ((α1, β1) to (αt, βt)) corresponding to each segment (S1 to St) may be randomly determined.

Therefore, as will be described later, if there is no separately prepared parameter information (p-profile), the other devices 300 and 500 use the voice conversion parameters ((α1, β1) to (αt, βt) applied to each segment S1 to s3 ) is impossible to determine. If necessary, the parameter acquiring unit 230 may further determine voice conversion parameters (not shown) for the silent segments b1 to b3.

The parameter information obtaining unit 231 may generate original parameter information (o-profile) by collecting voice conversion parameters ((α1, β1) to (αt, βt)) corresponding to each segment (S1 to St). .

That is, the parameter information acquisition unit 231 receives voice conversion parameters α1 and β1 corresponding to the first segment S1 from the parameter acquisition unit 230, and sequentially converts voice conversion parameters corresponding to other segments. After acquiring and receiving the speech conversion parameters (αt, βt) corresponding to the last segment (St), they are converted into speech conversion parameter(s) ((α1, β1) to (αt, βt)) for the segments (S1 to S1). The original parameter information (o-profile) may be generated by sequentially according to the order of St) or by combining them in another predefined order. According to embodiments, an operation of generating original parameter information (o-profile) may be omitted.

In addition, the parameter information acquisition unit 231 obtains at least one hash redaction result parameter (H(αk), H(βk) for at least one specific speech conversion parameter (αk, βk) from the hash redaction processing unit 250. )), and at least one hash redaction result parameter (H(αk), H(βk)) may be further used to generate parameter information (p-profile).

According to an embodiment, the parameter information acquisition unit 231 receives at least one voice conversion parameter ((α1, β1) to (αt, βt) corresponding to each segment (S1 to St) from the parameter acquisition unit 230. ) is received, and the parameter information is obtained by combining all or part of the speech conversion parameters ((α1, β1) to (αt, βt)) and at least one hash redaction result parameter (H(αk), H(βk)) (p-profile) can be obtained.

For example, the parameter information acquisition unit 231 arranges hash redaction result parameters H(αk) and H(βk) in a part corresponding to one or more specific segments Sk, and other segments S1 etc.) by arranging the voice conversion parameters ((α1, β1) to (αt, βt)) acquired by the parameter acquisition unit 230 and combining them sequentially or according to a predefined parameter information (p-profile) can also be done.

Here, the part corresponding to the specific segment (Sk) to which the hash redaction result parameters (H(αk), H(βk)) are to be arranged can be later restored by the voice restoration apparatus 300 for the corresponding segment (S1 to Sk). (permission) may be determined depending on whether or not.

For example, as shown in FIG. 4 , for converted voice data 241-L corresponding to a segment in which original voice can be restored in the future, the parameter information acquisition unit 231 determines the corresponding voice conversion parameters αL, βL) is acquired, and for the converted voice data 241-M corresponding to the segment in which the original voice cannot be restored, the corresponding hash redaction result parameters (H(αM), H(βM)) are obtained, and then , they may be combined according to the order of segments to generate parameter information (p-profile).

Here, a segment for which original voice restoration is possible or permitted may include a portion corresponding to the user's own speech, and a segment for which original voice restoration is impossible or permitted may include a portion corresponding to another person's speech. The parameter information (p-profile) may be transmitted to the communication unit 105 or the storage unit 107.

The voice conversion unit 240 may perform voice conversion for each segment S1 to St based on the selected voice conversion parameters ((α1, β1) to (αt, βt)).

In this case, the voice conversion unit 240 applies each voice conversion parameter ((α1, β1) to (αt, βt)) corresponding to each segment (S1 to St) to obtain a voice for each segment (S1 to St). You can also do conversions.

Here, voice conversion may be performed by applying voice conversion parameters ((α1, β1) to (αt, βt)) to each frame of the at least one segment (S1 to St), and the same segment (S1 to St) The same voice conversion parameters ((α1, β1) to (αt, βt)) are applied to frames belonging to, and different voice conversion parameters ((α1, β1) to (α1, β1) to frames belonging to different segments (S1 to St) are applied. (αt, βt)).

According to an embodiment, the voice conversion unit 240 may perform voice conversion for each of the segments S1 to St using a warping function. In this case, the voice conversion unit 240 may also perform voice conversion using a dual-factor warping function as described in Equation 1 below.

[Equation 1]

In Equation 1, ω is a normalized frequency having a value between 0 and π, and ω_s is a separation frequency to which warping is not applied. α and β are the above-described voice conversion parameters, respectively, and a breakpoint in Equation 1 (a point where the slope is changed) is determined according to their combination.

As described above, since the voice conversion parameters (α1, β1) to (αt, βt) may be determined differently for each segment S1 to St by the parameter acquisition unit 230, each segment S1 to St) is also performed differently.

Accordingly, voice conversion results corresponding to each of the segments S1 to St, that is, converted voice data 241-1 to 241-t can be obtained. The converted voice data 241-1 to 241-t may be delivered to the hash redaction processing unit 250 and the sensitive information processing unit 255, respectively, and may be delivered to the encoding unit 270 depending on the embodiment. there is. The encoding unit 270 may perform encoding processing on the converted voice data 241-1 to 241-t and then transfer the encoding result to the hash redaction processing unit 250.

The hash redaction processor 250 converts the converted voice data 241-1 to 241-t corresponding to each segment S1 to St as shown in FIG. 3 to protect personal information and prevent forgery/falsification. ), and hash redaction processing results corresponding to the respective segments (S1 to St), that is, hash redaction-processed voice data (251-1 to 251-t) can be obtained.

Hash redaction processing may be performed for each frame of the converted voice data 241-1 to 241-t. Here, each frame of the converted voice data 241-1 to 241-t may include a frame corresponding to each frame of the segments S1 to St. Hash redaction processing may be performed based on Equation 2 below.

[Equation 2]

In Equation 2, h_j is the hash redaction processing result 251-1 to 251-k corresponding to the j th segment (Sj, j is a natural number of 1 or greater) or the converted voice data 241-j, and H() Is a predetermined hash function, and may be arbitrarily determined by a user, a designer, or the processor 100 or predefined.

H(α_k) is the result obtained by applying one speech conversion parameter (α_k) to the hash function (ie, one hash redaction result parameter), and H(β_k) is another speech conversion parameter (β_k). ) to the hash function (ie, another hash redaction result parameter).

This hash redaction process has the following characteristics. Specifically, in the hash redaction process, all or part of the contents of the voice-converted frame are deleted to prevent identification of the speaker (ie, for voice privacy) by voiceprint recognition technology or the like, but information for voice verification (summary) Information (p-digest) can be obtained sufficiently as will be described later.

On the other hand, in the process, the acoustic characteristics of the original voice data (for example, the voice conversion parameters ((α1, β1) to (αt, βt))) are also hashed for verification (H(α_k), H( β_k)). In addition, hash redaction processing makes it easy to determine whether the voice data 90 to be processed is tampered with because a result different from that before the change is output when an acoustic or textual change is applied to a frame. In addition, since it is generally almost impossible to find other voice content or voice pitch that obtains the same hash redaction processing result as the hash redaction processing result of a specific voice content or voice pitch, such a hash redaction processing process It is also difficult to recover sensitive information removed by redaction processing.

In addition, the hash redaction processing unit 250 applies a hash function to all or some of the speech conversion parameters ((α1, β1) to (αt, βt)) corresponding to each segment, and obtains a corresponding hash redaction result Parameters H(α1), H(β1) to (H(αt), H(βt)) may be further obtained.

According to an embodiment, the hash redaction-processed voice data 251-1 to 251-t corresponding to each of the segments S1 to St may be transmitted to the summary information generator 260, and sensitive information It may be further passed to the processing unit 255 . In addition, the hash redaction result parameters (H(α1), H(β1) to (H(αt), H (βt)) may be transmitted to the parameter information acquisition unit 231. As described above, when generating the parameter information (p-profile), the parameter information acquisition unit 231 selects a part requiring voice modulation. For this, the received hash redaction result parameters (H(α1), H(β1) to (H(αt), H(βt)) instead of the voice conversion parameters (αk, βk) acquired by the parameter acquisition unit 230 ), parameter information (p-profile) may be generated by inserting hash redaction result parameters (H(αk), H(βk)) corresponding to the part requiring voice modulation.

The sensitive information processing unit 255 converts all or part (V1, V2) of the voice data (241-L, 241-M) converted by the voice conversion unit 240 into the converted voice data (241-L, 241-M). M) is replaced with hash redaction-processed voice data (251-L, 251-M) to perform processing on the voice data (241-L, 241-M), and as a result, each Sensitive information processing voice data 255-L, 255-M, and p-audio corresponding to the converted voice data 241-L and 241-M may be obtained.

Here, all or part (V1, V2) of the converted voice data (241-L, 241-M) may include sensitive information, and depending on the embodiment, all parts of the sensitive information (eg, passport) number) or only a part of sensitive information (for example, a part of a passport number).

Also, the converted voice data 241 -L and 241 -M may be voice data encoded by the encoding unit 270 . Sensitive information processing voice data (255-L, 255-M, p-audio) may be transmitted to the communication unit 105 or storage unit 107.

As shown in FIG. 3, the summary information generator 260 converts hash redaction processed voice data 251-1 to 251 corresponding to all or some of the segments S1 to St from the hash redaction processor 250. -t) may be received, and summary information (p-digest) may be generated by combining all or some of the received voice data 251-1 to 251-t subjected to hash redaction processing.

According to the embodiment, the summary information generation unit 260 sequentially combines the hash redaction-processed voice data 251-1 to 251-t corresponding to the original segments S1 to St to generate summary information p-digest. It is also possible to create

The generated summary information (p-digest) may be transmitted to the communication unit 105 or the storage unit 107. If necessary, the summary information (p-digest) is transmitted to the signature processing device 500, and the signature processing device 500 performs signature processing on the summary information (p-digest) to sign the signed summary information (sp-digest). ) can be created. The summary information (p-digest) or the signed summary information (sp-digest) may be used for verification of the speech recovery apparatus 300 later.

5 is a diagram for explaining an example of an encoding process.

The encoding unit 270 may encode the converted voice data 241-1 to 241-t in a specific format. Here, the specific format may include a predetermined audio file format (eg, WAV, AIFF, FLAC, TTA, MP3, AAC, or ATRAC).

For example, the encoding unit 270 may encode the converted voice data 241-1 to 241-t as shown in FIG. 5 in an Advanced Audio Coding (AAC) format.

All of the operations of the processor 100 described above must be applied to the same voice anonymous frame at the bit level. However, loss of production (loss of quality when using lossy compression) may cause discrepancies between real-time frames and stored versions due to audio encoding, invalidating digest information (p-digest) or signed digest information (sp-digest).

In order to prevent this, the audio may be kept uncompressed, but this has a problem in that the file size greatly increases. The advanced audio encoding format enables the speech data 90 to be processed and the data subject to hash redaction to maintain the sameness with each other even when the speech data 90 to be processed is loss-compressed, and in a different format (e.g., MP3) not only provides better performance in both recording quality and storage efficiency by providing higher quality sound and smaller size than MP3, but also has high versatility, preventing invalidation of such summary information (p-digest) and increasing capacity. to be able to solve

Voice data encoded in the advanced audio encoding format is delivered to the hash redaction processing unit 250, and the hash redaction processing unit 250 may perform hash redaction processing based on the received encoded voice data. .

According to an embodiment, the hash redaction processing unit 250 obtains encoded voice data, performs decoding on it, obtains voice data before encoding, performs hash redaction on the voice data before encoding, , The hash redaction processed voice data 251-1 to 251-t is obtained, and the hash redaction processed voice data 251-1 to 251-t is sent to the sensitive information processor 255 or the summary information generator ( 260) and/or the hash redaction result parameters (H(α1), H(β1) to (H(αt), H(βt)) may be transmitted to the parameter information acquisition unit 231.

Depending on the embodiment, advanced audio coding compatible encoding may be further performed on sensitive information processing voice data (p-audio) for audio delivery. This is provided to deal with secondary production losses.

In this case, as shown in FIG. 5 , a modified discrete cosine transform (MDCT) method of an advanced audio coding method may be used. Specifically, a window for two consecutive frames is compressed into one modified discrete cosine transform block, and the window is moved to the next block by overlapping frames.

Since the modified discrete cosine transform block is stored separately in the advanced audio encoding file, post-processing can be performed on a specific block (frame for hash redaction processing) while maintaining other blocks as they are. Accordingly, sensitive information processing voice data (p-audio) in an advanced audio coding format can be generated, and the original compressed modified discrete cosine transform block is maintained except for important frames. Here, important frames include frames subjected to hash redaction processing. The size of the above-described sensitive information processing audio data (p-audio) may be slightly larger than the size of the stored version (o audio). However, since most of the modified discrete cosine transform blocks are largely maintained except for some important specific blocks, the above-described encoding can have a high compression ratio. Since the converted part may be included in the audio file through lossless compression, when sensitive information processing voice data (p-audio) is given, the voice restoration apparatus 300 restores the same frame based on the summary information (p-profile). You can do it.

Hereinafter, the voice restoration apparatus 300 will be described with reference to FIGS. 6 to 10.

6 is a block diagram of an embodiment of a voice restoration apparatus.

The voice restoration device 300 receives sensitive information processing voice data (p-audio) generated by the voice processing device 100, and further receives parameter information (p-profile) as needed, and/or summary information. (p-digest) or signed summary information (sp-digest) is received, and based on this, voice can be restored or voice verification can be performed. As shown in FIG. 6, the voice restoration apparatus 300, in one embodiment, includes a processor 400 and, if necessary, further includes at least one of an input unit 303 and a storage unit 307. can do.

The input unit 303 may receive commands, data, and/or programs necessary for the operation of the voice restoration device 300 and transmit them to the storage unit 307 or the processor 400 . For example, the input unit 303 may be connected to an external memory device to receive sensitive information processing voice data (p-audio), parameter information (p-profile), and/or summary information (p-digest) therefrom. there is.

In addition, the input unit 303 is connected to a wired and/or wireless communication network to receive commands, data, and/or programs from the voice processing device 100, the signature processing device 500, and/or other devices. can

The input unit 303 may include a card reader, a disk reader, a data input/output terminal, a wired communication module, or a wireless communication module.

The storage unit 307 may temporarily or non-temporarily store commands, data, and/or programs related to the operation of the voice restoration device 300 . Here, the data may include sensitive information processing voice data (p-audio), parameter information (p-profile), and/or summary information (p-digest). Depending on the embodiment, the voice restoration apparatus 300 may further include an output unit (not shown) capable of outputting the verification result of the processor 400 or the restored voice to the outside or transferring the same to another information processing device. .

Since each of the input unit 303, storage unit 307, and output unit can be implemented in the same or partially modified form as the input unit 103, storage unit 107, and output unit of the above-described audio processing device 100, , A specific and detailed description will be omitted below.

The processor 400 may perform verification on sensitive information processing voice data (p-audio) and/or restore voice based on the sensitive information processing voice data (p-audio). Like the processor 200 described above, the processor 400 may be implemented using a central processing unit or a graphic processing unit.

The processor 400, in one embodiment, may include a decoding unit 401, a verification unit 410 and a sound source restoration unit 420.

The decoding unit 401 may perform decoding on the audio data encoded by the encoding unit 270 .

According to an embodiment, the decoding unit 401 may decode data encoded in the advanced audio coding format and deliver the decoding result to the verifying unit 410 . The decoding unit 401 may be omitted if unnecessary, such as when the audio processing device 100 does not perform an encoding process.

7 is a diagram for explaining an example of an operation of a verification unit.

The verification unit 410 receives the sensitive information processing voice data (p-audio) acquired by the input unit 303 or receives the voice data decoded by the decoding unit 401, and based on this, the authenticity of the voice data ( originality) can be verified.

According to an embodiment, the verification unit 410 may include a hash redaction processing unit 411 and a comparison unit 413 . Referring to FIG. 7, the hash redaction processor 411 obtains at least one segment Sm of the received voice data, performs hash redaction processing on the at least one segment Sm, A hash redaction processing result 411a for verification may be obtained.

Here, the hash redaction processing performed by the hash redaction processing unit 411 of the voice restoration device 300 may be the same as the hash redaction processing performed by the hash redaction processing unit 250 of the voice processing device 100. there is.

In addition, the at least one segment Sm includes, for example, parts U11 and U12 corresponding to speech, and according to an embodiment, a part converted by the speech conversion unit 240 of the speech processing apparatus 100. possible) and a part R1 corresponding to the voice subjected to hash redaction processing by the hash redaction processing unit 250 of the voice processing apparatus 100.

That is, the parts U11 and U12 corresponding to the voice may be parts that have not been subjected to hash redaction processing. In this case, hash redaction processing is performed on the parts (U11 and U12) corresponding to the voice, and the part (R1) corresponding to the redaction-processed voice is maintained as it is without separate conversion, and thus hash redaction for verification A processing result 411a is obtained.

The comparator 413 obtains summary information (p-digest) or signed summary information (sp-digest) corresponding to at least one segment (Sm), and converts the hash redaction processing result (411a) for verification into summary information. (p-digest) or against signed digest information (sp-digest).

Here, the summary information (p-digest) corresponding to at least one segment (Sm), the summary information generator 260 of the voice processing device 100 combines the results of hash redaction processing for the same segment (Sm). It may be summary information (p-digest) obtained by doing so.

Since hash redaction processing is performed in the same way for the same segment (Sm), if there is no forgery or falsification, the hash redaction recorded in the summary information (p-digest) or the signed summary information (sp-digest) The processing result and the hash redaction processing result 411a for verification are inevitably the same.

Therefore, the comparison unit 413 compares the received verification hash redaction processing result 411a with digest information (p-digest) or signed digest information (sp-digest), and if the two are identical, the corresponding segment It is determined that the audio data of (Sm) is not forged or modulated, and conversely, if the two are different from each other, it can be determined that the audio data of the corresponding segment (Sm) is forged or modulated. Accordingly, whether or not the voice is genuine can be determined more easily.

8 is a diagram for explaining an example of an operation of a restoration unit, and FIG. 9 is a diagram for explaining another example of an operation of a restoration unit.

The restoration unit 420 restores the sensitive information processing voice data (p-audio) so that the user of the voice restoration apparatus 300 can use the original voice data or the converted voice data 241-1 to 241-t. can make you listen

According to an embodiment, the restoration unit 420 converts the converted voice data (ie, the voice data converted by the voice conversion unit 240) corresponding to a specific segment (Sm) according to whether the original voice data can be restored. It may be restored as voice data and/or the converted voice may be restored as it is. Here, the restoration unit 420 may perform restoration using parameter information (p-profile).

Specifically, if the original voice data of the corresponding segment Sm can be restored, voice conversion parameters αM and βM corresponding to the corresponding segment Sm can be obtained from the parameter information p-profile. When the restoration unit 420 acquires the voice conversion parameters (αM, βM), as shown in FIG. 8, using the obtained voice conversion parameters (αM, βM), the converted voice parts (U11, U12) Inverse transformation is performed to obtain original speech data (421a).

On the other hand, the restoration unit 420 restores the hash redaction-processed audio portion R1 by maintaining it as it is or replacing it with a separately prepared sound (eg, a beep sound) (422a). Accordingly, while the voice data of the voice restoration apparatus 300 is substantially the same as the original voice data, sensitive information is masked and restored.

Conversely, if original speech data cannot be restored for a specific segment (Sm), the speech conversion parameters (αN, βN) corresponding to the corresponding segment (Sn) are absent in the parameter information (p-profile), and instead, the hash redaction result parameter (H(αN), H(βN)) is recorded.

Therefore, the restoration unit 420 cannot acquire voice conversion parameters (αN, βN) for voice restoration. In this case, since the restoration unit 420 cannot restore the original speech using the hash redaction result parameters H(αN) and H(βN), as shown in FIG. 9 , the converted speech parts U21 and U22 ) is restored as it is (421b). Accordingly, the voice data restored by the restoration unit 420 retains the voice converted by the above-described voice conversion unit 240 as it is.

Meanwhile, the restoration unit 420 may restore the hash redaction-processed audio portion R1 as it is or replace it with a separately prepared sound (eg, a beep sound) as described above (422a). Accordingly, the voice data restored by the restoration unit 420 is the same as the converted voice data 241-1 to 241-t, so that the anonymity of the conversation partner or conversation partner can be maintained, and at the same time, sensitive information can be masked and protected. be able to

The signature processing device 500 receives the summary information (p_digest) generated by the voice processing device 100 in real time, periodically or at any time, and performs signature processing on the summary information (p_digest). It may be performed to obtain signed summary information (sp-digest), and store it in a storage unit (not shown) of the signature processing device 500.

Here, the signature processing may be performed using, for example, a unique hash previously verified through reliable timestamping. Since the signed summary information (sp-digest) is stored in the signature processing device 500, unauthorized forgery, alteration, or invalidation of the summary information (p-direct) can be more robustly prevented.

The signature processing device 500 automatically transmits the signed summary information (sp-digest) to at least one of the voice processing device 100 and the voice restoration device 300 according to predefined settings or at the request of a user or decompressor. can When the voice processing device 100 receives the signed summary information (sp-digest), it replaces the existing summary information (p-digest) and stores it, and the voice recovery device 300 uses the signed summary information ( sp-digest) to perform verification.

According to embodiments, the processor 200 of the above-described voice processing device 100 may further perform a signature processing operation instead of the signature processing device 500 . That is, when the summary information (p-digest) is obtained, the voice processing apparatus 100 acquires the signed summary information (sp-digest) by performing a signature process based on a predetermined signature algorithm, and stores it ( 109) and/or transmitted to the voice restoration device 300.

The above-described speech processing apparatus 100 is specifically designed to perform one or more processes such as the above-described preprocessing, segment separation, speech conversion, encoding, parameter information acquisition, hashing redaction processing, sensitive information processing, and/or summary information acquisition. It may be implemented using a devised device, and the above-described voice restoration device 300 may be implemented using a device specially designed to perform one or more processes such as the above-described decoding, verification process, and sound source restoration, and signature processing. Apparatus 500 uses a specially designed device to perform reception of digest information (p-digest), signature processing on digest information (p-digest), and delivery of signed digest information (sp-digest). may be implemented.

Also, according to embodiments, at least one of the voice processing device 100, the voice restoration device 300, and the signature processing device 500 may be implemented by using one or more information processing devices alone or in combination. Here, one or more information processing devices are, for example, smart phones, tablet PCs, desktop computers, laptop computers, server hardware devices, smart watches, smart bands, microphones, voice recorders, video recording devices (camcorders) or action cam, etc.), head mounted display (HMD: Head Mounted Display) device, handheld game console, navigation device, personal digital assistant (PDA: Personal Digital Assistant), smart key, remote control device (remote control), digital television, set-top Boxes, digital media player devices, media streaming devices, DVD playback devices, compact disc (CD) playback devices, sound playback devices (such as artificial intelligence speakers), home appliances (such as refrigerators, fans, air conditioners, or washing machines), manned /Unmanned mobile vehicles (such as cars, buses or two-wheeled vehicles, mobile robots, wireless model vehicles or robot vacuum cleaners), manned/unmanned aerial vehicles (such as aircraft, helicopters, drones, model airplanes or model helicopters, etc.), home/industrial/ It may include, but is not limited to, military robots, industrial/military machines, medical devices, traffic controllers, electronic billboards, or automated teller machines (ATMs). Designers or users may consider and employ at least one of various devices capable of processing and controlling information in addition to the above-described information processing device as the above-described audio processing device 100 according to circumstances or conditions.

Hereinafter, an embodiment of a voice processing system will be described with reference to FIG. 10 .

10 is a diagram of one embodiment of a voice processing system.

As shown in FIG. 10, the voice processing system 1 performs signature processing on the recording terminal device 10 capable of performing voice recording and summary information (p-digest), and the signature processing It may include a server device 20 capable of storing summary information (sp-digest), receive sensitive information processing voice data (p-audio), etc. as needed, and use a playback terminal device 30 to reproduce it. can include more.

The recording terminal device 10, the server device 20, and the playback terminal device 30 may transmit data, programs, or commands through the wired or wireless communication network 2 either in one direction or in both directions.

Here, the wireless communication network 2 may include at least one of a short-range communication network and a mobile communication network. Here, the short-range communication network is built with WIFI, WIFI direct, or Bluetooth. The mobile communication network may be built based on a communication standard of 3GPP series (LTE or NR, etc.), 3GPP2 series, or IEEE series.

As described above, the recording terminal device 10 and the playback terminal device 30 may include, for example, an information processing device such as a smart phone or a desktop computer, and the server device 20 may include server hardware. devices and the like.

According to an embodiment, at least one of the recording terminal device 10 and the server device 20 performs at least one operation among a plurality of operations that can be performed by the processor 200 of the voice processing device 100 described above. may have been set up to do so.

For example, the recording terminal device 10 performs the operations and functions of the voice processing device 100 as described above to perform voice conversion on the voice data to be processed to obtain converted voice data; Hash redaction processing is performed on the original voice data or converted voice data, summary information (p-digest) obtained as a result of the hash redaction processing is generated, and original parameter information (o-profile) or parameter information (p- profile) and/or acquire sensitive information processing voice data (p-audio). In this case, the server device 20 may perform the operation of the signature processing device 500 described above. That is, the server device 20 performs signature processing on the summary information (p-digest), and then sends the signed summary information (sp-digest) to the recording terminal device 10 and the playback terminal device 30. At least one may be transmitted. For another example, the recording terminal device 10 acquires original voice data through recording, further obtains converted voice data as necessary, and transmits the converted voice data to the server device 20, and the server device 20 Recorded voice data or converted voice data is received from the recording terminal device 10, hashing redaction processing on the recorded voice data or converted voice data, acquisition of summary information (p-digest), or signature Performs acquisition of summarized information (sp-digest), generation of original parameter information (o-profile) or parameter information (p-profile), and/or acquisition of sensitive information processing voice data (p-audio) And, at least one of parameter information (o-profile, p-profile), summary information (p-digest, sp-digest) and sensitive information processing voice data (p-audio) obtained in the process is recorded by a terminal device ( 10) or the playback terminal device 20. In this case, the recording terminal device 10 may be arranged not to perform an operation performed by the server device 20 among the operations of the processor 200 described above. In addition to the above, processing operations to be performed by the recording terminal device 10 and the server device 20 may be defined in various ways.

The playback terminal device 30 may perform the operation of the voice restoration device 300 described above. In this case, the playback terminal device 30 receives the sensitive information processing audio data (p-audio) from at least one of the recording terminal device 10 and the server device 20, and the sensitive information processing audio data (p-audio). data (that is, at least one of parameter information (o-profile, p-profile) and summary information (p-digest, sp-digest)) required for audio) is received at the same time or at this time, and based on the received data It can restore and reproduce sensitive information processing voice data (p-audio), and output it visually or audibly.

Since the above-described examples of the recording terminal device 10, the server device 20, and the playback terminal device 30 and specific operations of each have already been described, detailed descriptions thereof will be omitted.

Hereinafter, an embodiment of a voice processing method will be described with reference to FIG. 11 .

11 is a flowchart of an embodiment of a voice processing method.

Referring to FIG. 11 , original voice data may be first obtained for voice processing (400). Acquisition of original voice data may be performed through a voice input unit such as a microphone provided in the voice processing device, or may be received and performed from a device other than the voice processing device (eg, a smartphone or a recorder). there is.

Subsequently, pre-processing may be further performed on the original voice data, if necessary (402). Preprocessing may include digital conversion of an analog signal, Fourier transform, fast Fourier transform, or data noise removal. Here, Fourier transform or fast Fourier transform may be performed in units of frames. A frame may be used as a minimum unit in voice processing.

Voice data may be separated into at least one segment (404). Each segment may be divided and acquired in correspondence with each speaker's utterance, and may include, for example, a plurality of sentence(s) or word(s).

At least one segment may be obtained by dividing, for example, a part (s) in which voice is present or voice is continuously connected to some extent based on a region (silent segment) in which voice is not present at all or extremely absent (silent segment). may be

After each segment is separated and acquired, voice conversion for each segment may be performed using at least one voice conversion parameter (406). Here, the voice conversion parameter is a value used for voice conversion and may be determined logically or experimentally.

For example, the voice conversion parameter may be larger than the above-mentioned separation frequency but smaller than pi (π). A voice conversion parameter may be determined for each segment, and in this case, the voice conversion parameter may be randomly determined for each segment. If necessary, it is also possible to further determine a voice conversion parameter (not shown) for the silent segment.

The voice conversion described above may be performed by applying voice conversion parameters to each frame of a segment. The same voice conversion parameters are applied to frames of the same segment, and different voice conversion parameters are applied to frames belonging to different segments. and may be performed. According to one embodiment, voice conversion may be performed using a warping function, for example, based on Equation 1 described above.

At the same time as the voice conversion process 406, hash redaction processing and/or sensitive information processing may be performed before or after the voice conversion process 406 (408).

Hash redaction processing may be performed on each piece of audio data of each segment, or may be performed on each frame of audio data. Hash redaction can also be performed using Equation 2 above.

Sensitive information processing may be performed by replacing all or part of converted or unconverted voice data with data subjected to hash redaction while corresponding to all or part of the corresponding voice data.

Here, all or part of the converted or unconverted voice data may include sensitive information. Sensitive information, for example, the name of the interlocutor(s), phone number, address, resident registration number, social security number, passport number, password, information corresponding to business secrets or personal privacy, etc. May contain information that you do not wish to include. As sensitive information is replaced with hash-redacted data, sensitive information processing voice data can finally be obtained.

In addition, parameter information and summary information may be obtained (410). Acquisition of parameter information may also be performed separately from acquisition of summary information immediately after the above-described parameter acquisition process 406 . Parameter information may be generated and obtained using at least one voice conversion parameter, and may be generated and obtained by further using at least one hash redaction result parameter as needed.

Here, as for at least the parameter information, a hash redaction result parameter is arranged instead of a corresponding voice conversion parameter in a portion corresponding to a segment where original voice restoration is not allowed or impossible (for example, a conversation counterpart's voice portion), and the original voice is restored. It may be generated by arranging a corresponding voice conversion parameter in a part (eg, a user's voice part) corresponding to the permitted or possible segment.

Summary information may be obtained by combining hash redaction processing results corresponding to each segment, that is, hash redaction processed voice data. In this case, summary information may be generated by sequentially arranging hash redaction processing results corresponding to the order of segments.

Sensitive information processing voice data, parameter information, and/or summary information may be stored and may be transferred to another device (a server device or a voice restoration device) according to an embodiment (412).

The summary information may be transmitted to the signature processing device, if necessary, and the signature processing device obtains the signed summary information by performing signature processing on the summary information and delivers it to at least one of the voice processing device and the voice restoration device. can

Hereinafter, an embodiment of a voice verification method will be described with reference to FIG. 12 .

12 is a flow chart of one embodiment of a voice verification method.

The voice verification method may be performed by a voice verification device such as a smart phone or a desktop computer.

According to an embodiment, as shown in FIG. 12, for voice verification, the voice verification device first transfers sensitive information processing voice data and summary information to a voice processing device or another device (eg, a server device) that stores these information. etc. (420).

As described above, sensitive information-processed voice data may include voice data from which sensitive information has been removed according to voice conversion and hash redaction processing on a portion corresponding to sensitive information.

The summary information may include information obtained by combining results of hash redaction processing on all or part of voice data, and may be signed according to embodiments.

Sequentially, if the sensitive information processing voice data is encoded in a predetermined format (eg, an advanced audio coding (AAC) format), decoding of the sensitive information processing voice data may be performed (422). If sensitive information processing voice data is not pre-encoded, the decoding process may be omitted.

Hash redaction processing is performed on all or part of sensitive information processing voice data (424). In this case, the hash redaction processing applied to the sensitive information processing voice data may be the same as the hash redaction processing performed by the voice processing device on the voice data.

On the other hand, additional hash redaction processing is not performed on the portion of sensitive information-processed voice data previously subjected to hash redaction processing by the voice processing device, and hash redaction processing is not performed on the other portions (ie, the portion corresponding to the voice). Action processing may be performed.

Sequentially, a result of hash redaction processing on sensitive information processing voice data (ie, a result of hash redaction processing for verification) and summary information (or signed summary information) may be compared (426). As described above, since hash redaction processing is performed in the same way for the same segment as when generating summary information for at least one segment of voice data, hash of summary information or signed summary information in the absence of forgery and falsification. The result of redaction processing and the result of hash redaction processing for verification are inevitably the same.

Therefore, if the hash redaction processing result for verification and the hash redaction processing result recorded in the summary information are the same (Yes in 428), it is determined that the sensitive information processing voice data is not forged and falsified, and its authenticity is recognized. (430).

Conversely, if the hash redaction processing result for verification is different from the hash redaction processing result recorded in the summary information (No in 428), the corresponding sensitive information processing voice data is determined to be forged or altered (432).

Accordingly, the voice verification device can determine the authenticity of voice data recorded or stored in the voice processing device and transmitted from the voice processing device.

Hereinafter, an embodiment of a voice restoration method will be described with reference to FIG. 13 .

13 is a flowchart of an embodiment of a voice restoration method.

As shown in FIG. 13 , the voice restoration apparatus may first receive sensitive information processing voice data and parameter information from the voice processing apparatus for voice restoration (440). Sensitive information processing voice data is voice data on which hash redaction processing has been performed on sensitive information as described above, and parameter information is data obtained by combining parameters used for each segment, and includes voice conversion parameters and hash redaction. It may be prepared by including at least one of the result parameters.

In this case, a speech conversion parameter is provided for a segment in which original speech can be restored, and a hash redaction result parameter is provided for a segment in which original speech cannot be restored.

Depending on the embodiment, if the sensitive information processing voice data is pre-encoded in a predetermined format (eg, an advanced audio coding (AAC) format), decoding of the sensitive information processing voice data may be further performed. Yes (442).

If sensitive information processing voice data is not encoded in a predetermined format, the decoding process may be omitted.

A voice conversion parameter corresponding to each segment of the sensitive information processing voice data may be retrieved from the parameter information (444).

If a voice conversion parameter is detected for a specific segment (YES in 446), the voice of the corresponding segment is restored using the detected voice conversion parameter (448).

Conversely, if the voice conversion parameter is not detected for a specific segment (ie, if only the hash redaction result parameter exists for the specific segment) (No in 446), the segment is restored as it is and/or predetermined according to predefined settings. It can be converted into a beep sound and restored (450).

Accordingly, when voice is restored, non-sensitive information is restored to be the same as or close to the original voice, but sensitive information is masked and restored as a beep sound. be able to prevent

The voice processing method according to the above-described embodiment may be implemented in the form of a program that can be driven by a computer device. The program may include commands, libraries, data files, and/or data structures alone or in combination, and may be designed and manufactured using machine language codes or high-level language codes.

The program may be specially designed to implement the above-described method, or may be implemented using various functions or definitions known and usable to those skilled in the art in the field of computer software.

Also, here, the computer device may be implemented by including a processor or a memory capable of realizing program functions, and may further include a communication device as needed.

A program for implementing the above-described audio processing method may be recorded on a recording medium readable by a device such as a computer. A computer-readable recording medium is, for example, a semiconductor storage medium such as ROM, RAM, SD card or flash memory (for example, a solid state drive (SSD), etc.), or a magnetic disk storage such as a hard disk or a floppy disk. At least one program capable of temporarily or non-temporarily storing one or more programs executed by a device such as a computer, such as a medium, an optical recording medium such as a compact disk or DVD, or a magneto-optical recording medium such as a floptical disk, etc. type of physical storage medium.

Although various embodiments of a voice processing device, a voice restoration device, a voice processing system, a voice processing method, a voice restoration method, and a voice verification method have been described above, a voice processing device, a voice restoration device, a voice processing system, a voice processing method, and a voice verification method have been described. The restoration method or the voice verification method is not limited to the above-described embodiment.

Various other systems, devices or methods that can be implemented by those skilled in the art by modifying and transforming based on the above-described embodiments are also the above-described voice processing device, voice restoration device, voice processing system, voice processing method, It may be an embodiment of a voice restoration method or a voice verification method. For example, the described method(s) may be performed in an order different from that described, and/or component(s) of a described system, structure, device, circuit, etc. may be coupled, connected, or otherwise used in a manner other than described. Even when combined or substituted or substituted by other components or equivalents, it may be an embodiment of the above-described voice processing device, voice restoration device, voice processing system, voice processing method, voice restoration method, and/or voice verification method.

10: terminal device for recording 20: server device
30: terminal device for playback 100: audio processing device
200: processor 210: pre-processing unit
220: segment separation unit 230: parameter acquisition unit
231: parameter information acquisition unit 240: voice conversion unit
250: hash redaction processing unit 251: sensitive information processing unit
260: summary information generator 270: encoding unit
300: voice restoration device 400: processor
401: decoding unit 410: verification unit
420: sound source restoration unit

Claims (6)

Acquiring sensitive information processing voice data obtained by hash redaction processing of sensitive information with respect to voice data, and parameter information including at least one of a voice conversion parameter and a hash redaction result parameter, wherein the hash redaction result parameter is an input unit obtained by performing hash redaction processing on a voice conversion parameter; and
a processor for restoring a part corresponding to the voice conversion parameter among the sensitive information processing voice data using the voice conversion parameter, and restoring a part corresponding to the hash redaction result parameter as it is or replacing a beep; Voice restoration device comprising a. According to claim 1,
The input unit further obtains summary information, wherein the summary information is obtained by performing hash redaction processing on all or some segments of the original voice data to obtain a hash redaction processing result, and combining the hash redaction acquisition results. voice restoration device. According to claim 2,
The processor performs hash redaction processing on the sensitive information processing voice data to obtain a hash redaction processing result for verification, and performs verification of the voice data by comparing the hash redaction processing result for verification and summary information. voice restoration device. obtaining sensitive information-processed voice data obtained by performing hash redaction processing on sensitive information with respect to voice data;
obtaining parameter information including at least one of a speech conversion parameter and a hash redaction result parameter; and
Restoring a part corresponding to the voice conversion parameter among the sensitive information processing voice data using the voice conversion parameter, and restoring the part corresponding to the hash redaction result parameter as it is or replacing a beep sound; Voice restoration method comprising a. According to claim 4,
The hash redaction result parameter is obtained by performing hash redaction processing on the voice conversion parameter. obtaining sensitive information-processed voice data obtained by performing hash redaction processing on sensitive information with respect to voice data;
Obtaining summary information, wherein the summary information is obtained by performing hash redaction processing on all or some segments of the original speech data to obtain a hash redaction processing result, and combining the hash redaction acquisition results. ;
obtaining a hash redaction processing result for verification by performing hash redaction processing on the sensitive information processing voice data; and
and verifying the voice data by comparing a result of hash redaction processing for verification and summary information.