JP2018155936A - Sound data edition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To synchronize two or more sound data throughout an entire period.SOLUTION: A sound data edition method includes a step of synchronizing first sound data based on a first demonstration demonstrated during reproduction of a reference sound and second sound data based on a second demonstration different from the first demonstration demonstrated during reproduction of the reference sound by using the reference sound.SELECTED DRAWING: Figure 7

Description

  The present invention relates to a technique for editing a plurality of sound data.

  A system for performing a concert by a plurality of performers who are geographically separated is known. For example, Patent Document 1 describes a system in which a corresponding terminal device generates performance data of each performance part in a performance session started with a synchronization signal as a reference. In this system, each performance data is corrected based on the synchronization signal.

JP 2014-153515 A

In the technique described in Patent Document 1, two or more sound data cannot be synchronized over the entire period after the timing at which synchronization is started.
In contrast, the present invention provides a technique for synchronizing two or more sound data over the entire period.

  The present invention provides a first sound data based on a first demonstration performed during reproduction of a reference sound and a second demonstration based on a second demonstration different from the first demonstration performed during reproduction of the reference sound. There is provided a sound data editing method including a step of synchronizing sound data using the reference sound.

  In the sound data editing method, the first sound data and the second sound data include data of a first channel and a second channel, respectively, the data of the first channel represents the reference sound, and the first sound data The 2-channel data may represent the sound of the demonstration.

  The sound data editing method includes a step of obtaining an input sound signal representing an input sound, and a step of adjusting at least one tempo of the first sound data and the second sound data based on the input sound signal. You may have.

  The sound data editing method includes a step of adjusting a tempo of the first sound data according to the reference sound, and a step of adjusting a tempo of the second sound data according to the reference sound. In the step of performing, the first sound data and the second sound data whose tempo is adjusted may be synchronized.

  In the synchronizing step, the first sound data is sound data included in the first moving image data, and the second sound data is sound data included in second moving image data different from the first moving image data. The first moving image data and the second moving image data may be synchronized.

  According to the present invention, two or more sound data can be synchronized over the entire period.

1 is a diagram illustrating a functional configuration of a sound data editing system 1 according to a first embodiment. The figure which illustrates the hardware constitutions of the user terminal. The figure which illustrates the hardware constitutions of the server. 6 is a sequence chart illustrating an outline of the operation of the sound data editing system 1. The sequence chart which illustrates the operation | movement which concerns on upload of sound data. The figure which illustrates UI screen for inputting the reproduction | regeneration instruction | indication of a reference sound. The sequence chart which illustrates the operation | movement which concerns on the synchronization of sound data. The figure which illustrates UI screen for selecting the performance animation synchronized. The figure which illustrates arrangement | positioning of each performance animation in a synthetic animation. The figure which illustrates functional composition of sound data editing system 3 concerning a 3rd embodiment. The conceptual diagram of a reverse alignment process. The UI screen for inputting the reproduction | regeneration instruction | indication of a reference sound in 3rd Embodiment. The flowchart which illustrates a reverse alignment process. The figure which illustrates functional composition of sound data editing system 4 concerning a 4th embodiment. The figure which illustrates the function structure of the alignment part. The flowchart which illustrates the operation | movement which concerns on the alignment reproduction | regeneration of sound data.

1. First embodiment 1-1. Configuration FIG. 1 is a diagram illustrating a functional configuration of a sound data editing system 1 according to the first embodiment. The sound data editing system 1 provides a service (hereinafter referred to as “sound data editing service”) that synchronizes sound data of a plurality of demonstrations performed while reproducing the same reference sound. Here, the demonstration means that an object accompanied by the generation of a sound that changes with time is performed in a dramatic manner, dancing, performing, singing, speaking, recitation, or performing by other methods. The target object represents an idea, emotion, or fact, and for example, belongs to the field of music, literary arts, performing arts, or academics. As an example, the demonstration here is a performance of music using a musical instrument, and the sound data is performance sound data. In this example, the sound data editing system 1 synchronizes the first sound data and the second sound data. The first sound data is data obtained by converting (accompanied) sound (an example of the first sound) based on the first performance. The second sound data is data obtained by converting (accompanied) voice (an example of the second voice) based on the second performance. Both the first demonstration and the second demonstration are demonstrations performed while reproducing the same reference sound. Here, “performed during playback of the same reference sound” means that the reference sound reproduced when each performance is performed is the same, and the first performance and the second performance are always performed simultaneously. It doesn't mean that. In addition, “the same reference sound” does not mean that the two reference sounds are completely the same. For example, a performance sound recorded by the same player on the same music is recorded. They may be different in time (so-called take differences), or may be recorded performance sounds of different players performing the same music. The first demonstration is performed by the user A (an example of the first user), and the second demonstration is performed by the user B (an example of the second user).

  The sound data editing system 1 includes a user terminal 10 and a server 20. The user terminal 10 is a client in the sound data editing service and provides a user interface. Here, two user terminals 10 are illustrated. When the user terminal of user A and the user terminal of user B and their elements are distinguished, subscripts are used as in the user terminal 10A and the user terminal 10B.

  The user terminal 10 includes a storage unit 11, a playback unit 12, a recording unit 13, a communication unit 14, and a UI unit 15. The storage unit 11 stores various data. In this example, the storage unit 11 stores data Dr for reproducing the reference sound. The reproduction unit 12 reproduces various data. In this example, the reproducing unit 12 reproduces the data Dr. The recording unit 13 converts the sound accompanying the performance performed during the reproduction of the reference sound into data, and stores it as sound data in the storage unit 11. The communication unit 14 communicates with other devices according to a predetermined communication standard. In this example, the communication unit 14 uploads the sound data of the demonstration to the server 20. The UI unit 15 provides a user interface for the user to input various instructions such as reference sound reproduction, recording start, and upload.

  The server 20 is a server in the sound data editing service. The server 20 communicates with the user terminal 10 via a network (not shown) such as the Internet. The server 20 includes a communication unit 21, a storage unit 23, and a synchronization unit 24. The communication unit 21 communicates with other devices according to a predetermined communication standard. In this example, the communication unit 21 receives sound data transmitted from the user terminal 10. The storage unit 23 stores sound data received by the communication unit 21. The synchronization unit 24 synchronizes two or more sound data and synthesizes them. The synthesized sound data is referred to as “synthesized sound data”. Here, “synthesis” of sound data refers to a process of mixing two or more sounds (mixing).

  The communication unit 21 of the server 20 distributes the synthesized sound data generated by the synchronization unit 24 to the user terminal 10. In the user terminal 10, the playback unit 12 plays back the synthesized sound data distributed from the server 20.

  FIG. 2 is a diagram illustrating a hardware configuration of the user terminal 10. The user terminal 10 includes a CPU (Central Processing Unit) 101, a memory 102, a storage 103, a communication IF 104, a display 105, an input device 106, a microphone 107, a speaker 108, and a camera 109, specifically, for example, a smartphone, A tablet terminal or a personal computer. The CPU 101 is a control device that executes a program and controls other hardware elements of the user terminal 10. The memory 102 is a main storage device that functions as a work area when the CPU 101 executes a program, and includes, for example, a RAM (Random Access Memory). The storage 103 is a nonvolatile auxiliary storage device that stores various programs and data, and includes, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive). The communication IF 104 is an interface for communicating with other devices, and includes, for example, a NIC (Network Interface Card). The display 105 is a display device that displays information, and includes, for example, an LCD (Liquid Crystal Display). The input device 106 is a device for the user to input instructions or information to the user terminal 10, and includes, for example, a touch sensor or a keyboard. The microphone 107 is a device that collects sound and converts the collected sound into an electrical signal. The speaker 108 is a device that outputs sound according to an electrical signal. The camera 109 is a photographing device for photographing a moving image.

  In this example, the storage 103 stores a program (hereinafter referred to as “client program”) for causing the computer device to function as a client in the sound data editing system 1. In a state where the CPU 101 is executing the client program, at least one of the memory 102 and the storage 103 is an example of the storage unit 11. The speaker 108 is an example of the playback unit 12. The microphone 107 and the camera 109 are examples of the recording unit 13. The display 105 and the input device 106 are examples of the UI unit 15. The communication IF 104 is an example of the communication unit 14.

  FIG. 3 is a diagram illustrating a hardware configuration of the server 20. The server 20 is a computer device having a CPU 201, a memory 202, a storage 203, and a communication IF 204. The CPU 201 is a control device that executes a program and controls other hardware elements of the server 20. The memory 202 is a main storage device that functions as a work area when the CPU 201 executes a program, and includes, for example, a RAM. The storage 203 is a nonvolatile auxiliary storage device that stores various programs and data, and includes, for example, an SSD or an HDD. The communication IF 204 is an interface for communicating with other devices, and includes, for example, a NIC.

  In this example, the storage 203 stores a program (hereinafter referred to as “server program”) for causing the computer device to function as a server in the sound data editing system 1. A communication IF 204 controlled by the CPU 201 executing the server program is an example of the communication unit 21. The CPU 201 executing the server program is an example of the synchronization unit 24. At least one of the memory 202 and the storage 203 controlled by the CPU 201 executing the server program is an example of the storage unit 23.

1-2. Operation FIG. 4 is a sequence chart illustrating an outline of the operation of the sound data editing system 1. In the following example, the sound data is audio data (audio track) included in data of a moving image (hereinafter referred to as “performance moving image”) that records a performance related to the performance of the instrument. Here, an example will be described in which data D [1] (an example of first moving image data), which is moving image data, and data D [2] (an example of second moving image data) are synchronized.

  In step S1, the user terminal 10 records moving image data. In step S <b> 2, the user terminal 10 uploads the moving image data to the server 20. Although not shown, a plurality of user terminals 10 can access the server 20, and a plurality of moving image data is uploaded to the server 20 from the plurality of user terminals 10. In this example, the plurality of moving image data includes data D [1] and data D [2]. The server 20 stores the plurality of moving image data.

  In step S <b> 3, the user terminal 10 transmits a synchronization request for synchronizing a plurality of moving image data to the server 20. This synchronization request includes information for specifying moving image data to be subjected to synchronization processing. In this example, moving image data to be subjected to synchronization processing includes data D [1] and data D [2]. In step S4, the server 20 synchronizes a plurality of moving image data designated by the synchronization request. Data generated by the synchronization processing is called synchronization data. In step S <b> 5, the server 20 transmits the synchronization data to the user terminal 10. In step S6, the user terminal 10 reproduces the synchronized data. Details of these processes will be described below. In the following description, functional elements such as the UI unit 15 are described as processing subjects. This means that a hardware element such as the CPU 101 that executes software such as a client program executes processing using other hardware elements. means.

1-2-1. Uploading Sound Data FIG. 5 is a flowchart illustrating an operation related to uploading sound data. The flow in FIG. 5 is started when, for example, the user terminal 10 instructs the start of recording of a performance (recording of a performance video).

  In step S11, the UI unit 15 accepts selection of a reference sound. The reference sound is a sound indicating the progress of the demonstration (in this example, the performance of the instrument). For example, the sound of the musical piece to be played (for example, a sound taken from a CD or the like) and a part of the musical piece (for example, a drum) (Part) performance sound, or a performance sound obtained by removing a part of the music (for example, vocal part). As described above, the reference sound is not meaningless in the music such as a click sound or a beep sound that is periodically repeated, and refers to a sound that itself constitutes at least a part of the music. At least one reference sound data is stored in at least one of the storage unit 11 and the storage unit 23 (server 20). The UI unit 15 searches the storage unit 11 and the storage unit 23 and generates a list of reference sounds. The UI unit 15 displays a list of reference sounds and prompts the user to select one reference sound. The user selects a desired reference sound from the displayed list. When receiving the selection of the reference sound by the user, the UI unit 15 notifies the reproduction unit 12 of the selected reference sound.

  In step S12, the recording unit 13 starts recording a performance moving image. The recording of the performance video is started, for example, when the user inputs a recording start instruction via the UI unit 15. The performance moving image includes video shot by the camera 109 and sound collected by the microphone 107. The performance movie is recorded in a predetermined data format (for example, a general-purpose movie format).

  In step S13, the playback unit 12 starts playback of the designated reference sound. The reproduction unit 12 reads the designated reference sound data from the storage unit 11 or the storage unit 23 and decodes the read data to generate a sound signal. The playback unit 12 outputs the generated sound signal to the speaker 108. Thus, the reference sound is sequentially output from the speaker 108. In this example, since the reference sound is output from the speaker 108, the sound is recorded with the performance sound superimposed on the reference sound in the sound track of the performance moving image (sound is superimposed instead of video). .

  FIG. 6 is a diagram illustrating a UI screen for inputting a reference sound reproduction instruction. This UI screen includes a window 501, a button 502, and a window 505. A window 501 is an area for displaying identification information (for example, a title) of a reference sound to be reproduced. A button 502 is a button for instructing reproduction or pause of the reference sound. When the button 502 is pressed with the reproduction sound stopped, reproduction of the reference sound is started. When the button 502 is pressed while the reproduction sound is being reproduced, the reproduction of the reference sound is paused.

  This UI screen further includes a button 506. A button 506 is a button for stopping the recording of the performance video. During playback of the reference sound, the user plays the instrument in accordance with the reference sound while listening to the reference sound. The recording unit 13 records a video of a user playing a musical instrument and a performance sound thereof together with a reference sound. While the reference sound is being reproduced, the UI screen of FIG. 6 is displayed, and the user can stop the recording of the performance video at an arbitrary timing.

  Refer to FIG. 5 again. In step S14, the recording unit 13 stops recording the performance moving image. Recording of the performance video stops when the user presses the button 506, for example. The processing in steps S11 to S14 is details of the processing in step S1 in FIG. In step S <b> 2, the communication unit 14 transmits (uploads) the moving image data of the performance moving image generated by the recording unit 13 to the server 20. When the reference sound data is acquired from the storage unit 11 instead of the storage unit 23, the communication unit 14 may transmit the reference sound data to the server 20 in addition to the moving image data. The storage unit 23 of the server 20 stores reference sound data.

  The communication unit 21 of the server 20 receives moving image data from the user terminal 10. The storage unit 23 stores the moving image data received by the communication unit 21. In the storage unit 23, the moving image data is stored in association with attribute information indicating the attribute of the moving image. The attribute information includes, for example, reference sound identification information, performance sound identification information, moving picture creator identification information, and upload date and time. The attribute information corresponding to the moving image data is not limited to this example. For example, the attribute information does not need to include all these pieces of information, and may include only one or a plurality of pieces of information. The reference sound identification information includes, for example, a song title. The performance sound identification information includes, for example, the name of the musical instrument that emitted the performance sound. These pieces of information are input by the user via the UI unit 15, for example.

1-2-2. Sound data synchronization (synthesis)
FIG. 7 is a flowchart illustrating an operation related to the synchronization of sound data (an example of a sound data editing method). The flow in FIG. 7 corresponds to the processing in steps S4 to S5 in FIG. Here, first, regarding step S <b> 3, items related to processing in which the user terminal 10 transmits a synchronization request for synchronizing a plurality of moving image data to the server 20 will be described.

  FIG. 8 is a diagram illustrating a UI screen for selecting a performance video to be synchronized. This UI screen includes a window 601, a text box 602, a button 603, and a button 604. The window 601 is an area for displaying a list of performance videos stored in the storage unit 23. In this example, a thumbnail image of the performance video and a title of the performance video are displayed. A text box 602 is an area for inputting a search key, and a button 603 is a button for instructing a search. When the search is executed, a list of performance videos included in the search result is displayed in the window 601. In the window 601, the performance video selected by the user is displayed separately from the performance video not selected. In this example, the performance movie “I tried playing Sakura Sakura's guitar” and the performance movie “I tried playing Sakura Sakura's bass” are selected. A button 604 is a button for instructing the synchronization of the moving image. When the button 604 is pressed by the user, the communication unit 14 transmits a synchronization request including information specifying a moving image to be synchronized to the server 20 (step S3).

ここで、音信号ｙｒはデータＤｒに含まれる音信号であり、音信号ｙｉはデータＤ［ｉ］に含まれる音データにより示される音信号である。式（１）は、音信号ｙｒ（ｔ）の始点と音信号ｙｉ（ｔ）の始点とを時間領域において一致させてから、音信号ｙｒ（ｔ）に対する音信号ｙｉ（ｔ）の時間差（時間軸上のシフト量）τを変数として両者間の信号波形の相関の程度を示した数値列を示す。 A description will be given with reference to FIG. 7 again. In step S41, the synchronization unit 24 of the server 20 calculates a parameter for synchronizing a plurality of moving image data specified by the synchronization request. This parameter is calculated using the reference sound as a key. Specifically, the synchronization unit 24 calculates a time difference τ that maximizes the cross-correlation Cri between the data Dr (reference sound) and the synchronization target data D [i] according to the following equation (1). In this example, the time difference τ is an example of a parameter for synchronizing a plurality of moving image data.

Here, the sound signal yr is a sound signal included in the data Dr, and the sound signal yi is a sound signal indicated by the sound data included in the data D [i]. Equation (1) is obtained by matching the start point of the sound signal yr (t) and the start point of the sound signal yi (t) in the time domain, and then the time difference (time) of the sound signal yi (t) with respect to the sound signal yr (t). A numerical sequence showing the degree of correlation of signal waveforms between the two using the amount of shift on the axis τ as a variable.

The synchronization unit 24 calculates a time difference τ from the data Dr with respect to all the data D [i] to be synchronized by the equation (1). The positional relationship between the two data D to be synchronized in the time domain is obtained from the time difference from the data Dr. For example, from the time difference τ [r, 1] between the data Dr and data D [1] and the time difference τ [r, 2] between the data Dr and data D [2], the data D [1] and data D [2 ] Is obtained by the following equation (2).

  In step S42, the synchronization unit 24 determines the positional relationship of the images on the screen when synchronizing the images of a plurality of performance videos. In this example, the screen of the synchronized video (hereinafter referred to as “synthetic video”) is divided into n areas (n is a natural number equal to or greater than the number of performance videos to be synchronized). Each of the plurality of areas is displayed in one area. The synchronization unit 24 determines the number n of screen divisions according to a predetermined algorithm, and assigns a performance video to each area.

  FIG. 9 is a diagram illustrating the arrangement of each performance video in the composite video. In this example, the performance video screen is divided into two in the vertical direction. The video “Sakura Sakura I tried playing the guitar” is assigned to the area on the left side of the screen, and the video “Sakura Sakura I played the bass” is assigned to the area on the right.

  Refer to FIG. 7 again. In step S43, the synchronization unit 24 determines the positional relationship of the sound images when synchronizing the sound of the performance video. In this example, the synchronization unit 24 does not perform the process of changing the sound image localization.

  In step S <b> 44, the synchronization unit 24 reads (that is, acquires) moving image data to be subjected to synchronization processing from the storage unit 23. In this example, the synchronization unit 24 reads the performance video data of the user A and the performance video data of the user B (that is, acquires these data). The audio track of the performance video data of the user A is an example of the first sound data of the demonstration performed during the reproduction of the reference sound, and the audio track of the performance video data of the user B is demonstrated during the reproduction of the reference sound. It is an example of the 2nd sound data of another demonstration.

  In step S45, the synchronization unit 24 generates composite moving image data. Details are as follows. The synchronization unit 24 determines the position in the time domain for the plurality of moving image data with the time difference calculated in step 41. The synchronization unit 24 synchronizes the video according to the screen layout determined in step S42, and further generates an audio track according to the sound image position determined in step S43. Steps S41 to S45 are details of step S4 in FIG. In step S5, the communication unit 21 transmits the composite moving image data generated by the synchronization unit 24 to the user terminal 10 that is the transmission source of the synchronization request.

  According to this embodiment, the sound data of the demonstration performed while reproducing the reference sound designated by the user can be synchronized with a simple operation.

1-2-3. Reproduction of sound data The communication unit 14 of the user terminal 10 receives (that is, downloads) synthesized moving image data. The playback unit 12 plays back the downloaded composite video. This process corresponds to the process of step S6 in FIG. A well-known technique is used for reproducing moving image data.

2. Second Embodiment In the first embodiment, all of the sound data to be subjected to the synchronization process is obtained by superimposing the performance sound on the reference sound. When such sound data is synchronized, there is a problem that it is difficult to synchronize only the performance sound except the reference sound. The second embodiment addresses this problem.

  In the second embodiment, the reference sound and the performance sound are separated in the sound data to be synchronized. Specifically, in the stereo 2-channel sound data, the sound signal of the reference sound is recorded in the left channel (an example of the first channel), and the sound signal of the performance sound is recorded in the right channel (an example of the second channel). The An example of a method in which the recording unit 13 separates and records the reference sound and the performance sound in the user terminal 10 is to output the reference sound through the headphones (not shown) without the speaker 108 when the user terminal 10 outputs the reference sound. To output a reference sound. The user performs while listening to the reference sound through the headphones. At this time, the recording unit 13 records the sound signal of the reference sound being reproduced (output) on the left channel, and records the sound signal of the performance sound input via the microphone 107 on the right channel. Another method for recording the reference sound and the performance sound separately is to record the performance sound without using the microphone 107, for example, using so-called line input. In this case, the reference sound may be output via the speaker 108 or may be output via the headphones without passing through the speaker 108.

  In the second embodiment, in step S41, the synchronization unit 24 determines that the sound signal recorded in the left channel of the data D to be synchronized (the sound signal of the reference sound) is the sound signal yi in Expression (1). Used as In the first embodiment, the time difference τ between the data D and the data Dr is calculated using the cross-correlation between the sound signal obtained by superimposing the reference sound and the performance sound and the sound signal of the data Dr (reference sound). In some cases, the time difference τ between the data D and the data Dr cannot be calculated accurately due to the balance between the performance sound and the reference sound in the data D. However, in this example, the time difference τ is calculated using the cross-correlation between the sound signal of the reference sound not including the performance sound in the data D and the sound signal of the data Dr. Therefore, the time difference τ can be calculated more accurately. That is, a plurality of data D can be synchronized more accurately.

  In the second embodiment, in step S45, the sound signal (the sound signal of the performance sound) recorded in the right channel is synthesized from the sound data included in the data D. The sound signal of the left channel (which is the sound signal of the reference sound) is not synthesized. 2. Description of the Related Art Conventionally, a performance video that plays a musical instrument in accordance with a music while playing a music (reference sound) behind is recorded, and this video is published on a video posting site on the Internet. Furthermore, the performance videos uploaded by other people are combined with their performance videos to create a video as if the two are performing together, and this is also made public. However, in these performance moving images, only a sound in which the reference sound and the performance sound are mixed is recorded, so that there is a problem that the reference sound becomes noise when synthesized with another person's performance moving image. On the other hand, in the present embodiment, the audio track of the synthesized moving image does not include the reference sound but includes only the performance sound. Therefore, when synchronizing a plurality of sound data, the two sound data can be synchronized and reproduced with less noise. Note that specific examples of the first channel for recording the sound signal of the reference sound and the second channel for recording the sound signal of the performance sound are not limited to the above-described embodiments. For example, the sound signal of the performance sound may be recorded on the left channel of the two stereo channels, and the sound signal of the reference sound may be recorded on the right channel. In another example, the sound signal of the reference sound may be recorded in one channel and the sound signal of the performance sound may be recorded in the other channel in an acoustic system having three or more channels (for example, surround 5.1 channel). .

3. Third Embodiment FIG. 10 is a diagram illustrating a functional configuration of a sound data editing system 3 according to a third embodiment. In the sound data editing system 3, the description of items common to the sound data editing system 1 according to the first embodiment is omitted. In the sound data editing system 3, the server 20 has a reverse alignment unit 22. The reverse alignment unit 22 performs reverse alignment processing on the sound data received by the communication unit 21. Although details will be described later, the reverse alignment process refers to a process for normalizing the tempo of the performance sound indicated by the sound data. The storage unit 23 stores the sound data subjected to the reverse alignment process.

  FIG. 11 is a conceptual diagram of the reverse alignment process. Here, the reference sound RA and the performance sound SA included in the performance video of the user A, and the reference sound R0 reproduced at a normal tempo are conceptually shown. Further, the tempo of the reference sound is also illustrated. In this figure, the horizontal axis indicates real time. The dot represented as the reference sound indicates a unit time (for example, a time corresponding to a 128th note) in the score of the reference sound. In the reference sound R0 reproduced at a normal tempo, the dots are uniformly arranged at equal intervals, and the unit time in the score progresses uniformly. That is, the tempo of the reference sound is constant T0 throughout the music. On the other hand, the reference sound RA used by the user A for recording the performance video is reproduced at a tempo T1 that is slower than normal during a part of the music (period D1), and is reproduced at a normal tempo T0 during other periods. It has been done. The reverse alignment process is a process of adjusting the tempo of the performance moving image so as to be synchronized with the reference data (in this example, the reference sound R0 reproduced at a normal tempo) throughout the music. In this example, the performance video of the user A is compressed in the time domain so that the reference sound RA in the time period D1 has the same tempo as the reference sound R0.

  FIG. 12 is a diagram illustrating a UI screen for inputting a reference sound reproduction instruction in the third embodiment. In this example, the playback unit 12 of the user terminal 10 can change the playback speed of the reference sound. This UI screen includes a window 501, a button 502, a button 503, a button 504, and a window 505. The button 503 is a button for increasing the playback speed. A button 504 is a button for slowing the reproduction speed. In this example, the playback unit 12 can change the playback speed of the reference sound. The playback speed is specified as a whole throughout the music of the reference sound, for example. For example, the reproducing unit 12 reproduces the reference sound at a constant speed that is 0.8 times the normal speed. Alternatively, the playback unit 12 plays back only a part of the reference sound specified in advance at a speed different from that of the other parts (for example, from the 35th bar to the 42nd bar at a normal 0.8 times speed, Other parts play at normal speed). In yet another example, the playback unit 12 may dynamically change the playback speed according to an instruction input during playback of the reference sound. For example, when the button 503 is pressed during playback of the reference sound, the playback unit 12 increases the subsequent playback speed.

  The sound data uploaded from the user terminal 10 is not stored as it is, but is stored in the storage unit 23 after being subjected to the reverse alignment processing by the reverse alignment unit 22.

  FIG. 13 is a flowchart illustrating the reverse alignment process. In step S221, the reverse alignment unit 22 specifies the time series of the tempo of the reference sound in the performance moving image. There is a known technique for estimating which part of a song is currently being played from the sound of the song being played in real time under the condition that the song is known and the score of the song is given in advance. Yes. This technique includes, for example, the following processing. The reverse alignment unit 22 first obtains a frequency spectrogram by dividing the audio waveform of the reference sound recorded in the left channel of the audio track of the performance moving image into a plurality of periods (frames) and performing constant Q conversion. The reverse alignment unit 22 extracts the onset time (sound generation start time) and the pitch from the frequency spectrogram. The inverse alignment unit 22 sequentially estimates the posterior distribution of the current state using Delayed-decision, and when the peak of the posterior distribution passes through the position regarded as the onset on the score, Laplace approximation and some statistics Output quantity. Specifically, when the reverse alignment unit 22 detects the nth event existing on the music data, the time T [n] at which the event is detected, the average position and variance on the score indicated by the posterior distribution are calculated. Output. The average position on the score is the estimated value of the pronunciation position u [n], and the variance is the estimated value of the observation noise q [n]. The details of the estimation of the pronunciation position are exemplified in, for example, Japanese Patent Application Laid-Open No. 2015-79183.

  The reverse alignment unit 22 determines a real time interval between at least one note before and after a note included in the reference sound (hereinafter referred to as “target note”) as a reference time interval (for example, a music piece of the reference sound). When a performance is performed at a constant tempo, a relative tempo at the time of sounding of the target note is obtained by comparing with a real time interval between the target note and other notes. The reverse alignment unit 22 obtains a tempo time series by performing this process on all the notes included in the reference sound.

  In step S222, the reverse alignment unit 22 sequentially identifies target periods (hereinafter referred to as “target periods”) from among a plurality of periods. In step S223, the reverse alignment unit 22 normalizes the tempo of the target period according to a predetermined standard. In order to keep the pitch (pitch) constant, the tempo adjustment is not simply expanding or reducing the waveform in the time axis direction, but a so-called time stretch technique (sometimes called a time expander or time compressor) Used. As a technique of time stretching, for example, a technique of adjusting a tempo by dividing a waveform of a synthesized sound into a plurality of blocks and arranging the blocks while shifting the positions of the blocks in the time domain is used. In order to reduce noise, a cross fade is used.

  In step S224, the reverse alignment unit 22 determines whether the processing has been completed for all periods included in the reference sound. If it is determined that there is a period in which the process is not completed (S224: NO), the reverse alignment unit 22 proceeds to step S222. When it is determined that the process has been completed for all the periods (S224: YES), the reverse alignment unit 22 ends the process. The reverse alignment unit 22 obtains a performance sound waveform with the tempo adjusted in this way.

  Although the detailed description is omitted here, the reverse alignment unit 22 adjusts the tempo so that the video of the performance video is also synchronized with the performance sound.

  2. Description of the Related Art Conventionally, a performance video that plays a musical instrument in accordance with a music while playing a music (reference sound) behind is recorded, and this video is published on a video posting site on the Internet. In addition, it is also possible to create a video as if the two perform a ensemble by synchronizing their performance video with a performance video uploaded by another person and publish it. However, in these performance moving images, the reference sound used by user A and the reference sound used by user B had to be reproduced at the same tempo. For example, when a performance sound output from a music sequencer is used as the reference sound, the tempo of the music can be arbitrarily set by the user. Even performers who do not play well with the original tempo may play if the tempo is lowered. However, if you change the tempo from the original song, you will not be able to synchronize with other users' videos. On the other hand, in the present embodiment, the performance video is recorded in a state adjusted to a reference tempo. Therefore, it is possible to synchronize with the performance video of another user regardless of the tempo of the reference sound reproduced when the performance video is recorded.

4). Fourth Embodiment FIG. 14 is a diagram illustrating a functional configuration of a sound data editing system 4 according to a fourth embodiment. In the sound data editing system 4, description of matters common to the sound data editing system 1 according to the first embodiment is omitted. In the sound data editing system 4, the user terminal 10 has an alignment unit 16. The alignment unit 16 dynamically adjusts the tempo at the time of reproducing the synthesized sound data according to information input from the outside.

  FIG. 15 is a diagram illustrating a functional configuration of the alignment unit 16. The alignment unit 16 has a function for aligning and reproducing the synthesized sound data. Alignment playback refers to playing back synthesized sound data while dynamically adjusting the timing or tempo according to an input signal (for example, a sound signal of a real-time performance sound of a musical instrument).

  The alignment unit 16 includes an input unit 161, an estimation unit 162, a prediction unit 163, and an output unit 164. The input unit 161 receives an input signal. The estimation unit 162 analyzes the input signal and estimates which position on the score is being played. Note that it is known which music is played by the input signal. Data indicating the musical score of the music is stored in the storage unit 11. The prediction unit 163 uses the estimated value supplied from the estimation unit 162 as an observation value to predict the next reproduction timing of the synthesized sound data. The output unit 164 outputs, to the playback unit 12, a playback command for a period to be sounded next, according to the predicted time input from the prediction unit 163.

  FIG. 16 is a flowchart illustrating an operation related to synchronized reproduction of synchronized sound data. The flow in FIG. 16 is started when, for example, the user instructs the start of alignment reproduction at the user terminal 10. In step S61, the input unit 161 starts accepting an input sound. That is, the input unit 161 acquires an input sound signal.

  In step S62, the estimation unit 162 estimates the score position. In step S63, the prediction unit 163 predicts the performance tempo. The performance tempo is predicted from the transition of the musical score position with respect to real time. In step S64, the prediction unit 163 predicts the next reproduction timing. The output unit 164 outputs to the playback unit 12 a playback command for the next period to be sounded according to the predicted time input from the prediction unit 163 (step S65). The processing of steps S62 to S64 is repeatedly executed periodically.

  The synthesized sound data is divided into a plurality of periods. The division into a plurality of periods is performed based on the event on the score of the corresponding reference sound. Specifically, the synthesized sound data is divided at positions corresponding to the bars of the reference sound, for example. The reproduction command output from the output unit 164 includes information for designating a period and a tempo to be reproduced in the synthesized sound data. The playback unit 12 plays back a specified period at a specified tempo. For example, a time stretch technique is used to adjust the tempo.

  According to this example, for example, a synthesized sound (synthetic moving image) can be reproduced in synchronization with a real-time musical instrument performance. For example, even when a tempo fluctuation or timing shift caused by music expression or performance technique occurs in a real-time instrument performance, the synthesized sound is reproduced following the real-time instrument performance. According to this example, it is possible to have a simulated ensemble experience in real time with your own musical instrument performance with synthetic sounds as accompaniment.

5. The present invention is not limited to the first to fifth embodiments described above, and various modifications can be made. Two or more of the above embodiments may be used in combination. In addition, at least one of the following modifications may be applied to each of the above-described embodiments or a combination of two or more.

5-1. Modification 1
In step S43, the synchronization unit 24 may adjust the localization of the sound signal included in the data D to be synchronized. For example, the synchronization unit 24 determines the sound image position of each performance sound according to a predetermined algorithm. Alternatively, the synchronization unit 24 selects each performance sound according to a combination of sound image positions selected by a user from templates prepared in advance (for example, left for guitar, right for keyboard, bass, drum, and vocal for center). Determine the position of the sound image. Further alternatively, the synchronization unit 24 determines the sound image position of each performance sound in accordance with a user instruction.

5-2. Modification 2
The synchronization processing in the synchronization unit 24 is not limited to that exemplified in the embodiment. A specific method for calculating the time difference τ, which is a parameter for synchronizing two pieces of data D, is not limited to the one using Expression (1) or (2). For example, a time difference that maximizes another feature such as MFCC (Mel-Frequency Cepstrum Coefficients) or PCP (Pitch Class Profile) is used as a parameter for synchronizing the two data D. Also good. In another example, the synchronization unit 24 may adjust the relative volume of each performance sound. For example, the volume is adjusted according to the attribute of the performance video. The attribute of the performance video is, for example, the attribute of the user who created the performance video (performer in the performance video). Specifically, when the proficiency level of the performer is lower than that of other users, the performance sound of the performance moving image is synthesized at a lower volume than that of other users. In another example, the performance movie attribute is an attribute of the instrument being played. Specifically, the performance sound of a specific musical instrument is synthesized at a lower volume than other musical instruments.

5-3. Modification 3
The synchronization unit 24 may replace the performance sound with other sound data when synchronizing the performance video of the user A with the performance video of the user B. The sound data used for replacement is obtained by, for example, extracting the parts included in the performance video from the sound data of the original music to be played. For example, if user A's performance video is a performance of a guitar part of a song, the sound of the guitar part is extracted from the sound data of the original song, and this sound is replaced with the performance sound of user A. It is done. According to this example, it is possible to make a so-called hit in a synthesized moving image.

5-4. Modification 4
The synchronization unit 24 may determine the sound image position in the synthesized moving image according to the arrangement of the video. For example, if you synchronize four performance videos of vocal, guitar, bass, and drum parts, the vocal image is on the front of the screen, the guitar image is on the right side of the screen, and the bass image is on the left side of the screen. When the drum images are arranged at the back of the screen, the sound image of the vocal may be in front, the sound image of the guitar on the right, the sound image of the bass on the left, and the sound image of the drum on the back. According to this example, it is possible to provide a synthesized moving image in which the positional relationship between video and sound images is matched.

5-5. Modification 5
Data that is synthesized in synchronism using the reference sound is not limited to video data and sound data. For example, a control signal used for video or audio synthesis (mixing), a video switching control signal, a control signal for controlling generation of an image object in CG, or a stage lighting control signal may be synthesized in synchronism. .

5-6. Modification 6
The sound data editing system 1 may provide an SNS (Social Networking Service) for providing a performance video as a synchronized material. In SNS, users are grouped according to, for example, performance level (level), music preference, performance tendency, instrument type, or phrase to be played.

5-7. Modification 7
The sharing of functions in the user terminal 10 and the server 20 is not limited to that illustrated in FIG. 1, FIG. 10, or FIG. 1, 10, or 14, some of the functions implemented in the user terminal 10 may be implemented in the server 20, or some of the functions implemented in the server 20 may be implemented in the user terminal 10. Also good. As an example, the user terminal 10 may have a function corresponding to the reverse alignment unit 22. In this case, the user terminal 10 performs reverse alignment processing on the performance moving image, and transmits the performance moving image data whose tempo is standardized to the server 20. In another example, the user terminal 10 may have a function corresponding to the synchronization unit 24. In this case, the user terminal 10 acquires a plurality of performance videos to be synchronized and synthesizes them. In yet another example, the server 20 may have a function corresponding to the alignment unit 16. In this case, the server 20 performs a process of acquiring the input sound via the user terminal 10 and reproducing the synthesized sound according to the input sound.

5-8. Modification 8
Some of the functions of the sound data editing systems 1 to 4 may be omitted. For example, the sound data editing system 1 may not have the reverse alignment unit 22. In this case, the performance video recorded in the user terminal 10 is stored in the server 20 as it is without being subjected to the reverse alignment process. If the reverse alignment process is not performed, a performance video recorded while reproducing a part of the reference sound in the time domain at a different tempo cannot be synchronized with another performance video.

  Even if the reverse alignment unit 22 is not provided, the performance sound tempo can be adjusted by using, for example, a time stretch technique for a performance video recorded while reproducing the reference sound at a constant tempo as a whole. In this case, information for specifying the tempo of the reference sound is provided from the user terminal 10, for example. If there is a restriction that the reference sound is limited to one that is reproduced at a constant tempo as a whole, the reference sound can be synchronized with other performance moving images without performing the reverse alignment process.

  Alternatively, when the sound data editing system 1 synchronizes, for example, the performance sound of the user A and the performance sound of the user B, the sound data editing system 1 uses either the time axis expansion or the time of the sound signal of either of them while changing the expansion ratio or the compression ratio. The relative tempo difference between user A and user B is specified by axial compression and calculating the cross-correlation coefficient between the two and specifying the expansion or compression ratio that maximizes the cross-correlation coefficient. May be. If the tempo difference is specified, the tempo of the performance sound of the user A and the performance sound of the user B can be matched using the time stretch technique.

5-9. Other Modifications The format of moving image data described in the embodiment and data associated therewith are merely examples. For example, the moving image data may have a data format unique to the sound data editing system 1. The moving image data may not include attribute information such as instrument identification information. In the embodiment, an example in which the reference sound is viewed via headphones and the performance sound and the reference sound are not mixed has been described. However, the reference sound may be output from the speaker 108, and the performance sound may be mixed with the reference sound. Even in such a case, the audio track of the performance video contains an audio channel that contains only the reference sound that was played back when the performance sound was recorded. The reference sound can be removed by a method such as subtracting a signal having a phase opposite to that of the channel in which only the reference sound is recorded from the above signal.

  The sound data to be synchronized is not limited to the performance sound of the musical instrument. As long as the sound is recorded with the sound generated along with the performance, the sound data to be synchronized may be any data. As already described, “demonstration” includes, for example, performance, singing, reading, and oral performance.

  The hardware configuration for realizing the functions of the sound data editing system 1 is not limited to those illustrated in FIGS. 2 and 3. The sound data editing system 1 may have any hardware configuration as long as the required function can be realized. In addition, the correspondence relationship between functions and hardware is not limited to that illustrated in the embodiment. For example, the functions implemented in the server 20 in the embodiment may be distributed and implemented in two or more devices.

  The program executed in the user terminal 10 and the server 20 may be provided by a storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, or may be downloaded via a communication line such as the Internet. In addition, this program does not have to include all the steps shown in FIGS. 4, 5, 7, and 13. Some of these steps may be omitted.

DESCRIPTION OF SYMBOLS 1 ... Sound data editing system, 3 ... Sound data editing system, 4 ... Sound data editing system, 10 ... User terminal, 11 ... Memory | storage part, 12 ... Reproduction | regeneration part, 13 ... Recording part, 14 ... Communication part, 15 ... UI part , 16 ... alignment unit, 20 ... server, 21 ... communication unit, 22 ... reverse alignment unit, 23 ... storage unit, 24 ... synchronization unit, 101 ... CPU, 102 ... memory, 103 ... storage, 104 ... communication IF, 105 ... Display, 106 ... Input device, 107 ... Microphone, 108 ... Speaker, 109 ... Camera, 201 ... CPU, 202 ... Memory, 203 ... Storage, 204 ... Communication IF

Claims (5)

First sound data based on a first demonstration performed during playback of a reference sound and second sound data based on a second demonstration different from the first demonstration performed during playback of the reference sound, A sound data editing method comprising the step of synchronizing using the reference sound. The first sound data and the second sound data include data of a first channel and a second channel, respectively.
The data of the first channel represents the reference sound,
The sound data editing method according to claim 1, wherein the second channel data represents a performance sound. Obtaining an input sound signal representing the input sound;
The sound data editing method according to claim 1, further comprising: adjusting a tempo of at least one of the first sound data and the second sound data based on the input sound signal. Adjusting the tempo of the first sound data according to the reference sound;
Adjusting the tempo of the second sound data according to the reference sound,
The sound data editing method according to any one of claims 1 to 3, wherein, in the synchronizing step, the first sound data and the second sound data whose tempo has been adjusted are synchronized. The first sound data is sound data included in the first moving image data,
The second sound data is sound data included in second moving image data different from the first moving image data,
The sound data editing method according to any one of claims 1 to 4, wherein, in the synchronizing step, the first moving image data and the second moving image data are synchronized.

Images