JP6819384B2 - Sequence data processing method and information processing equipment - Google Patents

Description

The present invention relates to a technique for processing sequence data.

It is known that at least a part of music recorded on a recording medium such as a CD (Compact Disc) is described by using sequence data such as MIDI (Musical Instrument Digital Interface). MIDI data is generally input manually by a user (worker) while listening to a musical piece. In this regard, Patent Document 1 discloses a technique for adding volume control information to performance data based on a given pattern.

Japanese Patent No. 3900187

When the user manually inputs the sequence data, there is a problem that the processing for inputting the information of the articulation (vibrato etc.) as a musical expression is complicated. Further, in the technique of Patent Document 1, there is a problem that volume control information that is not in the pattern cannot be added.

On the other hand, the present invention provides a technique for more easily processing sequence data based on a sound signal.

The present invention provides a sequence data processing method for processing at least a part of sequence data of a predetermined music based on a sound signal of the predetermined music.

The sound signal includes the performance sound of the musical instrument, and the sequence data may indicate a sequence for playing the musical instrument.

The sequence data includes information indicating a pitch, has a step of matching the sequence data with the sound signal, and may process the sequence data according to the result of the matching.

This sequence data processing method may include a step of filtering the processed sequence data.

The filtering process may be a process of suppressing a change in pitch in the sequence data.

The filtering process may be a process of suppressing the processing of sequence data for sounds whose sound length is shorter than the threshold value.

The filtering process may be a process of suppressing processing of sequence data for multiple sounds.

This sequence data processing method may include a step of receiving an instruction for editing the processed sequence data and a step of editing the sequence data in response to the instruction.

The present invention also provides an information processing device having a processing unit that processes at least a part of sequence data of a predetermined musical piece based on a sound signal of the predetermined musical piece.

According to the present invention, sequence data can be easily processed based on a sound signal.

The figure which illustrates the functional structure of the sequence data processing system 1 which concerns on 1st Embodiment. The figure which illustrates the hardware configuration of the sequence data processing system 1. The flowchart which illustrates the operation which concerns on 1st Embodiment. The flowchart which illustrates the detail of the processing process in step S3. The figure which illustrates the process in step S35. The figure which illustrates the sequence data processed by the processing part 13. The figure which illustrates the functional structure which concerns on 2nd Embodiment of the sequence data processing system 1. The flowchart which illustrates the operation which concerns on 2nd Embodiment. The figure which shows an example of the filtering process. The figure which shows another example of filtering. The figure which shows still another example of filtering. The figure which illustrates the functional structure which concerns on 3rd Embodiment of the sequence data processing system 1. The figure which illustrates the operation which concerns on 3rd Embodiment. The figure which illustrates the UI screen for editing sequence data.

1. 1. First Embodiment 1-1. Configuration FIG. 1 is a diagram illustrating the functional configuration of the sequence data processing system 1 according to the first embodiment. The sequence data processing system 1 processes the given sequence data. The sequence data is data that describes the sequence of processing performed in the processing device 2, that is, the content of the processing and its timing. The processing device 2 is a device that performs processing according to sequence data. In one example, the processing device 2 is a music sequencer or an automatic musical instrument, and the sequence data is data indicating a sequence for playing a predetermined musical piece in the automatic musical instrument. The sequence data is, for example, MIDI (Musical Instrument Digital Interface), and includes information indicating the pitch.

The sequence data processing system 1 includes a sequence input unit 11, a sound input unit 12, a processing unit 13, an output unit 14, and a UI (User Interface) unit 15. The sequence input unit 11 receives input of sequence data. The sound input unit 12 receives an input of a sound signal. In one example, the sound signal includes the performance sound of a musical instrument, and the sequence data is data for outputting the sound of the musical instrument in the music sequencer or data for performing performance in the automatic performance instrument. The processing unit 13 processes at least a part of the sequence data input via the sequence input unit 11 based on the sound signal input via the sound input unit 12. The output unit 14 outputs the processed sequence data. The UI unit 15 provides a user interface for processing sequence data.

FIG. 2 is a diagram illustrating a hardware configuration of the sequence data processing system 1. In this example, the sequence data processing system 1 is configured by the information processing device 10. The information processing device 10 is a computer device having a CPU (Central Processing Unit) 101, a memory 102, a storage 103, a communication IF 104, an input / output IF 105, an input device 106, a display 107, and a speaker 108, for example, a smartphone, a tablet terminal, or a personal computer. It is a computer. The CPU 101 is a control device that performs various calculations according to a program and controls other hardware elements of the information processing device 10. The memory 102 is a main storage device that functions as a work area when the CPU 101 executes processing, and includes, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). The storage 103 is an auxiliary storage device that stores various data and programs, and includes, for example, an SSD (Solid State Drive) or an HDD (Hard Disk Drive). The communication IF 204 is a device for communicating with another device according to a predetermined communication standard, and includes, for example, an antenna and a chipset for performing wireless communication. The input / output IF 105 is a device for inputting / outputting a signal to / from an external device, and includes, for example, a signal input terminal and an A / D converter. In this example, the sound signal is input via the input / output IF 105. The input device 106 is a device for the user to input information or instructions, and includes, for example, a touch device or a keyboard. The display 107 is a device that outputs an image. The speaker 108 is a device that outputs sound.

In this example, the storage 103 stores a program (hereinafter referred to as “sequence data processing program”) for operating the computer device as the sequence data processing system 1. While the CPU 101 is executing the sequence data processing program, at least one of the communication IF 104, the input / output IF 105, and the input device 106 is an example of the sequence input unit 11. The input / output IF 105 is an example of the sound input unit 12. The CPU 101 is an example of the processing unit 13. At least one of the storage 103, the input / output IF 105, and the display 107 is an example of the output unit 14. The input device 105, the display 107, and the speaker 108 are examples of the UI unit 15.

1-2. Operation FIG. 3 is a flowchart illustrating an operation (sequence data processing method) according to the first embodiment of the sequence data processing system 1. The flow of FIG. 3 is started, for example, when an instruction to start processing sequence data is input from the user via the UI unit 15. In the following, the functional elements such as the UI unit 15 are described as the main body of the processing, but this is because the hardware elements such as the CPU 101 executing the sequence data processing program perform the processing in cooperation with the other hardware elements. Means to do. Further, here, an example in which the processing device 2 is a music sequencer and the sequence data is MIDI data will be described.

In step S1, the sequence input unit 11 receives the input of sequence data. This sequence data is data for outputting the performance sound of a certain musical instrument (hereinafter referred to as "target musical instrument", for example, a guitar) in a certain musical piece (hereinafter referred to as "target song"; an example of a predetermined song). This sequence data is, for example, data manually input by a worker (user) who has heard the target song via the UI unit 15. This sequence data includes information indicating the pitch, length, and timing of the performance sound of the target instrument, but does not include, or even contains, information on articulation as a musical expression. is not it. The articulation referred to here includes, for example, expressions such as sound intensity, slur, staccato, vibrato, and choking.

In step S2, the sound input unit 12 receives the input of the sound signal. This sound signal is a signal indicating the performance sound of the target song. This performance sound includes not only the sound of the target musical instrument but also the sound of another musical instrument and the sound emitted from a sound source other than the musical instrument (for example, a human voice). In one example, this sound signal is a reproduction of the sound signal of a target song recorded on a medium such as a CD (Compact Disc).

In step S3, the processing unit 13 processes the sequence data based on the sound signal. The processing referred to here includes, for example, a processing for adding articulation information. The process of step S3 will be described later. In step S4, the output unit 14 outputs the processed sequence data. The output referred to here includes, for example, recording to a storage device or a storage medium, transmission by wired or wireless communication, and display on a display device.

FIG. 4 is a flowchart illustrating the details of the machining process in step S3. The processing process in step S3 includes a process of extracting main pitch component candidates from the sound signal, a process of matching the sound signal with the sequence data, and a process of transferring the pitch movement from the sound signal to the sequence data. .. In the following, the pitch at time t [k], which is indicated by the sequence data before processing, is expressed as Pb (t [k]). The time t [k] indicates the position of the target music in the time domain. The following processing is repeated by dividing the time domain into predetermined frames. The time t [k] indicates the time of the kth frame. Further, the pitch at time t, which is indicated by the sequence data after processing, is expressed as Pa (t [k]).

In step S31, the processing unit 13 obtains the pitch likelihood function L (p, t [k]) from the input sound signal. The pitch likelihood function is information indicating the time change of the likelihood of pitch (pitch). The pitch likelihood function L (p, t [k]) indicates the likelihood that the pitch is p at time t [k]. A well-known technique is used as a method for obtaining a pitch likelihood function from a sound signal. In one example, J. Salamon et. Al., Emilia Gomez and Jordi Bonada “SINUSOID EXTRACTION AND SALIENCE FUNCTION DESIGN FOR PREDOMINANT MELODY ESTIMATION”, Proc. Of the 14th Int. Conference on Digital Audio Effects (DAFx-11), Paris, France , September 19-23, 2011, the techniques described are used. The process of step S31 is an example of a process of extracting the main pitch component candidates from the sound signal.

In step S32, the processing unit 13 initializes the time t [k]. Specifically, the processing unit 13 is initialized to k = 0.

In step S33, the processing unit 13 determines whether the current frame includes the start position of the note. The sequence data includes information indicating the start position of the note. The processing unit 13 refers to the sequence data and determines whether the current frame includes the start position of the note. When it is determined that the current frame includes the start position of the note (step S33: YES), the processing unit 13 shifts the process to step S34. When it is determined that the current frame does not include the start position of the note (step S33: NO), the processing unit 13 shifts the process to step S35.

In step S34, the processing unit 13 uses the processed pitch Pa (t [k-1]) of the front frame as the pitch Pb (t [k]) before processing because it is used for the calculation in the next step S34. Set to the same value as. That is,
Pa (t [k-1]) = Pb (t [k])
(Pa (t [k-1]) calculated in step S34 is a tentative value to be used in the calculation in step S34, and is different from the information added to the sequence data). The process of step S34 is an example of a process of matching the sound signal and the sequence data. When the process of step S34 is completed, the processing unit 13 shifts the process to step S35.

In step S35, the processing unit 13 applies the pitch likelihood function L to a predetermined distribution function f (t [k-1]) centered on the pitch Pa (t [k-1]) after processing of the front frame. Multiply (p, t [k]). The pitch that gives a peak in the product of the two is defined as the pitch Pa (t [k]) of the current frame. That is,
Pa (t [k]) = p [max]
However, p [max] is p that gives the maximum value of f (t [k-1]) · L (p, t [k]). As the distribution function f, for example, a Gaussian distribution is used. Pa (t [k]) is converted into information according to the sequence data for Pb (t [k]) (for example, a pitch bend event in MIDI) and added to the sequence data. The process of step S35 is an example of a process of transferring the pitch movement from the sound signal to the sequence data.

FIG. 5 is a diagram illustrating the process in step S35. The background part of the figure shows the likelihood function. The horizontal axis shows the time and the vertical axis shows the frequency. The likelihood is indicated by the shade of color, and the dark (dark) part indicates that the likelihood is low, and the light (bright) part indicates that the likelihood is high. The pitch Pa (t [k-1]) and the distribution function f (t [k-1]) after processing the front frame are illustrated in the figure.

This will be described again with reference to FIG. In step S36, the processing unit 13 updates the frame to be processed to the next frame. When the frame to be processed is updated, the processing unit 13 shifts the processing to step S33 again. The processing unit 13 repeatedly executes the processing of steps S33 to S36 until the processing is completed for all the frames of the sequence data.

FIG. 6 is a diagram illustrating sequence data processed by the processing unit 13. In this figure, as in FIG. 5, the background portion shows the likelihood function. The pitch Pb before processing and the pitch Pa after processing are shown superimposed on the background. In this figure, the sequence data is drawn as a pitch curve showing the time change of pitch. According to this example, articulation information can be automatically added to the sequence data.

2. 2. Second Embodiment FIG. 7 is a diagram illustrating a functional configuration according to the second embodiment of the sequence data processing system 1. Hereinafter, the matters common to the first embodiment will be omitted. In the second embodiment, the sequence data processing system 1 has a filter unit 16. The filter unit 16 filters the sequence data processed by the processing unit 13. Here, the filtering process means a process of removing at least a part of the articulation information added in the processing unit 13.

FIG. 8 is a flowchart illustrating an operation (sequence data processing method) according to the second embodiment of the sequence data processing system 1. In step S5, the filter unit 16 filters the sequence data processed by the processing unit 13.

FIG. 9 is a diagram showing an example of filtering. In this example, the filtering process is a process of suppressing a sudden change in pitch. Sudden change in pitch (jump) when the matching between the sequence data and the pitch likelihood function does not go well, or when the transfer of the pitch movement from the pitch likelihood function to the sequence data does not go well. May occur. Therefore, in this example, when a sudden change in pitch occurs, it is considered that a problem has occurred in the processing in the processing unit 13, and the information that causes the sudden change in pitch is removed from the sequence data. .. In one example, since the processing unit 13 adds a pitch bend event to the sequence data, the filter unit 16 deletes the pitch bend event from the sequence data when the change in pitch indicated by the pitch bend event exceeds the threshold value. This threshold value is preset in the sequence data processing system 1. Alternatively, this threshold may be specified by the user via the UI unit 15.

According to this example, a pitch bend event having a magnitude exceeding the threshold value Th is replaced with a pitch bend event having the same magnitude as the threshold value Th. That is, as a result, the change in pitch exceeding the threshold value is suppressed to the same change in pitch as the threshold value. In another example, all pitch bend events may be deleted for a note to which a pitch bend event having a magnitude exceeding the threshold Th is added.

In this example, in addition to the magnitude of the change in pitch (compared to the threshold value), the filtering process may be performed in consideration of the stability of the change in pitch after the change in pitch. For example, the filter unit 16 may delete the pitch bend event only when the magnitude of the change in pitch is larger than the threshold value and the pitch after the change is unstable. The stability of the change in pitch is evaluated, for example, by the value obtained by dividing the value obtained by integrating the absolute value of the amount of change in pitch by the time length of the note.

FIG. 10 is a diagram showing another example of filtering. In this example, the filtering process is a process of suppressing the processing of short sounds. In general, articulation as a musical expression is often added to a note having a long note length to some extent, and articulation is rarely added to a note having a short note length. Therefore, in this example, the filter unit 16 deletes the articulation information from the sequence data for notes whose note length is shorter than the threshold value.

FIG. 11 is a diagram showing another example of filtering. In this example, the filtering process is a process of suppressing processing of polyphony. In general, articulation as a musical expression is often added to a single note, and articulation is rarely added to a compound note. Therefore, in this example, the filter unit 16 deletes the articulation information from the sequence data for the notes constituting the compound sound. Note that the compound sound means a plurality of notes emitted at the same timing.

See FIG. 8 again. In step S6, the output unit 14 outputs the sequence data filtered by the filter unit 16. The details of the output process are the same as in step S4 of FIG. According to these examples, it is possible to remove the articulation information added by mistake or the articulation information added to the portion where the articulation is unlikely to be added in the musical expression. In addition, two or more of the filter treatments exemplified in FIGS. 9 to 11 may be used in combination.

3. 3. Third Embodiment FIG. 12 is a diagram illustrating a functional configuration according to a third embodiment of the sequence data processing system 1. In the second embodiment, the sequence data processing system 1 has an editorial unit 17. The editorial unit 17 edits the sequence data according to an instruction from the user. The instruction from the user is input via the UI unit 15.

FIG. 13 is a diagram illustrating an operation (sequence data processing method) according to the third embodiment of the sequence data processing system 1. In step S7, the editing unit 17 edits the sequence data filtered by the filtering unit 16.

FIG. 14 is a diagram illustrating a UI screen for editing sequence data. This UI screen includes a background 91 and a pitch curve 92. The background 91 shows a pitch likelihood function obtained from a sound signal. The pitch curve 92 shows the time change of the pitch obtained from the sequence data after the filtering. The pitch curve 92 is displayed in a state where the pitch likelihood function displayed as the background 91 and the positions on the horizontal axis (time axis) and the vertical axis (frequency axis) are aligned. The UI unit 15 receives instructions for editing the pitch curve by a well-known technique such as drag and drop or parameter input in a dialog box. The editing unit 17 edits the sequence data according to the instruction input via the UI unit 15.

This will be described again with reference to FIG. In step S8, the output unit 14 outputs the sequence data edited by the editorial unit 17. The details of the output process are the same as in step S4 of FIG. According to this example, the pitch curve (sequence data) can be edited (corrected) according to the pitch likelihood function while visually confirming the pitch likelihood function. Although the example in which the editing unit 17 is added to the sequence data processing system 1 of the second embodiment has been described here, the editing unit 17 may be added to the sequence data processing system 1 of the first embodiment.

4. Modifications The present invention is not limited to the above-mentioned first to third embodiments, and various modifications can be carried out. Two or more of the above embodiments may be used in combination. In addition, at least one of the following modifications may be applied to each of the above-described embodiments or a combination of two or more.

The information added to the sequence data in the processing by the processing unit 13 is not limited to the pitch bend event, that is, the information indicating the change in pitch. In addition to or in addition to the information indicating the change in pitch, information indicating the change in parameters other than pitch may be added to the sequence data. In one example, the processing unit 13 may add information indicating a change in volume. In this case, the processing unit 13 uses the pitch likelihood function to calculate the average power for each note. The processing unit 13 converts the power thus obtained into information according to the format of the sequence data (for example, a velocity event in MIDI), and adds this information to the sequence data. According to this example, a change in volume (dynamics) as a musical expression can be added to the sequence data.

The processing unit 13 may align the positions of the sequence data and the sound signal on the time axis by using a so-called score alignment technique. The technique of score alignment is described in, for example, Japanese Patent Application Laid-Open No. 2015-79183. That is, the processing unit 13 adjusts the sounding timing and the final sound timing (for example, the timing of the note-on event and the note-off event in MIDI) in the sequence data by using the score alignment technique. According to this example, information indicating a timing shift (so-called tame, glue, or groove) as a musical expression can be added to the sequence data.

The method of matching the sequence data and the sound signal is not limited to the one using the pitch likelihood function. For example, a power spectrum may be used instead of the pitch likelihood function. The power spectrum is obtained by dividing the sound signal into frames and calculating the power in each frame for each frequency.

The sound signal is not limited to the data including the performance sound of the music, and the sequence data is not limited to the data for outputting the sound of the musical instrument in the music sequencer or the data for performing the performance in the automatic musical instrument. The sound signal may be, for example, reading or speaking, and the sequence data may be data for controlling the lighting device.

The hardware configuration of the data processing system 1 is not limited to that illustrated in FIG. The sequence data processing system 1 may have any hardware configuration as long as it can realize the required functions. For example, a plurality of information processing devices may cooperate to function as the data processing system 1. In this case, a plurality of information processing devices may form a server / client system, and some of the functions of the sequence data processing system 1 may be implemented on a server on the network.

The program executed in the information processing apparatus 10 may be provided by a storage medium such as an optical disk, a magnetic disk, or a semiconductor memory, or may be downloaded via a communication line such as the Internet. Also, this program does not have to include all the steps of FIG. Some of these steps may be omitted.

1 ... Sequence data processing system, 2 ... Processing device, 10 ... Information processing device, 11 ... Sequence input unit, 12 ... Sound input unit, 13 ... Processing unit, 14 ... Output unit, 15 ... UI unit, 16 ... Filter unit, 91 ... Background, 92 ... Pitch curve, 101 ... CPU, 102 ... Memory, 103 ... Storage, 104 ... Communication IF, 105 ... Input / output IF, 106 ... Input device, 107 ... Display, 108 ... Speaker

Claims (11)

A step of processing at least a part of the sequence data of a predetermined musical piece based on the sound signal of the predetermined musical piece, and
A step of filtering the processed sequence data was
Sequence data processing method having . The sequence data processing method according to claim 1 , wherein the filter processing is a process of suppressing a change in pitch in sequence data. The sequence data processing method according to claim 1 , wherein the filter processing is a process of suppressing processing of sequence data for sounds having a sound length shorter than a threshold value. The sequence data processing method according to claim 1 , wherein the filter processing is a process of suppressing processing of sequence data for multiple sounds. A step of processing at least a part of the sequence data of a predetermined musical piece based on the sound signal of the predetermined musical piece, and
A step of receiving an instruction to edit the processed sequence data, and
With the step of editing the sequence data according to the instruction
Sequence data processing method having . It has a step of processing at least a part of the sequence data of a predetermined musical piece based on the sound signal of the predetermined musical piece input via the AD converter.
The processing step is
The step of extracting the main pitch component candidates from the sound signal and
The step of matching the sound signal with the sequence data,
Based on the matching, the step of transferring the movement of the main pitch from the sound signal to the sequence data
Sequence data processing methods , including . The sound signal includes the performance sound of the musical instrument.
The sequence data processing method according to any one of claims 1 to 6, wherein the sequence data indicates a sequence for playing a musical instrument. The sequence data includes information indicating pitch.
It has a step of matching the sequence data with the sound signal.
The sequence data processing method according to any one of claims 1 to 7, wherein the sequence data is processed according to the result of the matching. At least part of the sequence data of a predetermined musical composition, and pressurizing coated portion you processing based on the sound signal of the predetermined piece of music,
An information processing device having a filter unit for filtering the processed sequence data . A processing unit that processes at least a part of the sequence data of a predetermined musical piece based on the sound signal of the predetermined musical piece, and
A UI unit that accepts instructions for editing the processed sequence data,
With an editorial department that edits the sequence data according to the instructions
Information processing device with. It has a processing unit that processes at least a part of the sequence data of a predetermined musical piece based on the sound signal of the predetermined musical piece input via the AD converter.
The processed part
Candidates for the main pitch components are extracted from the sound signal,
Matching the sound signal with the sequence data
Based on the matching, the movement of the main pitch is transferred from the sound signal to the sequence data.
Information processing device.