WO2023032137A1

WO2023032137A1 - Data modification method, data reproduction method, program, data modification device

Info

Publication number: WO2023032137A1
Application number: PCT/JP2021/032374
Authority: WO
Inventors: 善政磯崎; 聡史宇佐; 琢哉藤島; 和彦山本
Original assignee: ヤマハ株式会社
Priority date: 2021-09-02
Filing date: 2021-09-02
Publication date: 2023-03-09
Also published as: JPWO2023032137A1; CN117882129A

Abstract

In the data modification method according to one embodiment: a selection user interface is provided for selecting, from among a plurality of modification methods including a first modification method and a second modification method, a modification method to be applied to sound control data that defines sound timing information; when the application of the first modification is selected, the sound control data is modified by correcting the timing information and adding, to a specific data segment, correction information based on a correction amount of the timing information in said specific data segment; and when the application of the second modification method is applied, the sound control data is modified by correcting the timing information on the basis of the position of a beat based on a specific tempo.

Description

DATA CHANGE METHOD, DATA REPRODUCTION METHOD, PROGRAM AND DATA CHANGE DEVICE

The present disclosure relates to a method of changing data for controlling pronunciation.

　There is a technique for generating a sound waveform signal by performing automatic performance based on data for controlling sound production such as MIDI data (hereinafter referred to as sound production control data). In general, the sound generation control data defines the speed (tempo) of automatic performance. That is, when the sound generation control data is reproduced, the performance sound recorded in the sound generation control data is reproduced. As shown in Patent Documents 1, 2, and 3, techniques have also been developed for controlling the tempo according to the movement of the user. The tempo control by the user is realized, for example, by swinging the portable device held in the user's hand like a baton. Therefore, the performance sound recorded in the sound generation control data is not reproduced as it is, but the performance sound is reproduced with the reproduction speed changed according to the user's operation.

JP-A-8-272362 JP-A-9-16169 Utility Model Registration No. 3227548

Either automatic performance under tempo control by the user's actions or automatic performance under predetermined tempo control may be selected, and the selected automatic performance may be realized. In such a case, it is desirable that the sound generation control data capable of controlling the tempo by the user's motion and the sound generation control data capable of controlling the predetermined tempo are generated as common data. Since sounding control data with a predetermined tempo generally exists, it is possible to change the sounding control data with a predetermined tempo so that it can also be used as sounding control data that allows the user to control the tempo. desired.

One of the purposes of the present disclosure is to easily change sound generation control data in which performance sounds are recorded into data that enables tempo control by user's actions.

According to an embodiment of the present disclosure, a selection for selecting a modification method to be applied to pronunciation control data defining pronunciation timing information from a plurality of modification methods including a first modification method and a second modification method providing a user interface, correcting the timing information when applying the first change method is selected, and correcting the timing information for a predetermined data section according to the correction amount of the timing information in the data section. modifying the pronunciation control data by adding information to correct the timing information based on beat positions according to a predetermined tempo when applying the second modification method is selected; provides a data modification method comprising: modifying the pronunciation control data.

According to an embodiment of the present disclosure, based on pronunciation control data defining pronunciation timing information recorded at a predetermined tempo, positions of a plurality of beats are detected, and positions of the plurality of beats and the tempo are detected. modifying the sound generation control data by correcting the timing information based on the relationship of A data modification method is provided that includes:

According to one embodiment of the present disclosure, it is possible to easily change sound generation control data in which performance sounds are recorded into data that enables tempo control by user's actions.

It is a figure which shows the system configuration in one embodiment. 1 is a diagram showing the hardware configuration of a mobile device and an electronic musical instrument according to one embodiment; FIG. 6 is a flow chart showing processing in a print mode in one embodiment. FIG. 10 is a diagram showing an example of a recording user interface (before an instruction to start recording); FIG. 10 is a diagram showing an example of a recording user interface (after instructing to start recording); FIG. 10 illustrates an example of a selection user interface; 4 is a flow chart showing processing in a print mode (the part following FIG. 3) in one embodiment. 4 is a flow chart showing AI quantization processing in one embodiment. FIG. 10 is a diagram showing an example of a beat correction user interface (before instruction for correction). FIG. 10 is a diagram showing an example of a beat correction user interface (before instruction for correction). FIG. 4 is a diagram for explaining an example of sound generation control data; FIG. 4 is a flow chart showing data correction processing in one embodiment. FIG. 10 is a diagram for explaining an example of correcting timing information; FIG. FIG. 10 is a diagram for explaining a comparison of sound generation control data before and after change; 4 is a flowchart illustrating MIDI quantization processing in one embodiment. FIG. 10 is a diagram showing an example of a quantize setting user interface; 4 is a flow chart showing delay offset processing in one embodiment. FIG. 10 is a diagram showing an example of an offset setting user interface; FIG. 10 is a diagram for comparing and explaining sound generation control data before and after a delay offset; FIG. 10 is a diagram for comparing and explaining sound generation control data before and after a delay offset; 4 is a flowchart showing data reproduction processing in one embodiment. FIG. 10 is a diagram showing an example of a playback user interface;

Hereinafter, one embodiment of the present invention will be described in detail with reference to the drawings. The embodiments shown below are examples, and the present invention should not be construed as being limited to these embodiments. In the drawings referred to in this embodiment, the same parts or parts having similar functions are denoted by the same reference numerals or similar reference numerals (reference numerals followed by A, B, etc.). May be omitted.

[1. overall structure]
FIG. 1 is a diagram showing a system configuration in one embodiment. The system shown in FIG. 1 includes a mobile device 10 and a data management server 90 connected via a network NW such as the Internet, and further includes an electronic musical instrument 80 connected to the mobile device 10 . At least part of the functions of the electronic musical instrument 80 may be included in the portable device 10 . The mobile device 10 is a portable communication terminal such as a smart phone in this example. The electronic musical instrument 80 is an electronic keyboard device such as an electronic piano in this example.

The portable device 10 can record a performance on the electronic musical instrument 80 and reproduce the recorded performance on the electronic musical instrument 80 . In this reproduction, it is possible to select either to reproduce the recorded performance as it is or to reproduce at a speed according to the user's motion (for example, the motion of swinging the portable device 10 like a baton). . The portable device 10 can generate data corresponding to the recorded performance and modify the data so that it can be used in any of the above reproductions. Specific processing for realizing each of these functions will be described later.

The data management server 90 includes a control section 910 , a storage section 920 and a communication section 980 . Control unit 910 includes a CPU, RAM and ROM. Control unit 910 executes a program stored in storage unit 920 using the CPU, thereby performing processing according to instructions described in the program. Storage unit 920 includes a storage device such as a nonvolatile memory and a hard disk drive. Communication unit 980 includes a communication module for connecting to network NW and communicating with other devices. The data management server 90 stores part of the data generated in the portable device 10 in the storage unit 920 and executes processing using the data. The specific contents of the processing will be described later. In one embodiment, data management server 90 may not exist.

[2-1. Configuration of portable device]
FIG. 2 is a diagram showing the hardware configuration of the portable device and the electronic musical instrument according to one embodiment. Portable device 10 includes control unit 110 , storage unit 120 , display unit 130 , operation unit 140 , sensor unit 150 , speaker 170 , communication unit 180 and interface 190 . The mobile device 10 is not limited to including all of these components, and may include other components such as a camera and a position detection unit.

The control unit 110 is an example of a computer including a processor such as a CPU (Central Processing Unit) and a storage device such as a RAM. The control unit 110 executes a program 121 stored in the storage unit 120 using a CPU (processor), and causes the portable device 10 to implement functions for executing various processes described later.

The storage unit 120 is a storage device such as a non-volatile memory or hard disk drive. The storage unit 120 stores a program 121 executed by the control unit 110 and various data necessary for executing the program 121 . The program 121 is installed in the portable device 10 by being downloaded from the data management server 90 or another server via the network NW and stored in the storage unit 120 . The program 121 may be provided in a state recorded in a non-transitory computer-readable recording medium (for example, a magnetic recording medium, an optical recording medium, a magneto-optical recording medium, a semiconductor memory, etc.). In this case, the portable device 10 may be provided with a device for reading this recording medium. The storage unit 120 can also be said to be an example of a recording medium.

The data stored in the storage unit 120 includes performance record data 123, for example. The performance recording data 123 includes various data related to the performance, such as metadata such as the name of the music played and data related to the sound of the performance. The data on the performance sound includes sound generation control data 125 that defines the timing information of the sound generated by the performance. The sound generation control data 125 is data described in a predetermined format, for example, data described in MIDI format. In this case, the pronunciation timing information is information indicating the timing at which the sound is generated, and is indicated by information such as note-on, note-off, delta time, and tempo, for example. The sound control data 125 also includes other information corresponding to sound, such as pitch (note number).

The display unit 130 is a display having a display area for displaying various screens under the control of the control unit 110. The displayed screen includes a plurality of user interfaces, which will be described later. The operation unit 140 is an operation device that outputs a signal to the control unit 110 according to a user's operation. In this example, operation unit 140 is a touch sensor arranged on the surface of display unit 130 . Therefore, the display unit 130 and the operation unit 140 are used as a touch panel by mutual functions. The operation unit 140 may include switches and the like arranged on the housing of the mobile device 10 .

The sensor unit 150 outputs a signal corresponding to the movement of the mobile device 10 to the control unit 110 . The sensor unit 150 includes, for example, an acceleration sensor, a gyro sensor, etc., and measures the movement of the mobile device 10 . The speaker 170 generates sound by amplifying and outputting the sound waveform signal supplied from the control unit 110 .

The communication unit 180 is a wireless communication module for connecting to the network NW under the control of the control unit 110 and communicating with other devices such as the data management server 90 connected to the network NW. The interface 190 includes a communication module for communicating with other devices such as the electronic musical instrument 80 by infrared communication or short-range wireless communication. The interface 190 is used for communication without going through the network NW. Interface 190 may include modules that provide wired rather than wireless communication.

[2-2. Configuration of Electronic Musical Instrument]
Electronic musical instrument 80 is an electronic keyboard device such as an electronic piano, and includes performance operators 810 , tone generator section 830 , speaker 870 and interface 890 , as described above. The performance operator 810 includes a plurality of keys, and outputs a signal to the tone generator section 830 according to the operation of each key. Interface 890 includes a communication module for communicating wirelessly or by wire with external devices. In this example, the interface 890 connects with the interface 190 of the portable device 10 by short-range wireless communication to transmit and receive data in a predetermined format (MIDI format in this example).

The sound source unit 830 includes a DSP (Digital Signal Processor) and generates a sound waveform signal according to the pronunciation instruction signal. The sounding instruction signal corresponds to the signal output from performance operator 810 and the data transmitted from portable device 10 via interface 890 . Also, the tone generator section 830 converts the sound generation instruction signal corresponding to the signal output from the performance operator 810 into data in MIDI format and outputs the data to the interface 890 . As a result, the electronic musical instrument 80 can transmit data corresponding to the operation of the performance operator 810 (hereinafter sometimes referred to as performance information) to the portable device 10 .

[3. Recording mode]
Next, a description will be given of processing when a performance on the electronic musical instrument 80 is recorded on the portable device 10. FIG. A processing mode for recording a performance in this way is called a recording mode. When the user inputs an instruction to start the recording mode, the portable device 10 starts processing in the recording mode.

FIG. 3 is a flow chart showing processing in a recording mode in one embodiment. FIG. 4 is a diagram showing an example of the recording user interface (before instructing to start recording). FIG. 5 is a diagram showing an example of the recording user interface (after instructing recording). The control unit 110 provides the user with a recording user interface (recording UI) by displaying a screen described below on the display unit 130 (step S101). The recording user interface (hereinafter sometimes referred to as interface RD) shown in FIG. 4 is a display example before the start of performance recording is instructed. The interface RD includes an area TS for setting the recording tempo (hereinafter sometimes referred to as recording tempo), an area RT for displaying the recording time, and a record button RB for instructing start and end of recording. .

The control unit 110 waits until an instruction to start recording the performance is input (step S103; No). When an instruction to start recording is input to the portable device 10 by operating the record button RB shown in FIG. 4 (step S103; Yes), the control unit 110 displays the record button RB as shown in FIG. In order to change and start recording the performance, beats are provided to the user (step S105). Providing beats is realized by, for example, generating a metronome sound from the portable device 10 . A metronome sound is generated corresponding to each beat specified at intervals determined by the recording tempo. For example, if the recording tempo is 120, a metronome sound will be generated with a beat length of 500 milliseconds. The mobile device 10 may provide beats to the user not only by sound but also by light, vibration, or the like.

The control unit 110 records the sound generation control data 125 in the storage unit 120 based on the performance information provided from the electronic musical instrument 80 (step S107) until an instruction to end recording is input (step S109; No). At this time, beats are still provided to the user. When the recording end instruction is input to the portable device 10 by operating the recording button RB shown in FIG. 5 (step S109; No), the control section 110 ends the beat provision (step S111). Subsequently, the control unit 110 provides a selection user interface (selection UI) by displaying a screen described below on the display unit 130 (step S113).

By providing the selection user interface in this way, it is possible to provide the user with the processing to be performed after recording the sound generation control data 125 based on the performance. On the other hand, the control unit 110 may terminate once the recording of the sound generation control data 125 is completed according to the user's instruction. In this case, the control unit 110 may provide a selection user interface according to a user's instruction, and may further provide a user interface for designating the pronunciation control data 125 to be processed. Data other than the sound generation control data 125 recorded in the recording mode may be specified as the object of processing.

FIG. 6 is a diagram showing an example of a selection user interface. As shown in FIG. 6, a selection user interface (hereinafter sometimes referred to as an interface SD) is an interface for selecting a modification method to be applied to the recorded pronunciation control data 125 from a plurality of different modification methods. including. The plurality of change methods may be any change method as long as they change the sound generation control data 125 by mutually different processes. The plurality of modification methods may include at least two modification methods that modify the pronunciation control data 125 by correcting the timing information in different ways. In this example, the plurality of modification methods includes three modification methods. The three modification methods correspond to "AI Quantize" (first modification method), "MIDI Quantize" (second modification method) and "Delay Offset" (third modification method). The interface SD includes change selection buttons AB, MB, DB corresponding to each change method. "AI quantize" detects the beat position from the pronunciation control data 125, and corrects the timing information according to the relationship between the beat position specified from the original pronunciation control data 125 and the detected beat position. This is the process of changing the control data 125 . AI (Artificial Intelligence) technology is used to detect beat positions. "MIDI quantize" is a process of changing the sound generation control data 125 so as to correct the timing information according to the specified resolution. "Delay offset" is processing for changing the sound generation control data 125 so as to offset timing information corresponding to a plurality of sounds. These specific processing methods will be described later.

The interface SD includes a save button B1 for saving the sound control data 125, a delete button B2 for discarding the sound control data 125, and a play button PB for playing back the sound control data 125. When the play button PB is operated, the sound control data 125 is read and the sound indicated by the sound control data 125 is reproduced. This sound may be generated by the portable device 10 or may be generated by the electronic musical instrument 80 . When the electronic musical instrument 80 generates a sound, the portable device 10 may transmit a sound generation instruction signal to the electronic musical instrument 80 . A playback position is indicated by a playback marker PM. The user can change the playback position in the sound generation control data 125 by changing the position of the playback marker PM.

FIG. 7 is a flow chart showing processing in the recording mode (the part following FIG. 3) in one embodiment. Control unit 110 waits for an instruction input from the user to interface SD (step S201; No, S203; No, S205; No, S207; No, S209; No). A state in which the control unit 110 waits for an instruction in this manner is referred to as an instruction standby state.

When an AI quantize instruction is input by operating the change selection button AB (step S201; Yes), the control unit 110 executes the AI quantize process (step S300) and returns to the instruction standby state. AI quantize processing will be described later.

When a MIDI quantization instruction is input by operating the change selection button MB (step S203; Yes), the control unit 110 executes the MIDI quantization process (step S400) and returns to the instruction waiting state. MIDI quantize processing will be described later.

When the change selection button DB is operated to input a delay offset instruction (step S205; Yes), the control unit 110 executes the delay offset process (step S500) and returns to the instruction waiting state. The delay offset processing will be described later.

When the save button B1 is operated to input a data save instruction (step S207; Yes), the control unit 110 saves the sound generation control data 125 (step S211), and ends the recording mode processing. . When the delete button B2 is operated to input an instruction to discard the data (step S209; Yes), the control unit 110 discards the sound generation control data 125 (step S213) and terminates the recording mode processing. .

[4. AI quantize processing]
Next, AI quantize processing will be described.

FIG. 8 is a flowchart showing AI quantization processing in one embodiment. The control unit 110 converts the sound generation control data 125 into audio data (step S301). The audio data is data in which the sounding according to the timing information defined in the sounding control data 125 is indicated by a sound waveform signal. This sound waveform signal may be generated by the portable device 10 or may be generated by the electronic musical instrument 80 and received by the portable device 10 . Control unit 110 detects beats based on the audio data (step S303).

Beats can be detected based on changes in the amplitude of the sound waveform signal. In this example, AI technology is used for beat detection. By inputting audio data obtained by conversion into a trained model that has learned the relationship between audio data and beat positions, beat positions can be obtained from the trained model. A specific method for detecting beats is not limited to the case of using AI technology, and known beat detection methods can be applied. Beat detection is not limited to audio data, and may be performed on the pronunciation control data 125 .

The control unit 110 provides a beat correction interface (beat correction UI: first user interface) by displaying a screen described below on the display unit 130 (step S305). The control unit 110 waits until an instruction to correct the beat position or an instruction to perform data correction processing is input to the beat correction interface (step S311; No, S321; No). When an instruction to correct the beat position is input (step S311; Yes), the control section 110 corrects the beat position (step S313) and waits again (step S311; No, S321; No). When the data correction process is instructed (step S321; Yes), the control unit 110 executes the data correction process (step S330) and ends the AI quantize process. Data correction processing may be instructed without inputting an instruction to correct the beat position. In this case, data correction processing is executed using the detected beat position as it is.

FIG. 9 is a diagram showing an example of the beat correction user interface (before the correction instruction). FIG. 10 is a diagram showing an example of the beat correction user interface (before instructing correction). A beat correction interface (hereinafter sometimes referred to as an interface AD) includes an area TS for setting an average tempo, a line button LB and a tap button TB for setting a beat position editing method, a play button PB, and a redetection button. QB, and save button B3. For the average tempo, the tempo (recording tempo) when the sound generation control data 125 was recorded is displayed as an initial value. Normally, there is no need to change the initial value, but it may be changed. The interface AD described in FIGS. 9 and 10 is shown as an editing method when the line button LB is operated.

The interface AD further includes an entire area AA showing the pronunciation specified in the pronunciation control data 125 in the range of the entire song, and an enlarged area AW showing an enlarged range specified by the selection window SW. In these regions, the vertical axis indicates pitch, the horizontal axis indicates time, and each pronunciation is indicated by a note mark NM. The position of the selection window SW can be changed in the entire area AA, and by changing the length in the horizontal axis direction, it is possible to change the enlargement ratio in the time axis direction when displayed in the enlarged area AW.

The position of the beat detected from the sound generation control data 125 is indicated by the beat position line BL in the enlarged area AW. A circular beat marker BM is displayed below the beat position line BL. The beat marker BM is used when the user corrects the beat position. Of the beat markers BM, the beat markers BMs specified by the user as change targets are displayed in a form distinguishable from other beat markers BM. At this time, as shown in FIG. 9, the display form of the beat position line BL corresponding to the beat marker BMs may also change.

Shifting the beat marker BMs specified by the user in FIG. 9 to the left as shown in FIG. 10 corresponds to inputting a beat position correction instruction (FIG. 8, step S311; Yes). Thereby, the control unit 110 corrects the position of the beat corresponding to the beat marker BMs to the position of the beat marker BMs after movement ( FIG. 8 , step S313). Such correction is performed when the detected beat position and the beat position assumed by the user are different.

When the play button PB is operated, the control unit 110 plays back sounds corresponding to the sound generation control data 125 within the range displayed in the enlarged area AW. At this time, a sound indicating the beat (metronome sound) may be generated at the position of the detected beat. In this manner, the relationship between the detected beat position or the corrected beat position and the pronunciation timing may be recognized by the sound.

When the redetection button QB is operated, the control unit 110 returns to step S303 and executes beat detection. At this time, if there is a beat whose position has been corrected by the user, the control unit 110 may fix the position of the corrected beat and execute the beat detection process.

FIG. 11 is a diagram for explaining an example of sound generation control data. FIG. 11 shows the relationship between the timing information of the sound generation control data 125 and the positions of the beats at "during recording", "during beat detection", and "during beat correction". "When recording" indicates the sound generation control data 125 when recorded in the recording mode. "At time of beat detection" indicates the sound generation control data 125 when beats are detected in the AI quantize process. "At the time of beat correction" indicates the sound generation control data 125 when the detected beat position is corrected by the user. The horizontal axis indicates time t. The sound NT indicates the pronunciation position corresponding to the timing information. Since the sound generation control data 125 is not changed during the period from the time of recording to the time of beat correction, the position of the note NT does not change.

The beat positions SBT1, SBT2, . . . indicate the beat positions during recording. Therefore, both beats have the same length. The beat positions DBT1, DBT2, . The positions of the detected beats are not necessarily evenly spaced. Therefore, in the example of FIG. 11, there are beats whose positions (timings) are shifted between recording and beat detection. At the time of beat correction, the beat after the position of the beat is corrected by moving the beat marker BMs on the interface AD is shown. FIG. 11 shows an example in which the beat positions DBT4 and DBT6 are corrected. Correction of the beat position DBT4 corresponds to the example shown in FIG. Correction of the beat position DBT6 is correction for aligning the ornament NTs with the beat position. The purpose of such correction is related to data reproduction processing, and will be described later.

Operation of the save button B3 on the interface AD corresponds to input of an instruction for data correction processing (step S321; Yes). Therefore, when save button B3 is operated, control unit 110 executes data correction processing (step S330). At this time, the portable device 10 may associate the sound generation control data 125 or the audio data with the corrected beat position data and transmit the data to the data management server 90 . The data management server 90 registers the data received from the mobile device 10 in the database in the storage unit 920 . The data management server 90 can execute processing using data registered in the database. For example, such user-corrected data is used to improve the accuracy of beat detection technology, or to improve the accuracy of AI technology in beat detection (such as updating a learned model using registered data as teacher data). can also

FIG. 12 is a flowchart showing data correction processing in one embodiment. The control unit 110 divides the sound generation control data 125 at the beat positions DBT1, DBT2, . (step S331). The control unit 110 expands or contracts each of the plurality of divided data sections to a length corresponding to the section length corresponding to the average tempo (the length of one beat corresponding to the average tempo). Correspondingly, the position of each pronunciation included in each data section, that is, the timing information is corrected (step S333). The control unit 110 adds correction information according to the correction amount of the timing information to each data section (step S335), and combines the plurality of data sections to which the correction information with the corrected timing information is added (step S337). . Data correction processing will be described in more detail with reference to FIGS. 13 and 14. FIG.

FIG. 13 is a diagram for explaining an example of correcting timing information. FIG. 13 shows an example of correcting the timing information in the data section between beat position DBT4 and beat position DBT5 among a plurality of data sections. "After division" indicates the sound generation control data 125 in the data section before correction. "Corrected" indicates the sound generation control data 125 after correcting the timing information. In this example, the data section length before correction (length between beat position DBT4 and beat position DBT5) corresponds to tempo "110".

In this example, the average tempo is set to "120". Therefore, the data section length is changed to a length corresponding to the tempo "120". Since the length of the data section is multiplied by "110/120", the pronunciation timing information included in this data section is corrected to a position multiplied by "110/120" based on the beginning of the beat (beat position DBT4). . Thus, the correction amount is "110/120". The correction information is added as a tempo value "110" corresponding to the original data section length as a value corresponding to the correction amount at a position corresponding to the beat position DBT4. It may be added as a relative value to the average tempo as "110/120". If the data is in MIDI format, it is added as tempo change information. When the sound generation control data 125 in the corrected data section is reproduced at the tempo "110", the performance sound can be reproduced at the same timing as when the sound generation control data 125 was recorded.

FIG. 14 is a diagram for comparing and explaining sound generation control data before and after change. FIG. 14 shows the relationship between the timing information of the sound generation control data 125 and the positions of beats "during recording" and "after combining". "When recording" indicates the sound generation control data 125 when recorded in the recording mode, that is, before the timing information is changed by the AI quantize process. “After combining” indicates the sound generation control data 125 after combining the data sections to which the timing information has been corrected by AI quantization processing and the correction information has been added, that is, after the timing information has been changed. As shown in FIG. 14, the AI quantize process aligns the detected beats (or further corrected beats) to a predetermined tempo (average tempo in this example) so that the length of the original beat can be identified. The sound generation control data 125 is changed by adding correction information corresponding to the correction amount (in this example, the tempo value corresponding to the original beat length).

When the timing information is changed by the AI quantize process in this way, the sound generation control data 125 can take a data format described as a constant tempo, and by reproducing the correction information as a tempo change, It is also possible to reproduce the performance sound when the sound generation control data 125 was recorded. This is particularly effective for songs for which it is desired to reproduce a natural performance. The above is the description of the AI quantize process.

[5. MIDI quantize processing]
Next, MIDI quantization processing will be described.

FIG. 15 is a flowchart showing MIDI quantization processing in one embodiment. The control unit 110 provides a quantization setting user interface (quantization processing setting UI) by displaying a screen described below on the display unit 130 (step S401), and waits until an instruction for quantization processing is input ( Step S403; No). When a quantize process instruction is input (step S403; Yes), the control unit 110 executes the quantize process according to the setting (step S405), and ends the MIDI quantize process.

FIG. 16 is a diagram showing an example of a quantize setting user interface. The quantization setting user interface (hereinafter sometimes referred to as interface MD) includes an area QS for setting the resolution of quantization and an execution button B4 for inputting instructions for quantization processing. In this example the interface MD contains a window RW for selecting the resolution to be set. If the quantization resolution is "1/8", the timing information is corrected so that the timing is aligned in units of eighth notes. This processing is similar to general quantization processing. The interface MD may include an area for setting conditions for sounds that are not subject to quantization. A sound not to be quantized may, for example, be moved by the same amount as the nearest quantized sound. Sounds excluded from the quantization process may be, for example, sounds whose length is less than half the resolution, or sounds whose velocity (volume) is lower than a predetermined value. Conversely, the interface MD may include an area in which sounds to be quantized can be specified.

According to the MIDI quantization process, when the rhythm fluctuates during the recording of the sound generation control data 125, it is possible to remove the fluctuation of the rhythm of the sound generation contained in the sound generation control data 125 and align it with the position of the beat. can. This is particularly effective for songs in which rhythm is important, such as dance music. The above is an explanation of MIDI quantization.

[6. Delay offset processing]
Next, the delay offset processing will be explained.

FIG. 17 is a flowchart showing delay offset processing in one embodiment. The control unit 110 provides an offset setting user interface (offset setting UI: second user interface) by displaying a screen described below on the display unit 130 (step S501), and an instruction for offset processing is input. (Step S503; No). When the offset processing instruction is input (step S503; Yes), the control unit 110 executes the offset processing according to the setting (step S505), and ends the delay offset processing.

FIG. 18 is a diagram showing an example of an offset setting user interface. The offset setting user interface (hereinafter sometimes referred to as interface DD) includes an area OS for setting an offset amount and an execution button B5 for inputting an offset processing instruction. In this example, interface DD includes window RW for selecting the amount of offset to be set. If the offset amount is "30", the sound generation control data 125 is changed by correcting the timing information so as to delay the entire timing by 30 ticks.

The interface DD may have an area for specifying the sound to be processed. By specifying the processing target sound, the timing information may be corrected for the target sound and a plurality of sounds following the target sound. If the sound to be processed is not specified, the first sound in the pronunciation control data 125 may be specified as the sound to be processed. Two examples of the case where the sound to be processed is designated and corrected will be described with reference to FIGS. 19 and 20. FIG.

19 and 20 are diagrams for comparing and explaining the sound generation control data before and after the delay offset. 19 and 20 both show the relationship between the timing information of the sound generation control data 125 and the position of beats "at the time of beat detection" and "after offset". "At time of beat detection" indicates the sound generation control data 125 when beats are detected in the AI quantize process. "After offset" indicates the pronunciation control data 125 after the ornamental tone NTs has been designated as the sound to be processed and the offset processing has been performed. Since it is assumed that the grace note is aligned with the beat position by the offset processing, it is assumed that the offset amount is specified so as to move the grace note NTs to the beat position DBT6.

In the first example shown in FIG. 19, the timing information is corrected so that the grace note NTs moves to the beat position DBT6 according to the offset amount. The timing information of all of the plurality of tones following the ornamental tone NTs is corrected by the same offset amount. At this time, the position of the correction information may be further changed. Changing the position of the correction information corresponds to changing the data section corrected by the correction information. The amount by which the position of the correction information is changed may correspond to the amount of offset.

In the second example shown in FIG. 20, the timing information is similarly corrected so that the grace note NTs moves to the beat position DBT6. On the other hand, for a plurality of tones following the ornamental tone NTs, the timing information is corrected such that the further away from the ornamental tone NTs, the smaller the offset amount. At this time, the timing information may be corrected only for sounds within a predetermined number of beats (for example, within four beats) from the grace note NTs.

[7. Data reproduction processing]
Next, a process of reading the sound generation control data 125 generated as described above (for example, the “post-combined” sound generation control data 125 shown in FIG. 14) and reproducing a sound waveform signal in the sound source section 830 will be described. . Portable device 10 outputs a sound generation instruction signal based on sound generation control data 125 to electronic musical instrument 80 so that sound waveform signal is reproduced in sound source unit 830 . When the portable device 10 receives an instruction from the user to start reproducing the sound generation control data 125, the portable device 10 starts reproducing the data. In the data reproduction process, modes for reproducing the sound generation control data 125 include an auto mode and a control mode.

In the auto mode, the sound generation control data 125 is read out at a predetermined tempo, and the tempo is corrected using the correction information set corresponding to each beat, thereby substantially correcting the position of the beat and reproducing the sound waveform. This mode realizes signal playback and reproduces the performance sound when the data was recorded. If the correction information is not used when reading the sound generation control data 125, as shown in FIG. By changing the reading speed, it is possible to reproduce the performance sound when the data was recorded. When using sounding control data 125 that has not been subjected to AI quantization processing, the sounding control data 125 is read at a predetermined tempo because correction information is not included.

The control mode is a mode for reading out the sound generation control data 125 in response to a progression instruction signal that advances the beat, and reproducing the sound when played at a speed corresponding to the signal. By instructing the progression of beats by the progression instruction signal, the timing at which each beat starts is controlled in reading the sound generation control data 125, and the tempo between beats is controlled based on the beats instructed in the past. be. The advance instruction signal is generated according to the user's motion, for example, according to the timing of changing the direction of swinging the portable device 10 like a baton. Control unit 110 can specify the positions of beats arranged at regular intervals in sound generation control data 125 from tick values if the beats are in the MIDI format. The data position to which the correction information is added in the pronunciation control data 125 may be specified as the beat position.

As described above, it may be desirable to correct the ornaments so that they match the beat position. Ornaments are often positioned slightly before the position of the beat. In such a case, it is desirable that the ornamented tone present at the position of the beat and the ornamental note present before it are reproduced continuously. On the other hand, when operating in the control mode, since the pronunciation control data 125 is read with the positions of beats as delimiters, the ornament sounds are reproduced at an early timing, and the ornament sound and the ornamented sound are reproduced with a large gap between them. situation arises. Assuming such a case, by correcting the timing information so that the ornament is positioned at the position of the beat, it is possible to reproduce the ornament and the ornamented tone as one unit without separating them. can.

FIG. 21 is a flowchart showing data reproduction processing in one embodiment. The control unit 110 provides a playback user interface (playback UI) by displaying a screen described below on the display unit 130 (step S801), and waits until an instruction to start data playback is input (step S803). ; No).

When an instruction to start data reproduction is input (step S803; Yes), control unit 110 controls sound generation by reproducing sound generation control data 125 until an instruction to stop data reproduction is input (step S821; No). Execute the process. When the auto mode is selected as the reproduction mode (step S811; Yes), the control unit 110 executes sound generation control processing in the auto mode (step S813). On the other hand, when the auto mode is not selected, that is, when the control mode is selected as the reproduction mode (step S813; No), sound generation control processing in the control mode is executed (step S815). When an instruction to stop data reproduction is input (step S821; Yes), control unit 110 terminates the data reproduction process.

FIG. 22 is a diagram showing an example of a playback user interface. The playback user interface (hereinafter sometimes referred to as interface PD) includes a switch button PMB for switching between auto mode and control mode. The switch button PMB includes an image that indicates whether it is operating in auto mode or in control mode. It may be selected so that the mode is switched each time the switching button PMB is operated, or one of the modes (for example, the control mode) is switched while the switching button PMB is being operated (while the switching button PMB is being touched). It may be selected.

When operating in auto mode, a stop button B6 is displayed to stop the progress of the song. In the control mode, if the user stops swinging the mobile device 10, the song will stop progressing. Therefore, the stop button B6 does not have to be displayed as shown in FIG. Operating the stop button B6 corresponds to inputting an instruction to stop data reproduction (step S821; Yes). good.

Thus, according to the data change method of one embodiment, the sound generation control data 125 recorded according to the performance can be changed to data that can be used in a plurality of different reproduction methods such as the auto mode and the control mode. can be done. It is also possible to change the sound generation control data 125 by selecting an appropriate change method depending on the song.

[8. Modification]
The present disclosure is not limited to the embodiments described above, and includes various other modifications. For example, the above-described embodiments have been described in detail in order to explain the present disclosure in an easy-to-understand manner, and are not necessarily limited to those having all the described configurations. Part of the configuration of each embodiment can be added, deleted, or replaced with another configuration. Some modifications will be described below.

(1) In the AI quantization process, correction information was added for each beat, but may be added for every two or more beats, or may be added for each sound. .

(2) The interface SD allows selection of a process of analyzing an image of a musical score, extracting bar lines, musical notes, performance symbols, etc., and correcting the timing information of the sound generation control data 125 based on the extracted information. good too.

(3) When the portable device 10 executes the data reproduction method, the progress instruction signal is generated by shaking the portable device 10 in the control mode. The advance instruction signal may be generated by shaking the sensor terminal. The sensor terminal includes, for example, an acceleration sensor, a gyro sensor, etc. as a configuration corresponding to the sensor unit 150, and has a function for measuring the movement of the sensor terminal and a function for connecting to the interface 190 of the mobile device 10. It's fine if you do. When a sensor terminal is used, the portable device 10 may be realized by a stationary desktop personal computer or the like instead of a portable device. Some or all of the above-described functions implemented in the mobile device 10 may be implemented by information processing devices such as one or more servers connected via the network NW. For example, when the sound control data 125 is recorded in the recording mode, the portable device 10 transmits the sound control data 125 to the server. The server may execute processing for changing the sound control data 125 and transmit the changed sound control data 125 to the portable device 10 .

[9. Overview of Solution]
According to one embodiment of the present disclosure described above, it is possible to describe the configuration shown below.

A data modification method of the present disclosure provides a selection user interface for selecting a modification method to be applied to pronunciation control data defining pronunciation timing information from a plurality of modification methods including a first modification method and a second modification method. and correcting the timing information when applying the first change method is selected, and providing correction information corresponding to the amount of correction of the timing information in the data section for a predetermined data section. modifying the pronunciation control data by adding, and correcting the timing information based on beat positions according to a predetermined tempo when applying the second modification method is selected; modifying the pronunciation control data.

The data change method of the present disclosure may provide a recording user interface for setting a tempo and instructing performance recording, and when the start of performance recording is instructed, beat information is recorded at the set tempo. The sound control data may be recorded based on performance information input while the beat information is being provided, and when an instruction to stop the performance recording is given, the recorded sound control data may be A selection user interface may be provided for selecting the modification method to be applied to.

In the data modification method of the present disclosure, the plurality of modification methods may further include a third modification method, and when applying the third modification method is selected, specified predetermined pronunciation timing information and timing information of a plurality of pronunciations following the predetermined pronunciation may be changed to change the pronunciation control data.

The data modification method of the present disclosure detects the positions of a plurality of beats based on pronunciation control data that defines timing information for pronunciation recorded at a predetermined tempo, and detects the relationship between the positions of the plurality of beats and the tempo. and modifying the sound generation control data by adding correction information to a predetermined data section according to the amount of correction of the timing information in the data section.

The data modification method of the present disclosure may further include providing beat information at a predetermined tempo, and recording the sound generation control data based on performance information input while providing the beat information. , the correction information may be added corresponding to the timing of providing the beat information.

Detecting the positions of the plurality of beats may include converting the pronunciation control data into audio data and detecting the positions of the plurality of beats based on the audio data.

The data modification method of the present disclosure may further include providing a first user interface for correcting the detected beat position, wherein modifying the pronunciation control data comprises: and correcting the timing information based on the relationship between the positions of the plurality of beats and the set tempo after being corrected via the method.

According to the data modification method of the present disclosure, the recorded pronunciation control data or the audio data and the beat positions corrected via the first user interface may be associated with each other and registered in a database. good.

The data modification method of the present disclosure may provide a second user interface for correcting the timing information of the predetermined pronunciation in the pronunciation control data, and the predetermined pronunciation instructed through the second user interface. The sound control data may be further changed by correcting the timing information of , and further correcting the timing information of a plurality of sounds following the predetermined sound.

The data change method of the present disclosure may correct the timing information of the predetermined pronunciation instructed via the second user interface, and when further correcting the timing information of a plurality of pronunciations following the predetermined pronunciation, , the sound generation control data may be further changed by further changing the data section targeted for the correction information.

A data reproduction method of the present disclosure is a data reproduction method for reproducing a sound waveform signal by a sound source unit based on the sound generation control data changed by the data change method described above, wherein the sound waveform signal is transmitted to the sound source unit providing a third user interface for selecting a method of generating a pronunciation instruction signal for reproducing from a plurality of reproduction methods including a first reproduction method and a second reproduction method, wherein the first reproduction method is selected read the sound generation control data according to a predetermined tempo, generate the sound generation instruction signal using the correction information, and advance the beat when the second reproduction method is selected. reading the sound control data in the data section corresponding to the sound generation instruction signal to generate the sound generation instruction signal.

The data modification method of the present disclosure provides a recording user interface for instructing recording of a performance, and when an instruction to start recording the performance is given, the method defines pronunciation timing information based on input performance information. When the control data is recorded and the stop of the performance recording is instructed, a change method to be applied to the recorded sound generation control data is selected from a plurality of change methods including a first change method and a second change method. providing a selection user interface for playing, modifying the sound generation control data based on the selected modification method, and reproducing performance sounds based on the modified sound generation control data.

According to the present disclosure, it can also be used as a program for causing a computer to execute the data modification method or the data reproduction method, and can be used as a data modification device for executing the data modification method and a data reproduction device for executing the data reproduction method. can also That is, at least part of the portable device 10 can function as a data modification device or a data reproduction device.

10: portable device, 110: control unit, 120: storage unit, 121: program, 123: performance recording data, 125: sound generation control data, 130: display unit, 140: operation unit, 150: sensor unit, 170: speaker, 180: communication unit, 190: interface, 80: electronic musical instrument, 810: performance operator, 830: sound source unit, 870: speaker, 890: interface, 90: data management server, 910: control unit, 920: storage unit, 980 : Communications department

Claims

providing a selection user interface for selecting a modification method to be applied to pronunciation control data defining pronunciation timing information from a plurality of modification methods including a first modification method and a second modification method;
By correcting the timing information and adding correction information to a predetermined data section according to the amount of correction of the timing information in the data section when applying the first change method is selected , changing the pronunciation control data,
modifying the pronunciation control data by correcting the timing information based on the position of the beat according to a predetermined tempo when applying the second modification method is selected;
data modification methods, including
providing a recording user interface for setting the tempo and directing performance recording;
When the start of the performance recording is instructed, providing beat information at the set tempo,
recording the sound generation control data based on performance information input while providing the beat information;
2. The data modification method according to claim 1, wherein said selection user interface for selecting a modification method to be applied to said recorded sound generation control data is provided when said stop of performance recording is instructed.
The plurality of change methods further includes a third change method,
When applying the third change method is selected, the pronunciation control is performed by correcting the timing information of a designated predetermined pronunciation and correcting the timing information of a plurality of pronunciations following the predetermined pronunciation. 3. The data modification method according to claim 1, wherein the data is modified.
Detecting positions of a plurality of beats based on pronunciation control data defining pronunciation timing information recorded at a predetermined tempo,
correcting the timing information based on the relationship between the positions of the plurality of beats and the tempo;
changing the sound generation control data by adding correction information corresponding to a correction amount of the timing information in a predetermined data section to the data section;
data modification methods, including
Provides beat information at a given tempo,
recording the sound generation control data based on performance information input while the beat information is being provided;
further comprising
5. The data modification method according to claim 4, wherein said correction information is added corresponding to the timing of providing said beat information.
Detecting the positions of the plurality of beats includes:
converting the sound generation control data into audio data;
detecting positions of a plurality of beats based on the audio data;
including,
6. The data change method according to claim 4 or 5.
further comprising providing a first user interface for correcting the detected beat position;
Changing the pronunciation control data includes correcting the timing information based on a relationship between the positions of the plurality of beats after correction via the first user interface and a set tempo. 7. The data change method according to any one of claims 4 to 6.
8. The method according to claim 7, further comprising: registering the recorded pronunciation control data or the audio data in a database in association with the beat position corrected via the first user interface. data change method.
providing a second user interface for correcting predetermined pronunciation timing information in the pronunciation control data;
Further changing the pronunciation control data by correcting the timing information of the predetermined pronunciation instructed through the second user interface and further correcting the timing information of a plurality of pronunciations following the predetermined pronunciation. ,
9. The data modification method according to any one of claims 4 to 8, further comprising:
When correcting the timing information of the predetermined pronunciation instructed via the second user interface, and further correcting the timing information of a plurality of pronunciations following the predetermined pronunciation, the data section subject to the correction information 10. The data modification method according to claim 9, further modifying the pronunciation control data by further modifying the .
11. A data reproduction method for reproducing a sound waveform signal by a tone generator based on the sound generation control data changed by the data change method according to any one of claims 4 to 10,
providing a third user interface for selecting from a plurality of reproduction methods including a first reproduction method and a second reproduction method a method of generating a pronunciation instruction signal for causing the sound source unit to reproduce a sound waveform signal;
when the first reproduction method is selected, reading the sound generation control data according to a predetermined tempo and generating the sound generation instruction signal using the correction information;
generating the sound generation instruction signal by reading the sound generation control data of the data section corresponding to the progress instruction signal for advancing the beat when the second reproduction method is selected;
data recovery methods, including;
providing a selection user interface for selecting a modification method to be applied to pronunciation control data defining pronunciation timing information from a plurality of modification methods including a first modification method and a second modification method;
By correcting the timing information and adding correction information to a predetermined data section according to the amount of correction of the timing information in the data section when applying the first change method is selected , changing the pronunciation control data,
modifying the pronunciation control data by correcting the timing information based on the position of the beat according to a predetermined tempo when applying the second modification method is selected;
A program that causes a computer to run
providing a recording user interface for directing performance recording;
when the start of performance recording is instructed, recording sound generation control data defining sound generation timing information based on the input performance information;
A selection user interface for selecting a modification method to be applied to the recorded sound generation control data from a plurality of modification methods including a first modification method and a second modification method when the stop of the performance recording is instructed. to provide
changing the pronunciation control data based on the selected change method;
reproducing performance sounds based on the changed sound generation control data;
data modification methods, including