JP2020064177A - Information processing apparatus and program - Google Patents

Abstract

To make a performance sound of a player and a musical score sound on a musical score correspond to each other even if a performance speed changes.SOLUTION: A recognition part 12 recognizes, from performance sound data representing a performance sound absorbed by a sound collection part 11, a pitch and sound pressure of the performance sound. An identification part 13 identifies, if the degree of matching between a pitch of a first musical score sound included in musical score data and the pitch of the performance sound recognized from the performance sound data exceeds a threshold, a start timing of the first performance sound corresponding to the first musical score sound. Further, the identification part 13 identifies, if the degree of matching between a pitch of a second musical score sound following the first musical score sound of the musical score data and a pitch of a performance sound following the first performance sound recognized from the performance sound data exceeds a threshold, an end timing of the first performance sound corresponding to the first musical score sound.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technique for associating a performance sound of a performer with a score sound on a score.

  In order to evaluate the performance of the performer, it is necessary to specify which position on the score the performer is playing (for example, refer to Patent Document 1).

JP, 2007-133199, A

  With the spread of mobile terminals such as smart phones in recent years, there is a demand for a service that collects a performance of a piano or the like by this type of terminal and evaluates the performance. In such a service, for example, the musical score display is transited in accordance with the performance speed determined on the terminal, and the user plays in accordance with the progress of the musical score, so that a musical sound corresponding to the same timing is produced. A method of comparing with the sound on the score can be considered.

  However, in such a method, since the performance speed is predetermined, there is a problem that the evaluation cannot be performed correctly when the performance is performed by the user.

  Therefore, an object of the present invention is to make the performance sound of the performer correspond to the score sound on the score even when the performance speed changes.

  In order to solve the above-mentioned problems, the present invention, in the performance sound data indicating the collected performance sound, a recognition unit that recognizes the pitch of the performance sound, the pitch of the score sound indicated by the score data, and the recognition. A specifying unit that specifies a correspondence relationship between the musical score sound and the musical performance sound based on the degree of coincidence with the pitch of the musical performance sound recognized in the musical sound data by the unit, and the specifying unit is included in the musical score data. When the degree of coincidence between the pitch of the first musical note sound and the pitch of the musical performance sound recognized in the musical performance sound data exceeds a threshold value, the first musical sound corresponding to the first musical note sound. Is specified, the pitch of the second musical note sound following the first musical note sound in the musical score data, and the pitch of the musical performance sound following the first musical sound recognized in the musical sound data. If the degree of coincidence with the Identifying the end timing of the first performance sound corresponding to an information processing apparatus according to claim.

  An evaluation unit may be provided that evaluates the performance by comparing the musical score sound and the performance sound that are in the correspondence relationship specified by the specification unit.

  The specifying unit continuously and temporally compares the pitch of one score sound in the score data with the pitch of the performance sound recognized in the performance sound data a predetermined number of times, and compares the respective pitches. If the degree of coincidence exceeds the threshold value, it may be specified that the musical score sound and the performance sound correspond to each other.

  The predetermined number of times may be different depending on the length of the score sound, the tempo in the score data, or the tempo of performance.

  The start timing or the end timing of the performance sound corresponding to the musical score sound by the specifying unit depending on whether the musical sound data of the same pitch is continuous in the musical score data or the musical sound of different pitches is continuous. The method of specifying the may be different.

  When the first musical note sound and the second musical note sound of the same pitch are consecutive in the musical score data, the specifying unit matches the pitches of the first musical note sound and the second musical note sound. In the section of the performance sound whose degree exceeds the threshold value, the first performance sound corresponding to the first musical score sound is specified based on the increase / decrease in the sound pressure of the performance sound, and the first musical sound corresponding to the second musical sound is identified. The second performance sound may be specified.

  When the first musical note sound and the second musical note sound with different pitches are continuous in the musical score data, the specifying unit determines the pitch of the first musical note sound included in the musical score data and the performance sound data. When the degree of coincidence with the pitch of the performance sound recognized in step S1 exceeds a threshold value, the start timing of the first performance sound corresponding to the first musical score sound is specified, and the first timing is specified in the musical score data. When the degree of coincidence between the pitch of the second musical score sound following the musical score sound and the pitch of the musical performance sound following the first musical performance sound recognized in the musical performance data exceeds a threshold value, the first musical sound The end timing of the first performance sound corresponding to the musical score sound may be specified.

  The specifying unit includes a pitch of a second musical note following the first musical note in the musical score data and a pitch of a musical note following the first musical note recognized in the musical note data. If the period in which the degree of coincidence does not exceed the threshold has passed a predetermined period, it may be determined that the first performance sound corresponding to the first musical score sound has ended.

  The specifying unit determines that a predetermined number of times continues when a period in which the degree of coincidence between the pitch of the second musical score sound and the pitch of the performance sound following the first performance sound does not exceed a threshold has passed a predetermined number of times. By comparing a combination of pitches of a plurality of musical score sounds that are temporally continuous in the musical score data with a combination of pitches of a plurality of musical performance sounds that are continuously recognized in time in the musical performance sound data, You may make it specify the correspondence of a musical score sound and a performance sound.

  Further, according to the present invention, in a computer, a recognition unit for recognizing a pitch of a performance sound in performance sound data indicating a collected performance sound, a pitch of a music score indicated by the score data, and the recognition unit are provided. A specifying unit that specifies a correspondence relationship between the musical score sound and the musical performance sound based on the degree of coincidence with the pitch of the musical performance sound recognized in the musical performance sound data. When the degree of coincidence between the pitch and the pitch of the performance sound recognized in the performance sound data exceeds a threshold value, the start timing of the first performance sound corresponding to the first musical score sound is specified, In the score data, the degree of coincidence between the pitch of the second score sound following the first score sound and the pitch of the performance sound following the first performance sound recognized in the performance sound data has a threshold value. If it exceeds, the above-mentioned corresponding to the above-mentioned first musical note sound It provides a program for functioning as a specifying unit configured to specify the end timing of a performance sound.

  According to the present invention, even when the performance speed changes, the performance sound of the performer and the score sound on the score can be associated with each other.

It is a figure which shows an example of a structure of the performance evaluation system which concerns on one Embodiment of this invention. It is a block diagram which shows an example of the hardware constitutions of a user terminal. It is a block diagram showing an example of functional composition of a user terminal. It is a flow chart which shows an example of basic operation of a user terminal. It is a flow chart which shows an example of specific processing of a user terminal. It is a figure explaining an example of the correspondence of musical score sound and performance sound.

[Constitution]
FIG. 1 is a diagram showing an example of a performance evaluation system 1 of the present embodiment. The performance evaluation system 1 includes a user terminal 10 such as a smart phone or a tablet, and a server device 20. The network 2 communicably connects the user terminal 10 and the server device 20. The network 2 is, for example, a LAN (Local Area Network), a WAN (Wide Area Network), or a combination thereof, and includes a wired section or a wireless section. Note that FIG. 1 shows one user terminal 10 and one server device 20, but there may be a plurality of each.

  When playing a musical instrument such as a piano, the user puts the user terminal 10 close to himself and activates the performance evaluation program installed in the user terminal 10. When the user's performance starts, the user terminal 10 evaluates the superiority or inferiority of the performance by identifying which sound on the score corresponds to the sound played by the user and comparing the two. The server device 20 provides the performance evaluation program and data used during execution of the program to the user terminal 10, or acquires and stores the evaluation result and the like obtained by executing the performance evaluation program from the user terminal 10. To do. When the performance evaluation program is executed, the user terminal 10 and the server device 20 may operate in cooperation with each other, or the user terminal 10 alone may operate. In this embodiment, an example in which the user terminal 10 operates alone will be described.

  FIG. 2 is a diagram illustrating an example of the hardware configuration of the user terminal 10. The user terminal 10 is physically configured as a computer device including a processor 1001, a memory 1002, a storage 1003, a communication device 1004, an input device 1005, an output device 1006, and a bus connecting these. In the following description, the word "device" can be read as a circuit, a device, a unit, or the like. The hardware configuration of the user terminal 10 may be configured to include one or a plurality of each device illustrated in the figure, or may be configured not to include some devices.

  Each function in the user terminal 10 causes a predetermined software (program) to be loaded on hardware such as the processor 1001 and the memory 1002, so that the processor 1001 performs an arithmetic operation, controls communication by the communication device 1004, and controls the memory 1002. Also, it is realized by controlling at least one of reading and writing of data in the storage 1003.

  The processor 1001 operates an operating system to control the entire computer, for example. The processor 1001 may be configured by a central processing unit (CPU) including an interface with peripheral devices, a control device, an arithmetic device, a register, and the like. Further, for example, a baseband signal processing unit, a call processing unit, etc. may be realized by the processor 1001.

  The processor 1001 reads a program (program code), a software module, data, and the like from at least one of the storage 1003 and the communication device 1004 into the memory 1002, and executes various processes according to these. As the program, a program that causes a computer to execute at least a part of the operations described below is used. The functional block of the user terminal 10 may be implemented by a control program stored in the memory 1002 and operating in the processor 1001. Various types of processing may be executed by one processor 1001, but may be executed simultaneously or sequentially by two or more processors 1001. The processor 1001 may be implemented by one or more chips. The program may be transmitted from the network 2 to the user terminal 10 via an electric communication line.

  The memory 1002 is a computer-readable recording medium, and is configured by at least one of a ROM (Read Only Memory), an EPROM (Erasable Programmable ROM), an EEPROM (Electrically Erasable Programmable ROM), a RAM (Random Access Memory), and the like. May be done. The memory 1002 may be called a register, a cache, a main memory (main storage device), or the like. The memory 1002 can store an executable program (program code), a software module, etc. for implementing the method according to the present embodiment.

  The storage 1003 is a computer-readable recording medium, for example, an optical disk such as a CD-ROM (Compact Disc ROM), a hard disk drive, a flexible disk, a magneto-optical disk (for example, a compact disk, a digital versatile disk, a Blu-ray disc). At least one of a (registered trademark) disk, a smart card, a flash memory (for example, a card, a stick, and a key drive), a floppy (registered trademark) disk, a magnetic strip, or the like. The storage 1003 may be called an auxiliary storage device. The storage 1003 stores a performance evaluation program and a score data group described later.

  The communication device 1004 is hardware (transmission / reception device) for performing communication between computers via at least one of a wired network and a wireless network, and is also called, for example, a network device, a network controller, a network card, a communication module, or the like. The communication device 1004 includes a high frequency switch, a duplexer, a filter, a frequency synthesizer, and the like in order to realize at least one of frequency division duplex (FDD) and time division duplex (TDD), for example. May be composed of For example, the transmission / reception antenna, the amplifier unit, the transmission / reception unit, the transmission line interface, and the like may be realized by the communication device 1004. The transmitter / receiver may be implemented by physically or logically separating the transmitter and the receiver.

  The input device 1005 is an input device (for example, a keyboard, a mouse, a microphone, a switch, a button, a sensor, etc.) that receives an input from the outside. The output device 1006 is an output device (for example, a display, a speaker, an LED lamp, etc.) that outputs to the outside. The input device 1005 and the output device 1006 may be integrated (for example, a touch panel).

  The respective devices such as the processor 1001 and the memory 1002 are connected by a bus for communicating information. The bus may be configured by using a single bus, or may be configured by using a different bus for each device.

  The user terminal 10 includes hardware such as a microprocessor, a digital signal processor (DSP), an ASIC (Application Specific Integrated Circuit), a PLD (Programmable Logic Device), and an FPGA (Field Programmable Gate Array). It may be configured, and a part or all of each functional block may be realized by the hardware. For example, the processor 1001 may be implemented using at least one of these hardware.

  FIG. 3 is a block diagram showing an example of the functional configuration of the user terminal 10. As shown in FIG. 3, in the user terminal 10, functional blocks of a sound collection unit 11, a recognition unit 12, a specification unit 13, and an evaluation unit 14 are realized.

  The sound collection unit 11 collects a sound (called a performance sound) when the user plays an instrument such as a piano.

  The recognition unit 12 recognizes the pitch (pitch) and intensity (sound pressure) of the performance sound in the performance sound data indicating the performance sound collected by the sound collection unit 11.

  The specifying unit 13 stores score data. The musical score data includes information indicating a sound, such as sounding timing, pitch, sound pressure, and length of each sound constituting the music. Each sound indicated by the score data is called a score sound. The specifying unit 13 correlates the musical score sound with the musical performance sound based on the degree of coincidence between the musical pitch sound pitch indicated by the musical score data and the musical pitch of the musical performance sound recognized by the recognizing unit 12 in the musical performance sound data. Specify. The specifying unit 13 specifies the correspondence between the performance sound and the score sound based on the change in pitch, that is, specifies the correspondence relationship between the performance sound and the score sound without depending on the length of the sound. Even if the user's performance speed changes, the performance sound and the musical score sound can be associated with each other.

  The evaluation unit 14 evaluates the performance of the user by comparing the musical score sound and the performance sound that are in the correspondence relationship specified by the specification unit 13. Further, the evaluation unit 14 outputs the evaluation result as an index value (for example, score or level) of performance superiority or inferiority. The output form at this time may be any form such as a display form or a voice output form as long as it can notify the user.

  As described above, when the user terminal 10 and the server device 20 work together, the functional blocks illustrated in FIG. 3 are realized by the user terminal 10 and the server device 20. For example, an example is conceivable in which the user terminal 10 realizes the sound collecting unit 11 and the recognizing unit 12, and the server device 20 realizes the specifying unit 13 and the evaluating unit 14, but the present invention is not limited to this example.

[motion]
The operation of the user terminal 10 will be described with reference to FIGS. 4 to 6. The procedure of each process shown in FIGS. 4 and 5 is described in the performance evaluation program installed in the user terminal 10. In FIG. 4, the user operates the user terminal 10 to activate the performance evaluation program, and designates a song to be played by himself. When the user performs an operation to start playing on the user terminal 10, the sound collecting unit 11 of the user terminal 10 collects the performance sound of the musical instrument played by the user (step S11: sound collection processing).

  Next, the recognition unit 12 recognizes the pitch and sound pressure of the performance sound in the performance sound data indicating the performance sound collected by the sound collection unit 11 (step S12: recognition processing).

  Here, FIG. 6 is a diagram illustrating an example of a correspondence relationship between a musical score sound and a performance sound. In FIG. 6, (A) shows an example of musical score sound data in which the vertical axis represents pitch and the horizontal axis represents time. (B) shows an example of performance sound data in the section of the musical score corresponding to (A), where the vertical axis represents the pitch and the horizontal axis represents the time. (C) shows an example of performance sound data in the section corresponding to (B), where the vertical axis represents sound pressure and the horizontal axis represents time. For example, in the example of the score data of FIG. 6A, the pitch C4 is in the first section (time T0 to T1), the pitch G4 is the second section (time T1 to T2), and the third section (time T2 to T2). At T3), the pitch is E4.

  The recognition unit 12 analyzes the frequency of the performance sound data at a sufficiently short time interval (for example, 20 msec) to detect the pitch. When a pitch within a range determined as the same pitch is detected continuously for a predetermined number of times (for example, 20 msec × 4 times) in time, the recognition unit 12 recognizes the pitch as the pitch of the performance sound. To do. At this time, the timing at which a pitch within the range determined as the same pitch is first detected is recognized as the start timing of the performance sound of that pitch. By such processing, for example, in the example of FIG. 6B, the pitch C4 is recognized in the first section (time T0 to T1), and the pitch G4 is recognized in the second section (time T1 to T2). The pitch E4 is recognized in the third section (time T2 to T3).

  The recognition unit 12 also recognizes the pitch of the performance sound data and the sound pressure thereof. When the musical instrument is a keyboard musical instrument such as a piano, the sound pressure when the user's finger presses the keyboard is the highest, and thereafter, the sound pressure is attenuated with the passage of time. For this reason, as illustrated in FIG. 6C, in each of the first section, the second section, and the third section, the sound pressure at the beginning of the section is the highest and gradually decreases thereafter.

  In FIG. 4, the identifying unit 13 reads out the musical score data of the musical composition designated by the user from the storage 1003, and the pitch of the musical score sound indicated by the musical score data and the performance sound recognized in the performance sound data by the recognition unit 12. The correspondence between the musical score sound and the performance sound is specified based on the degree of coincidence with the pitch of (step S13: specifying process).

  Here, FIG. 5 is a flowchart showing an example of the identification processing in step S13. When the user gives an instruction to start playing and the collection of playing sounds starts, the identifying unit 13 first determines whether an attack has occurred (step S131). Here, the attack means that the sound pressure of the performance sound recognized by the recognition unit 12 exceeds the first threshold when the keyboard of the piano is tapped, and the difference between the sound pressures continuously detected is the first. 2 It means exceeding a threshold value. The first threshold is the pitch threshold Th illustrated in FIG. 6C, which is a reference value for determining whether or not the keyboard of the piano has been tapped. The second threshold value is a reference value for determining whether or not the keyboard of the piano is tapped based on the sound pressure difference in a time interval (for example, 20 msec) that is one unit when the recognition unit 12 recognizes the sound pressure. Is.

  When an attack occurs (step S131; Yes), the identifying unit 13 determines whether the pitch of the performance tone recognized by the recognizing unit 12 in the above-described procedure and the pitch of the first musical score of the musical score data match. It is determined whether or not a certain threshold is exceeded (hereinafter expressed as whether or not the pitch of the performance sound matches the pitch of the first score sound of the score data) (step S132). When determining that the pitch of the performance sound matches the pitch of the first score sound in the score data, the identifying unit 13 determines that the user has started playing the musical instrument (step S133).

  The specifying unit 13 detects a first musical sound on the musical score data (referred to as a first musical sound) and a subsequent musical sound (that is, a second musical sound on the musical score data and referred to as a second musical sound). It is determined whether the pitches are the same (step S134). When the first pitch sound and the second pitch sound have the same pitch, that is, the same sound continues in the score data (step S134; same sound), the specifying unit 13 causes the identifying sound to be recognized in steps S131 to S132. It is determined whether or not a certain time has elapsed from the start timing of the (first playing sound) (step S135). When a certain time has elapsed from the start timing of the first performance sound (step S135; Yes), the specifying unit 13 detects that the sound pressure of the performance sound recognized by the recognition unit 12 exceeds the threshold Th and is continuously detected. It is determined whether or not the difference in sound pressure exceeds the second threshold value, that is, whether or not an attack has occurred (step S136).

  When an attack occurs (step S136; Yes), the specifying unit 13 determines the pitch of the second musical score sound that is the same as the first musical score sound, and the performance sound (second sound) recognized by the recognition unit 12 at the time of this attack. It is determined whether or not the pitch of the performance sound) matches (step S137). When the pitch of the second musical note sound and the pitch of the second musical performance sound match (step S137; Yes), the identifying unit 13 ends the first musical sound at the timing of the matching. It is the end timing, and it is determined that it is the start timing at which the second performance sound is started (step S138). In this way, when the musical score data of the same pitch continues in the musical score data, the specifying unit 13 determines the sound pressure of the musical performance sound in the musical performance sound section in which the degree of coincidence with the musical pitch of the musical score sound exceeds the threshold value. Based on the increase / decrease, the first performance sound corresponding to the first music sound is specified, and the second performance sound corresponding to the second music sound is specified.

  Then, the specifying unit 13 sets the sound of interest as the next musical sound on the musical score data (step S139), and transitions to the processing of step S134. That is, the specifying unit 13 treats the second musical note sound in the musical score data as the above-mentioned first musical note sound and the third musical note sound in the musical score data as the above-mentioned second musical note sound in the processing of step S134 and thereafter. handle.

  In step S136, even when the attack has not occurred (step S136; No) and the timeout has occurred after a predetermined period of time (step S140; Yes), the specifying unit 13 determines the timing of the timeout. It is determined that it is the end timing when the first performance sound ends and the start timing when the second performance sound starts (step S138). The predetermined period is, for example, a period equal to or longer than the length of the first musical score sound (for example, the length of the first musical score sound × 1.2). Then, the specifying unit 13 sets the sound of interest as the next musical sound on the musical score data (step S139), and transitions to the processing of step S135.

  Further, in step S137, when the pitch of the second musical score sound that is the same as the first musical score sound and the pitch of the performance sound do not match (step S137; No), a predetermined period elapses and a time-out occurs. Also (step S140; Yes), the specifying unit 13 determines that the time-out timing is the end timing when the first performance sound is ended and the start timing when the second performance sound is started. (Step S138). The predetermined period is, for example, a period equal to or longer than the length of the first musical score sound (for example, the length of the first musical score sound × 1.2). That is, the identifying unit 13 determines the degree of coincidence between the pitch of the second musical note sound following the first musical note sound in the musical score data and the pitch of the musical performance sound following the first musical sound recognized in the musical sound data. If the period in which the value does not exceed the threshold has exceeded the predetermined period, it is determined that the first performance sound corresponding to the first musical score sound has ended. Then, the specifying unit 13 sets the sound of interest as the next musical sound on the musical score data (step S139), and transitions to the processing of step S135.

  In step S135 described above, the reason why the attack of the first performance sound and the attack of the second performance sound should be separated to some extent in time is the reason why the attack of the first performance sound should wait for a certain period of time before proceeding to step S136. is there.

  On the other hand, in step S134, when the score sounds of different pitches are continuous in the score data (step S134; abnormal sound), the case where the score sounds of the same pitch as described above are continuous is specified by the specifying unit 13 Are different in the method of specifying the start timing or the end timing of the performance sound corresponding to the musical score sound. First, as in step S135, the identifying unit 13 determines whether or not a fixed time has elapsed from the start timing of the first performance sound (step S141). When a certain time has elapsed from the start timing of the first performance sound (step S141; Yes), the specifying unit 13 has passed the pitch of the second musical score sound and a certain time from the start timing of the first performance sound. It is then determined whether or not the pitch of the second performance sound recognized by the recognition unit 12 matches (step S142). When the pitch of the second musical score sound and the pitch of the second performance sound match (step S142; Yes), the specifying unit 13 ends the matching timing at which the first performance sound ends. It is the end timing, and it is determined that it is the start timing at which the second performance sound is started (step S138). Then, the specifying unit 13 sets the sound of interest as the next musical sound on the musical score data (step S139), and transitions to the processing of step S135.

  Also in step S142, when the pitch of the second musical score sound and the pitch of the second musical performance sound do not match (step S142; No), a predetermined period elapses and time-out occurs (step S143). Yes)), and the identifying unit 13 determines that the time-out timing is the end timing when the first performance sound ends and the start timing when the second performance sound starts (step S138). . This predetermined period is, for example, a period equal to or longer than the length of the second musical score sound (for example, the length of the second musical score sound × 1.2). Then, the specifying unit 13 sets the sound of interest as the next musical sound on the musical score data (step S139), and transitions to the processing of step S135.

  The specifying unit 13 performs the above processing on all the musical score sound data, and thereby the correspondence between the musical score sound and the performance sound is specified.

  Returning to the explanation of FIG. 4 again, the evaluation unit 14 evaluates the performance of the user by comparing the musical score sound and the performance sound that are in the correspondence relationship specified by the specification unit 13 (step S14: evaluation processing). This comparison is performed from the viewpoint of how faithfully the performance sound reproduces the musical score sound. Then, the evaluation unit 14 outputs the evaluation result as an index value of performance superiority or inferiority.

  According to the embodiment described above, the performance sound of the performer and the score sound on the score can be associated with each other even when the performance speed changes. Further, in the case of a musical instrument such as a piano, there is a possibility that the pitch of the performance sound cannot be correctly recognized because the sound pressure decreases after a certain time has passed from the beginning of the performance pronunciation, but according to the present embodiment. For example, since the end timing of the first performance sound is specified by detecting the second performance sound next to the first performance sound whose sound pressure has decreased, the length of the performance sound can be more accurately determined. It becomes possible to specify. A technique called DP matching (dynamic programming) is known as a technique related to tracking on a score, which is the object of the present embodiment, but the technique of the present embodiment is a simpler process than DP matching. Due to the procedure, the processing speed is fast. Therefore, according to the present embodiment, it becomes possible to evaluate the performance earlier during the performance.

[Modification]
The present invention is not limited to the above embodiments. The embodiment described above may be modified as follows. Further, the following two or more modified examples may be combined and implemented.
[Modification 1]
In a musical instrument, a plurality of sounds may be played at the same time. This is called a chord. When a chord is played, the recognizing unit 12 uses known techniques to recognize the pitch of the notes forming the chord together with their certainty (most probable first rank to Xth rank). When such a chord exists on the musical score, the identifying unit 13 determines the pitch of each of a plurality of notes (for example, the notes of pitches C4, G4, and E4) that compose the chord and the playing sound by the recognition unit 12. The pitch recognized in the first rank is compared. Then, when the pitch of the recognized first-order performance sound matches any of the plurality of notes that form the chord, the specifying unit 13 determines the performance sound corresponding to the score sound of the chord in the score data. Is started, and it is determined that the performance sound corresponding to the music sound preceding the chord in the music score data has ended.

[Modification 2]
In the embodiment, the recognition unit 12 detects the pitch of the performance sound at a certain time interval (for example, 20 msec), and is determined as the same pitch continuously for a predetermined number of times (for example, 20 msec × 4 times). When the pitch within the range is detected, the pitch is recognized as the pitch of the performance sound. This predetermined number of times may be a fixed value, or may differ depending on the length of the musical score sound, the tempo of the musical score data, or the tempo (performance speed) of the performance. For example, the recognizing unit 12 may increase the above-described predetermined number of times as the length of the musical score sound of interest becomes longer. Further, the recognition unit 12 may decrease the above-mentioned predetermined number of times as the tempo determined in the score data is faster. Further, since the specifying unit 13 can specify the tempo (performance speed) of the performance by specifying the start timing and the end timing of the performance sound, the recognition unit 12 determines that the above-mentioned predetermined number of times is set as the performance tempo is faster. May be less.

[Modification 3]
In the embodiment, the identifying unit 13 determines the pitch of the second musical note sound following the first musical note sound in the musical score data and the pitch of the musical performance sound following the first musical sound recognized in the musical sound data. When the period in which the degree of coincidence does not exceed the threshold has passed the predetermined period, it is determined that the first performance sound corresponding to the first musical score sound has ended. In the case where the period during which the degree of coincidence between the pitch of the second musical note sound and the pitch of the musical performance sound following the first musical performance sound does not exceed the threshold value has passed a predetermined time It may occur. For example, there is a case where the user fails to play a number of times in succession, and in such a case, the playing position in the score data may be unknown. In such a case, the specifying unit 13 determines the combination of the pitches of a plurality of musical score sounds that are temporally continuous in the musical score data and the pitches of a plurality of musical performance sounds that are recognized in the musical performance data continuously in time. The correspondence may be identified by identifying the correspondence between the musical score sound and the performance sound. Specifically, the recognition unit 12 divides the performance sound data into a plurality of performance sounds with the attack occurrence position as a boundary, and recognizes the pitch of each performance sound. The specifying unit 13 searches the score data before and after the playing position is unknown for a score sound that matches the pitch of the recognized performance sound, and further, matches the next performance sound with the pitch of the score sound. Determine whether to do. Accordingly, when a combination of a certain number or more of consecutive pitches matches, the identifying unit 13 determines that position in the score data as a performance position, and restarts the process of FIG. 5 described above.

[Modification 4]
Although the user terminal 10 such as a smartphone or a tablet is illustrated as an example of the information processing apparatus according to the present invention, the present invention can be applied to any information processing apparatus that realizes the functional blocks illustrated in FIG.
Further, in the present invention, the musical instrument used for performance is not limited to the piano exemplified in the embodiment.

[Other modifications]
Note that the block diagrams used in the description of the above embodiment show blocks of functional units. These functional blocks (components) are realized by an arbitrary combination of at least one of hardware and software. The method of realizing each functional block is not particularly limited. That is, each functional block may be realized by using one device physically or logically coupled, or directly or indirectly (for example, two or more devices physically or logically separated). , Wired, wireless, etc.) and may be implemented using these multiple devices. The functional blocks may be realized by combining the one device or the plurality of devices with software.

  Functions include judgment, decision, judgment, calculation, calculation, processing, derivation, investigation, search, confirmation, reception, transmission, output, access, solution, selection, selection, establishment, comparison, assumption, expectation, and observation. Broadcasting, notifying, communicating, forwarding, configuration, reconfiguring, allocating, mapping, assigning, etc., but not limited to these. I can't. For example, a functional block (component) that causes transmission to function is called a transmitting unit or a transmitter. In any case, as described above, the implementation method is not particularly limited.

  For example, the user terminal according to the embodiment of the present disclosure may function as a computer that performs the process of the wireless communication method of the present disclosure.

  The notification of information is not limited to the aspect / embodiment described in the present disclosure, and may be performed using another method. For example, the information is notified by physical layer signaling (for example, DCI (Downlink Control Information), UCI (Uplink Control Information)), upper layer signaling (for example, RRC (Radio Resource Control) signaling, MAC (Medium Access Control) signaling, It may be implemented by notification information (MIB (Master Information Block), SIB (System Information Block)), another signal, or a combination thereof. Further, the RRC signaling may be called an RRC message, and may be, for example, an RRC connection setup (RRC Connection Setup) message, an RRC connection reconfiguration message, or the like.

  Each aspect / embodiment described in the present disclosure is LTE (Long Term Evolution), LTE-A (LTE-Advanced), SUPER 3G, IMT-Advanced, 4G (4th generation mobile communication system), 5G (5th generation mobile communication). system), FRA (Future Radio Access), NR (new Radio), W-CDMA (registered trademark), GSM (registered trademark), CDMA2000, UMB (Ultra Mobile Broadband), IEEE 802.11 (Wi-Fi (registered trademark) )), IEEE 802.16 (WiMAX (registered trademark)), IEEE 802.20, UWB (Ultra-WideBand), Bluetooth (registered trademark), systems using other suitable systems, and extensions based on these. It may be applied to at least one of the next-generation systems. Further, a plurality of systems may be combined and applied (for example, a combination of at least one of LTE and LTE-A and 5G).

  The processing procedure, sequence, flowchart, etc. of each aspect / embodiment described in the present disclosure may be interchanged as long as there is no contradiction. For example, the methods described in this disclosure present elements of the various steps in a sample order, and are not limited to the specific order presented.

  Information and the like can be output from the upper layer (or lower layer) to the lower layer (or upper layer). Input / output may be performed via a plurality of network nodes.

  The input / output information and the like may be stored in a specific place (for example, a memory) or may be managed using a management table. Information that is input / output can be overwritten, updated, or added. The output information and the like may be deleted. The input information and the like may be transmitted to another device.

  The determination may be performed based on a value represented by 1 bit (0 or 1), may be performed based on a Boolean value (Boolean: true or false), or may be compared by numerical values (for example, a predetermined value). (Comparison with value).

Each aspect / embodiment described in the present disclosure may be used alone, in combination, or may be switched according to execution. Further, the notification of the predetermined information (for example, the notification of “being X”) is not limited to the explicit notification, and is performed implicitly (for example, the notification of the predetermined information is not performed). Good.
Although the present disclosure has been described in detail above, it is obvious to those skilled in the art that the present disclosure is not limited to the embodiments described in the present disclosure. The present disclosure can be implemented as modified and changed modes without departing from the spirit and scope of the present disclosure defined by the description of the claims. Therefore, the description of the present disclosure is for the purpose of exemplifying description, and does not have any restrictive meaning to the present disclosure.

Software, whether called software, firmware, middleware, microcode, hardware description language, or any other name, instructions, instruction sets, code, code segments, program code, programs, subprograms, software modules. , Application, software application, software package, routine, subroutine, object, executable, thread of execution, procedure, function, etc. should be construed broadly.
Also, software, instructions, information, etc. may be sent and received via a transmission medium. For example, the software uses a wired technology (coaxial cable, optical fiber cable, twisted pair, Digital Subscriber Line (DSL), etc.) and / or wireless technology (infrared, microwave, etc.) to use a website, When sent from a server, or other remote source, at least one of these wired and wireless technologies is included within the definition of transmission medium.

The information, signals, etc. described in this disclosure may be represented using any of a variety of different technologies. For example, data, instructions, commands, information, signals, bits, symbols, chips, etc. that may be referred to throughout the above description include voltage, current, electromagnetic waves, magnetic fields or magnetic particles, optical fields or photons, or any of these. May be represented by a combination of
The terms described in the present disclosure and terms necessary for understanding the present disclosure may be replaced with terms having the same or similar meanings. For example, at least one of the channel and the symbol may be a signal (signaling). The signal may also be a message. Moreover, a component carrier (CC: Component Carrier) may be called a carrier frequency, a cell, a frequency carrier, or the like.

  The terms "system" and "network" used in this disclosure are used interchangeably.

Further, the information, parameters, etc. described in the present disclosure may be represented by using an absolute value, may be represented by using a relative value from a predetermined value, or by using other corresponding information. May be represented. For example, the radio resources may be those indicated by the index.
The names used for the above parameters are not limiting in any way. Further, the formulas and the like that use these parameters may differ from those explicitly disclosed in this disclosure. Since different channels (eg PUCCH, PDCCH, etc.) and information elements can be identified by any suitable name, the different names assigned to these different channels and information elements are in no way limited names. is not.

In the present disclosure, terms such as “mobile station (MS)”, “user terminal”, “user equipment (UE)”, and “terminal” may be used interchangeably. .
Mobile stations are defined by those skilled in the art as subscriber stations, mobile units, subscriber units, wireless units, remote units, mobile devices, wireless devices, wireless communication devices, remote devices, mobile subscriber stations, access terminals, mobile terminals, wireless. It may also be referred to as a terminal, remote terminal, handset, user agent, mobile client, client, or some other suitable term.

  The user terminal may be called a transmitting device, a receiving device, a communication device, or the like. Note that at least one of the user terminals may be a device mounted on a mobile body, the mobile body itself, or the like. The moving body may be a vehicle (eg, car, airplane, etc.), an unmanned moving body (eg, drone, self-driving car, etc.), or a robot (manned type or unmanned type). ) May be sufficient. Note that the user terminal also includes a device that does not necessarily move during communication operation. For example, the user terminal 10 may be an IoT (Internet of Things) device such as a sensor. The user terminal in the present disclosure may be replaced by the base station. In this case, the base station may have the function of the above-described user terminal 10.

  The terms "determining" and "determining" may encompass a wide variety of actions. "Judgment", "decision" means, for example, judgment (judging), calculation (calculating), calculation (computing), processing (processing), derivation (deriving), investigating (investigating), searching (looking up, search, inquiry) (Eg, searching in a table, database, or another data structure), ascertaining what is considered to be “judgment” and “decision”, and the like. In addition, "decision" and "decision" include receiving (eg, receiving information), transmitting (eg, transmitting information), input (input), output (output), access (accessing) (for example, accessing data in a memory) may be regarded as “judging” and “deciding”. In addition, "judgment" and "decision" are considered to be "judgment" and "decision" when things such as resolving, selecting, choosing, choosing, establishing, and comparing are done. May be included. That is, the “judgment” and “decision” may include considering some action as “judgment” and “decision”. In addition, "determination (decision)" may be read as "assuming," "expecting," "considering," and the like.

  The terms "connected," "coupled," or any variation thereof, mean any direct or indirect connection or coupling between two or more elements, It can include the presence of one or more intermediate elements between two elements that are “connected” or “coupled”. The connections or connections between the elements may be physical, logical, or a combination thereof. For example, “connection” may be read as “access”. As used in this disclosure, two elements are in the radio frequency domain, with at least one of one or more wires, cables and printed electrical connections, and as some non-limiting and non-exhaustive examples. , Can be considered to be “connected” or “coupled” to each other, such as with electromagnetic energy having wavelengths in the microwave region and the light (both visible and invisible) region.

  As used in this disclosure, the phrase “based on” does not mean “based only on,” unless expressly specified otherwise. In other words, the phrase "based on" means both "based only on" and "based at least on."

  Any reference to elements using the designations "first," "second," etc. as used in this disclosure does not generally limit the amount or order of those elements. These designations may be used in this disclosure as a convenient way to distinguish between two or more elements. Thus, references to the first and second elements do not mean that only two elements may be employed, or that the first element must precede the second element in any way.

  The “means” in the configuration of each of the above devices may be replaced with “unit”, “circuit”, “device”, and the like.

  Where the terms “include”, “including” and variations thereof are used in this disclosure, these terms are inclusive, as is the term “comprising”. Is intended. Furthermore, the term "or" as used in this disclosure is not intended to be an exclusive or.

  In the present disclosure, where translations add articles, such as a, an, and the in English, the disclosure may include that the noun that follows these articles is in the plural.

  In the present disclosure, the term “A and B are different” may mean “A and B are different from each other”. The term may mean that “A and B are different from C”. The terms "remove", "coupled" and the like may be construed as "different" as well.

DESCRIPTION OF SYMBOLS 1 ... Performance evaluation system, 2 ... Network, 10 ... User terminal, 11 ... Sound collection part, 12 ... Recognition part, 13 ... Specification part, 14 ... Evaluation part, 20 ... Server device, 1001 ... Processor, 1002 ... Memory, 1003 Storage 1004 Communication device 1005 Input device 1006 Output device

Claims (10)

A recognition unit that recognizes the pitch of the performance sound in the performance sound data indicating the collected performance sound,
A specifying unit that specifies a correspondence relationship between the score sound and the performance sound based on the degree of coincidence between the pitch of the score sound indicated by the score data and the pitch of the performance sound recognized in the performance sound data by the recognition unit. With and
The specifying unit is
When the degree of coincidence between the pitch of the first musical note sound included in the musical score data and the pitch of the musical performance sound recognized in the musical performance data exceeds a threshold value, it corresponds to the first musical note sound. Specify the start timing of the first performance sound,
In the score data, the degree of coincidence between the pitch of the second score sound following the first score sound and the pitch of the performance sound following the first performance sound recognized in the performance sound data has a threshold value. An information processing apparatus, characterized in that, when the time exceeds, the end timing of the first performance sound corresponding to the first musical score sound is specified. The information processing apparatus according to claim 1, further comprising an evaluation unit that evaluates the performance by comparing the musical score sound and the performance sound that are in the correspondence relationship specified by the specifying unit. The specifying unit continuously and temporally compares the pitch of one score sound in the score data with the pitch of the performance sound recognized in the performance sound data a predetermined number of times, and compares the respective pitches. The information processing apparatus according to claim 1 or 2, characterized in that, when the degree of coincidence of (1) exceeds a threshold value, it is specified that the musical score sound and the performance sound correspond. The information processing apparatus according to claim 3, wherein the predetermined number of times differs depending on the length of the musical score sound, the tempo in the musical score data, or the tempo of performance. The start timing or the end timing of the performance sound corresponding to the musical score sound by the specifying unit depending on whether the musical sound data of the same pitch is continuous in the musical score data or the musical sound of different pitches is continuous. The information processing apparatus according to any one of claims 1 to 4, characterized in that a method of specifying is different. When the first musical note sound and the second musical note sound of the same pitch are consecutive in the musical score data, the specifying unit matches the pitches of the first musical note sound and the second musical note sound. In the section of the performance sound whose degree exceeds the threshold value, the first performance sound corresponding to the first musical score sound is specified based on the increase / decrease in the sound pressure of the performance sound, and the first musical sound corresponding to the second musical sound is identified. The information processing apparatus according to claim 5, wherein the performance sound of 2 is specified. When the first musical note sound and the second musical note sound with different pitches are continuous in the musical score data, the specifying unit determines the pitch of the first musical note sound included in the musical score data and the performance sound data. In the case where the degree of coincidence with the pitch of the performance sound recognized in 1 exceeds a threshold value, the start timing of the first performance sound corresponding to the first musical score sound is specified,
In the score data, the degree of coincidence between the pitch of the second score sound following the first score sound and the pitch of the performance sound following the first performance sound recognized in the performance sound data has a threshold value. The information processing apparatus according to claim 5 or 6, wherein when it exceeds, the ending timing of the first performance sound corresponding to the first musical score sound is specified. The specifying unit includes a pitch of a second musical note following the first musical note in the musical score data and a pitch of a musical note following the first musical note recognized in the musical note data. The first performance sound corresponding to the first musical score sound is determined to have ended when a period in which the degree of coincidence does not exceed a threshold value has passed a predetermined time period. The information processing apparatus according to any one of items. The specifying unit determines that a predetermined number of times continues when a period in which the degree of coincidence between the pitch of the second musical score sound and the pitch of the performance sound following the first performance sound does not exceed a threshold has passed a predetermined number of times. By comparing a combination of pitches of a plurality of musical score sounds that are temporally continuous in the musical score data with a combination of pitches of a plurality of musical performance sounds that are continuously recognized in time in the musical performance sound data, 9. The information processing apparatus according to claim 8, wherein the correspondence relationship between the musical score sound and the performance sound is specified. Computer,
A recognition unit that recognizes the pitch of the performance sound in the performance sound data indicating the collected performance sound,
A specifying unit that specifies a correspondence relationship between the score sound and the performance sound based on the degree of coincidence between the pitch of the score sound indicated by the score data and the pitch of the performance sound recognized in the performance sound data by the recognition unit. If the degree of coincidence between the pitch of the first musical note sound included in the musical score data and the pitch of the musical performance sound recognized in the musical performance data exceeds a threshold, the first musical score A start timing of a first performance sound corresponding to a note is specified, and a pitch of a second music sound following the first music sound in the music score data and the first pitch recognized in the performance sound data. To function as a specifying unit that specifies the end timing of the first performance sound corresponding to the first musical score sound when the degree of coincidence with the pitch of the performance sound following the performance sound exceeds a threshold value. Program of.

Images