JP2023144076A - Program, information processing method and information processing device - Google Patents

Abstract

To provide a program capable of generating reference data which is further appropriate as reference when scoring singing from sound source data.SOLUTION: A program causes a computer of an information processing device to perform a sound source data acquisition step for acquiring sound source data including a melody part of music; a singing voice separation extraction step for separating and extracting first singing voice data of a first singer and second singing voice data of a second singer different from the first singer from the sound source data, and a reference data generation step for generating first reference data of a portion which the first singer sings in the melody part on the basis of the first singing voice data and generating second reference data of a portion which the second singer sings in the melody part on the basis of the second singing voice data.SELECTED DRAWING: Figure 8

Description

The present disclosure relates to a program, an information processing method, and an information processing apparatus that generate reference data from sound source data of a song.

By canceling the band of the center-localized vocal signal in the singing data of the song data and subtracting the remaining data from the original data (singing data) and extracting it as main vocal data, the singing data of the main melody is extracted from the song data. A karaoke apparatus is known which has a program main vocal data extraction means for extracting main vocal data, and a reference data generation means for generating reference data that is a standard for scoring singing from the extracted main vocal data. (For example, see Patent Document 1).

Japanese Patent Application Publication No. 2015-225302

However, with the technology shown in Patent Document 1, the extracted main vocal data may not necessarily match the part sung as karaoke. For example, production parts where human voices are used in areas other than singing parts, sounds of musical instruments that were incorrectly extracted due to problems with extraction accuracy, etc., may be mixed in with the extracted main vocal data. there is a possibility. If reference data is generated using main vocal data including non-singing parts that do not match the karaoke singing parts, it may not be appropriate as a standard for scoring singing.

The present disclosure has been made to solve such problems. The purpose is to provide a program, an information processing method, and an information processing device that can generate reference data that is more appropriate as a standard for scoring singing from sound source data that includes a melody part of a song.

A program according to the present disclosure is a program to be executed by a computer of an information processing device, and includes a sound source data acquisition step of acquiring sound source data including a melody part of a song, and a step of acquiring sound source data including a melody part of a song, and a step of acquiring sound source data of a first singer from the sound source data. a singing voice separation and extraction step of separating and extracting first singing voice data and second singing voice data of a second singer different from the first singer; generating first reference data for a part of the melody part sung by the first singer; and based on the second singing voice data, second reference data for a part of the melody part sung by the second singer. A reference data generation step of generating the reference data is executed by the computer of the information processing device.

An information processing method according to the present disclosure is an information processing method executed by a computer of an information processing device, and includes a sound source data acquisition step of acquiring sound source data including a melody part of a song, and a step of acquiring sound source data including a melody part of a song. a singing voice separation and extraction step of separating and extracting first singing voice data of a person and second singing voice data of a second singer different from the first singer; Generate first reference data of a part of the melody part sung by the first singer, and generate second reference data of the part of the melody part sung by the second singer based on the second singing voice data. and a reference data generation step of generating reference data.

The information processing device according to the present disclosure includes a sound source data acquisition unit that acquires sound source data including a melody part of a song, and from the sound source data, first singing voice data of a first singer is different from the first singer. a singing voice separation and extraction unit that separates and extracts second singing voice data of a second singer; and a singing voice separating and extracting unit that separates and extracts second singing voice data of a second singer; 1 reference data, and generates second reference data of a portion of the melody part sung by the second singer based on the second singing voice data.

According to the program, information processing method, and information processing device according to the present disclosure, it is possible to generate reference data that is more appropriate as a standard for scoring singing from sound source data including the melody part of a song. play.

1 is a diagram showing the overall configuration of a communication system according to Embodiment 1. FIG. 1 is a block diagram showing the configuration of a terminal included in the communication system according to Embodiment 1. FIG. 1 is a block diagram showing the configuration of a server included in the communication system according to Embodiment 1. FIG. FIG. 3 is a diagram showing an example of synchronized lyrics data stored in a server included in the communication system according to the first embodiment. 5 is a flowchart illustrating an example of processing in the communication system according to the first embodiment. FIG. 3 is a block diagram showing the configuration of main parts in a modified example of the server included in the communication system according to the first embodiment. 7 is a flowchart illustrating an example of processing in a modification of the communication system according to the first embodiment. FIG. 2 is a block diagram showing the configuration of a server included in the communication system according to Embodiment 2. FIG. 7 is a flowchart illustrating an example of processing in the communication system according to Embodiment 2. FIG.

Embodiments for implementing a program, an information processing method, and an information processing apparatus according to the present disclosure will be described with reference to the accompanying drawings. In each figure, the same or corresponding parts are given the same reference numerals, and overlapping explanations will be simplified or omitted as appropriate. In the following description, for convenience, the positional relationship of each structure may be expressed based on the illustrated state. Note that the present disclosure is not limited to the following embodiments, and any combination of embodiments, modification of any component of each embodiment, or modification of each embodiment may be made without departing from the spirit of the present disclosure. Any component of the embodiment can be omitted.

Embodiment 1.
Embodiment 1 of the present disclosure will be described with reference to FIGS. 1 to 7. FIG. 1 is a diagram showing the overall configuration of a communication system. FIG. 2 is a block diagram showing the configuration of a terminal included in the communication system. FIG. 3 is a block diagram showing the configuration of a server included in the communication system. FIG. 4 is a diagram showing an example of synchronized lyrics data stored in a server included in the communication system. FIG. 5 is a flowchart showing an example of processing in the communication system. FIG. 6 is a block diagram showing the configuration of main parts in a modified example of the server included in the communication system. FIG. 7 is a flowchart showing an example of processing in a modified example of the communication system.

As shown in FIG. 1, a communication system 400 according to this embodiment includes a server 100 and a terminal 200. In the communication system 400, a server 100 and a terminal 200 are communicably connected via a network 300. In the configuration example described here, the server 100 provides a music source data distribution service, a karaoke scoring service, etc. to the terminal 200 owned by the user via the network 300. Note that the number of terminals 200 connected to network 300 is not limited to two, and may be one or three or more.

The network 300 plays a role of connecting one or more terminals 200 and one or more servers 100. That is, the network 300 refers to a communication network that provides a connection path so that the terminal 200 can transmit and receive data after connecting to the server 100. One or more portions of network 300 may or may not be a wired network or a wireless network.

The network 300 is, for example, an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), or a wireless LAN. : WLAN), Wide Area Network (WAN), Wireless WAN (WWAN), Metropolitan Area Network (MAN), part of the Internet, Public Switched Telephone Network Ephone Network: PSTN), mobile phone network, ISDN (Integrated Service Digital Networks), wireless LAN, LTE (Long Term Evolution), CDMA (Code Division Multiple Access), Blue Bluetooth (registered trademark) or satellite communication, etc. , or a combination of two or more of these. Network 300 may include one or more networks 300.

The terminal 200 may be any information processing terminal that can implement the functions of the embodiments of the present disclosure. Terminal 200 may be, for example, a smartphone, a mobile phone (feature phone), a computer (e.g., desktop PC, laptop PC, tablet PC, etc.), a media computer platform (e.g., cable, satellite set-top box, digital video recorder), handheld. These include computer devices (eg, PDAs (Personal Digital Assistants), e-mail clients, etc.), wearable terminals (glass-type devices, watch-type devices, etc.), or other types of computers, or communication platforms. Further, the terminal 200 may be expressed as an information processing terminal.

The server 100 has a function of providing predetermined services to the terminal 200. The server 100 may be any information processing device that can implement the functions of the embodiments of the present disclosure. Server 100 may be, for example, a server device, a computer (e.g., desktop PC, laptop PC, tablet PC, etc.), a media computer platform (e.g., cable, satellite set-top box, digital video recorder), a handheld computer device (e.g., PDA , email clients, etc.) or other types of computers or communication platforms. Further, the server 100 may be expressed as an information processing device. If there is no need to distinguish between the server 100 and the terminal 200, the server 100 and the terminal 200 may or may not each be expressed as an information processing device.

The configuration of each terminal 200 is basically the same. Next, the configuration of the terminal 200 will be described with reference to FIG. 2. The terminal 200 includes a terminal control section 230, a terminal storage section 220, a terminal communication section 210, an input/output section 240, a display section 250, a microphone 260, a speaker 270, and a camera 280. The hardware components of the terminal 200 are interconnected via, for example, a bus. Note that the hardware configuration of the terminal 200 does not necessarily include all the components described here. For example, the terminal 200 may or may not be configured such that an individual component or multiple components such as the camera 280 can be removed.

The terminal communication unit 210 transmits and receives various data via the network 300. The communication may be performed by wire or wirelessly, and any communication protocol may be used as long as mutual communication can be performed. The terminal communication unit 210 has a function of communicating with the server 100 via the network 300. The terminal communication section 210 includes a terminal transmitting section 211 and a terminal receiving section 212. The terminal transmitter 211 transmits various data to the server 100 according to instructions from the terminal controller 230. Terminal receiving section 212 receives various data transmitted from server 100 and transmits it to terminal controlling section 230. Further, the terminal communication unit 210 may be expressed as a terminal communication I/F (interface). In addition, when the terminal communication unit 210 is composed of a physically structured circuit, it may be expressed as a terminal communication circuit.

The input/output section 240 includes an input section and an output section. The input unit is a device for inputting various operations to the terminal 200. The output unit is a device that outputs the processing results processed by the terminal 200. The input/output section 240 may have an input section and an output section integrated, or may be separated into an input section and an output section, or may not.

The input unit is realized by any one of all types of devices, or a combination thereof, that can receive input from the user and transmit information related to the input to the terminal control unit 230. The input unit includes, for example, a touch panel, a touch display, hardware keys such as a keyboard, a pointing device such as a mouse, a camera (operation input via moving images), and a microphone (operation input via voice).

The output unit is realized by any of all types of devices capable of outputting the processing results processed by the terminal control unit 230, or a combination thereof. The output unit includes, for example, a touch panel, a touch display, a speaker (audio output), a lens (for example, 3D (Three Dimensions) output or hologram output), a printer, and the like.

The display unit 250 is realized by any one of all types of devices capable of displaying data according to the display data written in the frame buffer, or a combination thereof. The display unit 250 is, for example, a touch panel, a touch display, a monitor (for example, a liquid crystal display, an OELD (Organic Electroluminescence Display), etc.), a head mounted display (HDM), a projection mapping, a hologram, an air display, etc. (which may or may not be the case) includes devices that can display images, text information, etc. Note that these display units 250 may or may not be capable of displaying display data in 3D.

Note that when the input/output unit 240 has a touch panel, the input/output unit 240 and the display unit 250 may have substantially the same size and shape and may be disposed facing each other.

The terminal control unit 230 has a physically structured circuit for executing functions realized by codes or instructions included in a program, and is realized by, for example, a data processing device built into hardware. Ru. Therefore, the terminal control unit 230 may or may not be expressed as a control circuit.

The terminal control unit 230 includes, for example, a central processing unit (CPU), a microprocessor, a processor core, a multiprocessor, and an application-specific IC (ASIC). ific Integrated Circuit), FPGA (Field Programmable Gate Array), etc.

The terminal storage unit 220 has a function of storing various programs and various data necessary for the operation of the terminal 200. The terminal storage unit 220 includes various storage media such as, for example, a hard disk drive (HDD), a solid state drive (SSD), a flash memory, a random access memory (RAM), and a read only memory (ROM). Further, the terminal storage unit 220 may or may not be expressed as a memory.

The terminal 200 stores a program in the terminal storage unit 220, and by executing this program, the terminal control unit 230 executes processing as each unit included in the terminal control unit 230. That is, the programs stored in the terminal storage section 220 cause the terminal 200 to implement each function executed by the terminal control section 230. In other words, the processor of the terminal 200 executes a program stored in the memory, and the hardware and software of the terminal 200 cooperate to realize the functions of each part of the terminal 200. Note that this program may or may not be expressed as a program module.

Microphone 260 is used for inputting audio data. Speaker 270 is used to output audio data. Camera 280 is used to acquire moving image data and/or still image data.

Next, the configuration of the server 100 will be described with reference to FIG. 3. The server 100 includes a server control section 130, a server storage section 120, and a server communication section 110. The hardware components of the server 100 are interconnected via, for example, a bus.

The server control unit 130 has a physically structured circuit to execute functions realized by codes or instructions included in a program, and is realized by, for example, a data processing device built in hardware. Ru. The server control unit 130 is typically a central processing unit (CPU), and may or may not be a microprocessor, processor core, multiprocessor, ASIC, FPGA, or the like. In the present disclosure, the server control unit 130 is not limited to these.

The server storage unit 120 has a function of storing various programs and various data necessary for the operation of the server 100. The server storage unit 120 is realized by various storage media such as HDD, SSD, and flash memory. However, in the present disclosure, the server storage unit 120 is not limited to these. Further, the server storage unit 120 may or may not be expressed as a memory.

The server communication unit 110 transmits and receives various data via the network 300. The communication may be performed by wire or wirelessly, and any communication protocol may be used as long as mutual communication can be performed. The server communication unit 110 has a function of communicating with the terminal 200 via the network 300. The server communication section 110 includes a server transmission section 111 and a server reception section 112. The server transmitter 111 transmits various data to the terminal 200 according to instructions from the server controller 130. Additionally, the server communication unit 110 receives various data transmitted from the terminal 200 and transmits it to the server control unit 130. Further, the server communication unit 110 may be expressed as a server communication I/F (interface). Further, when the server communication unit 110 is composed of a physically structured circuit, it may be expressed as a server communication circuit.

Note that the server 100 may include an input/output unit and a display as a hardware configuration. The input/output unit is realized by a device that inputs various operations to the server 100. The input/output unit is realized by any one of all types of devices capable of receiving input from a user and transmitting information related to the input to the server control unit 130, or a combination thereof. The display is typically realized by a monitor (eg, a liquid crystal display, an OELD, etc.). In this case, for example, the hardware of the server 100 may or may not have a display that can be removed.

The server 100 stores a program in the server storage unit 120, and by executing this program, the server control unit 130 executes processing as each unit included in the server control unit 130. That is, the programs stored in the server storage unit 120 cause the server 100 to implement each function executed by the server control unit 130. In other words, in the server 100, a processor executes a program stored in a memory, and the hardware and software of the server 100 cooperate to realize the functions of each part of the server 100. Note that this program may or may not be expressed as a program module.

Note that the terminal control unit 230 of the terminal 200 and/or the server control unit 130 of the server 100 include not only a CPU having a control circuit but also an integrated circuit (IC (Integrated Circuit) chip, LSI (Large Scale Integration)), etc. Each process may or may not be realized by a logic circuit (hardware) or a dedicated circuit formed in the system. Further, these circuits may be realized by one or more integrated circuits, and the plurality of processes shown in the embodiments of the present disclosure may be realized by one integrated circuit, or even if not. good. Further, LSIs are sometimes called VLSIs, super LSIs, ultra LSIs, etc. depending on the degree of integration.

Further, the program (for example, a software program, a computer program, or a program module) according to an embodiment of the present disclosure may or may not be provided in a state stored in a computer-readable storage medium. . The storage medium is a "non-temporary tangible medium" that can store a program. Furthermore, the program may or may not be for realizing part of the functions of the embodiments of the present disclosure. Furthermore, it may or may not be a so-called difference file (difference program) that can realize the functions of the embodiments of the present disclosure in combination with a program already recorded on a storage medium.

The storage medium may include one or more semiconductor-based or other integrated circuits (ICs), such as field programmable gate arrays (FPGAs) or application specific integrated circuits (ASICs), hard disk drives, etc. (HDD), hybrid hard drive (HHD), optical disk, optical disk drive (ODD), magneto-optical disk, magneto-optical drive, floppy diskette, floppy disk drive (FDD), magnetic tape, solid-state drive (SSD) , a RAM drive, a secure digital card, or a drive, any other suitable storage medium, or a suitable combination of two or more thereof. Storage media may be volatile, non-volatile, or a combination of volatile and non-volatile, where appropriate. Note that the storage medium is not limited to these examples, and may be any device or medium as long as it can store the program. Further, the storage medium may or may not be expressed as a memory.

Further, the program of the present disclosure may or may not be provided to the server 100 and/or the terminal 200 via any transmission medium (communication network, broadcast waves, etc.) that can transmit the program. . When the program is provided via a transmission medium, the server 100 and/or the terminal 200 realizes the functions of the plurality of functional units shown in each embodiment by, for example, executing the program downloaded via the Internet or the like. It is possible to do so.

Embodiments according to the present disclosure may also be implemented in the form of a data signal embedded in a carrier wave, where the program is embodied by electronic transmission. Furthermore, at least part of the processing in the server 100 and/or the terminal 200 may or may not be realized by cloud computing configured by one or more computers. Further, at least a part of the processing in the terminal 200 may be performed by the server 100, or may not be configured. In this case, at least part of the processing of each functional unit of the terminal control unit 230 of the terminal 200 may or may not be performed by the server 100. Further, at least a part of the processing in the server 100 may be performed by the terminal 200, or may not be performed. In this case, at least some of the processing of each functional unit of the server control unit 130 of the server 100 may or may not be performed by the terminal 200.

Note that the program of the present disclosure may be implemented using, for example, a script language such as ActionScript or JavaScript (registered trademark), an object-oriented programming language such as Objective-C or Java (registered trademark), or a markup language such as HTML5. obtain.

In the communication system 400 according to this embodiment, as shown in FIG. 3, the server storage section 120 of the server 100 includes a sound source data storage section 121 and a lyrics data storage section 122. The sound source data storage unit 121 stores sound source data of songs. The sound source data is data consisting of waveform data in which an accompaniment part and a melody part of a song are mixed. The melody part is the main melody part, and is usually a vocal part sung by a person. Various audio file formats such as WAVE, AIFF, MP3, AAC, and FLAC can be used for the sound source data.

The lyrics data storage unit 122 stores lyrics data of songs. The lyrics data is data of lyrics of a melody part of a song. In the configuration example described here, the lyrics data is synchronous lyrics data. The synchronized lyrics data includes multiple lyrics phrases. In the synchronized lyrics data, each lyric phrase is associated with a display start time in order to be displayed in synchronization with the reproduction of the sound source data of the song.

For example, an LRC file can be used as such synchronized lyrics data. An example of an LRC file that is synchronized lyrics data is shown in FIG. The LRC file is a text file that follows a specific format. In the LRC file format, each line is a lyric phrase that is displayed at once. The number surrounded by "[" and "]" at the beginning of each line is the display start time of the lyrics phrase in that line. The display start time is mm:ss. It is written in xx format. mm indicates minutes, ss indicates seconds, and xx indicates 1/100 seconds.

In the communication system 400 according to this embodiment, the server 100 includes, as functions realized by the server control unit 130, a sound source data acquisition unit 131, a lyrics data acquisition unit 132, a singing part identification unit 133, and a reference data generation unit 134. It is equipped with The sound source data acquisition unit 131 performs a process of acquiring sound source data stored in the sound source data storage unit 121. The lyrics data acquisition unit 132 performs a process of acquiring lyrics data stored in the lyrics data storage unit 122.

The singing part identifying unit 133 identifies the singing part in the sound source data acquired by the sound source data acquiring unit 131 based on the lyrics data acquired by the lyrics data acquiring unit 132. In the configuration example described here, the lyrics data acquired by the lyrics data acquisition unit 132 is synchronous lyrics data. Then, the singing part identifying unit 133 identifies the singing part in the sound source data based on the display start time of each lyric phrase included in the synchronized lyrics data.

The identification of a singing part by the singing part specifying unit 133 will be explained using a specific example. For example, in the case of the synchronized lyrics data shown in FIG. 4, there is no lyrics until 11.14 seconds have elapsed from the start of the song as a prelude. The period from 11.14 seconds to 1 minute 10.68 seconds is the first singing part. Also, from 1 minute 10.68 seconds to 1 minute 34.16 seconds is an interlude with no lyrics. The second part is sung from 1 minute 34.16 seconds to 2 minutes 16.71 seconds. Then, from 2 minutes 16.71 seconds onwards, there is a postlude. The prelude, interlude, and postlude are non-sung parts.

The reference data generation unit 134 generates reference data of the melody part of the sound source data. The reference data is data regarding the sound elements of the melody part. Sound elements that are the subject of reference data include pitch, pronunciation timing, note length, and the like.

The reference data generation unit 134 first extracts human singing voice data from the sound source data. To extract singing audio data from sound source data, for example, if the sound source data is a stereo sound source, the human singing audio (vocal part) is often localized in the center. This can be done using known methods such as extracting frequency components or removing waveform components of timbres such as musical instruments that are not human voices. Then, the reference data generation unit 134 detects the pitch of the extracted singing voice data using a known pitch detection algorithm, for example, and outputs the detected pitch as reference data.

In particular, in the server 100 according to this embodiment, the reference data generation unit 134 uses the sound source data of the singing part specified by the singing part identification unit 133 to generate reference data of the melody part. In other words, the reference data generation unit 134 generates reference data for the melody part after excluding the non-sung parts from the sound source data of the song.

In the example of the synchronized lyrics data shown in FIG. 4 described above, the reference data generation unit 134 generates the first singing part from 11.14 seconds to 1 minute 10.68 seconds and the first singing part from 1 minute 34 seconds out of the sound source data. Reference data for the melody part is generated using the second singing part from 16 seconds to 2 minutes and 16.71 seconds. Therefore, the reference data generation unit 134 uses the portions from 11.14 seconds to 1 minute 10.68 seconds and from 1 minute 34.16 seconds to 2 minutes 16.71 seconds of the sound source data of the song, Generate reference data for the melody part. This excludes the non-sung parts from 11.14 seconds after the song starts, from 1 minute 10.68 seconds to 1 minute 34.16 seconds, and after 2 minutes 16.71 seconds, and then calculates the melody part. In other words, it generates reference data.

As shown in FIG. 3, the server storage section 120 further includes a reference data storage section 123. The reference data storage unit 123 stores reference data for each song generated by the reference data generation unit 134.

In the configuration example shown in FIG. 3, the server 100 further includes an accompaniment part acquisition section 135 as a function realized by the server control section 130. The accompaniment part acquisition unit 135 performs so-called vocal cancellation processing on the sound source data acquired by the sound source data acquisition unit 131, and extracts and acquires the accompaniment part of the music piece. Known methods can be used for vocal cancellation processing. The accompaniment part data acquired by the accompaniment part acquisition unit 135 can be used as so-called karaoke sound source data.

The server transmission section 111 of the server 100 transmits the sound source data of the music stored in the sound source data storage section 121 to the terminal 200. As a result, a distribution service for music source data is realized. Further, the server transmitting unit 111 may transmit the lyrics data of the song stored in the lyrics data storage unit 122 to the terminal 200. Further, the server transmitter 111 may transmit the accompaniment part data acquired by the accompaniment part acquirer 135 to the terminal 200.

The terminal receiving unit 212 of the terminal 200 receives the sound source data, lyrics data, and accompaniment part data of the song transmitted from the server 100. In the configuration example shown in FIG. 2, the terminal 200 includes a reproduction processing section 231, a display processing section 232, and a singer voice acquisition section 233 as functions realized by the terminal control section 230. The reproduction processing unit 231 reproduces the sound source data of the music piece received by the terminal reception unit 212, and outputs it from the speaker 270 or the like.

Furthermore, the reproduction processing unit 231 reproduces the data of the accompaniment part of the music received by the terminal reception unit 212, and causes the data to be output from the speaker 270 or the like. At this time, the display processing section 232 causes the display section 250 to display the lyrics data of the song received by the terminal receiving section 212. As mentioned above, in the configuration example described here, the lyrics data is synchronous lyrics data. Therefore, the display processing unit 232 can display the lyrics data in synchronization with the playback of the accompaniment part data of the song by the playback processing unit 231. The singer voice acquisition unit 233 acquires the voice input to the microphone 260 as the singer's singing voice data while the reproduction processing unit 231 is playing back data of the accompaniment part of the song.

The terminal transmitter 211 transmits the singing voice data acquired by the singer voice acquirer 233 to the server 100. The server reception unit 112 of the server 100 receives the singer's singing voice data transmitted from the terminal 200.

In the configuration example shown in FIG. 3, the server 100 further includes a singing voice acquisition section 136 and an evaluation section 137 as functions realized by the server control section 130. The singing voice acquisition unit acquires the singing voice data received by the server receiving unit 112. The evaluation unit 137 of the server 100 compares and evaluates the singing voice data of the singer acquired by the singing voice acquisition unit 136 with the reference data stored in the reference data storage unit 123. For example, the evaluation unit 137 detects the pitch of the singer's singing voice data using the known pitch detection algorithm described above. The evaluation unit 137 then compares the pitch with the standard pitch of the reference data, and evaluates the higher the degree of agreement between these pitches. The evaluation result by the evaluation section 137 is transmitted to the terminal 200 via the server transmission section 111 and the terminal reception section 212, and displayed on the display section 250, for example.

Further, reference data may be transmitted from the server 100 to the terminal 200. In this case, for example, a so-called guide melody can be displayed on the display unit 250 using the reference data while the data of the accompaniment part, which is a karaoke sound source, is being played back on the terminal 200. Furthermore, the evaluation unit 137 may be provided in the terminal 200 instead of the server 100.

Next, an example of the operation of the communication system 400 according to this embodiment will be described with reference to the flowchart in FIG. 5. First, in step S10, the sound source data acquisition unit 131 of the server 100 acquires sound source data stored in the sound source data storage unit 121. In subsequent step S11, the lyrics data acquisition section 132 acquires the lyrics data stored in the lyrics data storage section 122. After step S11, the server control unit 130 next performs the process of step S12.

In step S12, the singing part identifying unit 133 identifies the singing part in the sound source data acquired in step S10, based on the lyrics data acquired in step S11. In subsequent step S13, the reference data generation unit 134 generates reference data of the melody part using the sound source data of the singing part identified in step S12. After step S13, the server 100 next performs the process of step S14.

In step S14, the reference data storage unit 123 of the server 100 stores the reference data generated in step S13. Further, the accompaniment part acquisition unit 135 extracts and acquires an accompaniment part from the sound source data acquired in step S10. Then, the server transmitter 111 transmits the sound source data, lyrics data, and accompaniment part data to the terminal 200. After step S14, the terminal 200 next performs the process of step S15.

In step S15, the terminal receiving unit 212 of the terminal 200 receives the sound source data, lyrics data, and accompaniment part data of the song transmitted from the server 100 in step S14. Then, the reproduction processing unit 231 of the terminal 200 reproduces the data of the accompaniment part of the music received by the terminal reception unit 212, and causes the data to be output from the speaker 270 or the like. Furthermore, the display processing unit 232 of the terminal 200 causes the display unit 250 to display the lyrics data of the song received by the terminal receiving unit 212. After step S15, the terminal 200 next performs the process of step S16.

In step S16, the singing voice acquisition unit 136 of the terminal 200 acquires the singing voice data of the singer input into the microphone 260. In subsequent step S17, the terminal transmitter 211 transmits the singing voice data acquired in step S16 to the server 100. The server reception unit 112 of the server 100 receives the singer's singing voice data transmitted from the terminal 200. After step S17, the server 100 next performs the process of step S18.

In step S18, the singing voice acquisition unit 136 of the server 100 acquires the singing voice data received by the server receiving unit 112 in step S16. In subsequent step S19, the evaluation unit 137 of the server 100 compares and evaluates the singer's singing voice data acquired in step S18 with the reference data stored in the reference data storage unit 123 in step S14. In the subsequent step S20, the evaluation result by the evaluation unit 137 is transmitted to the terminal 200 via the server transmission unit 111 and the terminal reception unit 212, and displayed on the display unit 250, for example.

In the above flow, step S10 corresponds to a sound source data acquisition step, step S11 corresponds to a lyrics data acquisition step, step S12 corresponds to a singing part identification step, and step S13 corresponds to a reference data generation step. Step S18 corresponds to a singing voice acquisition step, and step S19 corresponds to an evaluation step.

According to the server 100 of the communication system 400 configured as described above, the singing part in the sound source data is specified using the lyrics data, and the reference data of the melody part is generated using the sound source data of the specified singing part. generate. This prevents production parts where human voices are used other than singing parts, and sounds that are incorrectly extracted due to extraction accuracy etc. to be reflected in the reference data. It is possible to generate reference data that is more appropriate as a standard for scoring singing.

Next, a modification of the server 100 included in the communication system 400 according to this embodiment will be described. In this modification, the singing part identifying section 133 of the server 100 includes a singing voice extracting section 141, a text converting section 142, and a collating section 143, as shown in FIG. The singing voice extraction unit 141 extracts human singing voice data from the sound source data acquired by the sound source data acquisition unit 131. Extraction of human singing voice data from sound source data can be performed using the known method described above.

The text converting unit 142 converts the singing voice data extracted by the singing voice extracting unit 141 into text using known voice recognition processing to generate singing text data. The collation unit 143 collates the singing text data generated by the text conversion unit 142 and the lyrics data acquired by the lyrics data acquisition unit 132 to identify the singing portion in the sound source data. That is, the matching unit 143 identifies a portion where the singing text data and lyrics data match as a singing portion. In other words, the matching unit 143 identifies a portion where the singing text data and lyrics data do not match as a non-singing portion.

Next, an example of the operation of the server 100 according to this modification will be described with reference to the flowchart in FIG. 7. The flowchart in FIG. 5 shows the sub-process of step S12 (singing part identification step) in the flowchart in FIG. First, in step S30, the singing voice extraction unit 141 extracts human singing voice data from the sound source data acquired in step S10 of FIG. 5, and obtains the extracted singing voice data. In subsequent step S31, the text converting unit 142 converts the singing voice data acquired in step S30 into text by voice recognition processing to generate singing text data, and acquires the generated singing text data. In further subsequent step S32, the collation unit 143 collates the singing text data acquired in step S31 with the lyrics data acquired in step S11 of FIG. 5 to identify the singing part in the sound source data.

In the above flow, step S30 corresponds to a singing voice extraction step, step S31 corresponds to a text conversion step, and step S32 corresponds to a collation step.

In this modification, the lyrics data does not need to include the display start time for each lyrics phrase. Therefore, according to such a modification, the singing part/non-singing part of the sound source data is identified using lyrics data that is not synchronous lyrics data, and the sound of the non-singing part is suppressed from being reflected in the reference data. It is possible to generate reference data that is more appropriate as a standard for scoring singing.

Embodiment 2.
Embodiment 2 of the present disclosure will be described with reference to FIGS. 8 and 9. FIG. 8 is a block diagram showing the configuration of a server included in the communication system. FIG. 9 is a flowchart showing an example of processing in the communication system.

The program, information processing method, and information processing apparatus according to the second embodiment will be described below, focusing on the differences from the first embodiment. The configuration whose description is omitted is basically the same as that of the first embodiment. In the following description, structures similar to or corresponding to those of the first embodiment will be described with the same reference numerals used in the description of the first embodiment.

In the communication system 400 according to this embodiment, as shown in FIG. 8, the server 100 further includes a singing voice separation and extraction section 150 as a function realized by the server control section 130. The singing voice separation and extraction section 150 separates and extracts each of the singing voice data of a plurality of people from the sound source data acquired by the sound source data acquisition section 131. For example, the singing voice separation and extraction unit 150 first extracts human singing voice data from the sound source data. Extraction of human singing voice data from sound source data can be performed using the known method described above. Then, if the extracted singing voice data includes singing voice data of a plurality of people, the singing voice separation and extraction unit 150 separates the singing voice data for each person. The separation at this time can be performed by a known method such as waveform analysis of singing voice data. Note that the singing voice separation and extraction unit 150 may separate each of the singing voice data of a plurality of people directly from the sound source data.

In this embodiment, the singing part specifying unit 133 determines the singing part in the sound source data based on the singing voice data of a specific person among the singing voice data of a plurality of people extracted by the singing voice separating and extracting unit 150. Identify. The specific person's singing voice data here is, for example, the singing voice data of a main vocalist. The singing part specifying unit 133 determines the singing audio data of a specific person using, for example, the localization, volume, and proportion of the singing audio data of a plurality of people in the playback time of the song. In addition, if the gender of the main vocalist is known from the artist name of the song, the singing audio data of a specific person is determined based on whether each of the singing audio data of multiple people has a male or female voice. Good too.

The singing part identifying unit 133 identifies the singing part of the singing voice data of the specific person determined in this way, and sets this as the singing part of the sound source data of the song. Then, the reference data generation unit 134 generates reference data of the melody part using the sound source data of the singing part specified by the singing part identification unit 133.

Note that the singing voice data of a specific person is not limited to one person's singing voice data, and may be singing voice data of two or more people. In this case, the singing part identification unit 133 identifies each of the singing parts of the plurality of people in the sound source data based on each of the singing voice data of the specific plurality of people. Then, the reference data generation unit 134 generates reference data of the melody part using the respective sound source data of the singing parts of the plurality of people specified by the singing part identification unit 133. By doing so, in the case of a song in which a plurality of singers alternately perform the melody part, reference data can be generated separately for each of the plurality of singers who perform the melody part. Therefore, for example, it is possible to reproduce a karaoke accompaniment in which only an arbitrary singer among the singers in charge of the melody part is canceled, and score only the part sung by the singer using the reference data.

Next, an example of the operation of the communication system 400 according to this embodiment will be described with reference to the flowchart of FIG. 9. Steps S10 and S11 in the flowchart of the same figure are the same as steps S10 and S11 in the flowchart of FIG. 5, so the explanation thereof will be omitted here. After step S11, the server control unit 130 next performs the process of step S40.

In step S40, the singing voice separation and extraction unit 150 separates and extracts each of the plurality of people's singing voice data from the sound source data acquired in step S10. This step S40 corresponds to a singing voice separation and extraction step. After step S40, the server control unit 130 next performs the process of step S12. In step S12, the singing part identifying unit 133 identifies the singing part in the sound source data based on the singing voice data of a specific person among the singing voice data of a plurality of people extracted in step S40. Steps S13 to S20 after step S12 are the same as steps S13 to S20 in the flowchart of FIG. 5, so the description thereof will be omitted here.

Note that, in this embodiment, the server 100 does not need to include the singing part identifying section 133. In this case, the reference data generation unit 134 generates the reference data of the melody part based on the singing voice data of a specific person among the singing voice data of a plurality of people extracted by the singing voice separation and extraction unit 150. Further, in this case, the process of step S12 in the flow diagram of FIG. 9 is not performed, and after step S40, the server control unit 130 next performs the process of step S13. Then, in step S13, the reference data generation unit 134 generates reference data of the melody part based on the singing voice data of a specific person among the singing voice data of the plurality of people extracted in step S40. Even with such a configuration, reference data can be generated based on the singing voice data of a specific person among the singing voice data of multiple people, so that it is possible to generate reference data based on the singing voice data of a specific person among the singing voice data of multiple people. can be suppressed from being reflected in the reference data, and it is possible to generate reference data that is more appropriate as a standard when scoring singing.

In addition, in the configuration example of each embodiment described above, the sound source data acquisition unit 131, the lyrics data acquisition unit 132, and the singing part identification unit 133 (including the singing voice extraction unit 141, the text conversion unit 142, and the collation unit 143) , a reference data generation section 134, an accompaniment part acquisition section 135, a singing voice acquisition section 136, an evaluation section 137, and a singing voice separation/extraction section 150 are provided in a single server 100. However, each of these units may be provided not in the single server 100 but in a distributed manner in a plurality of server devices. Further, the functions of each part of the server 100 may be realized by cooperation of a plurality of devices.

Although the invention according to the present disclosure has been explained based on the drawings and examples, it should be noted that those skilled in the art can easily make various changes and modifications based on the present disclosure. Therefore, it should be noted that these variations and modifications are included within the scope of the invention according to the present disclosure. For example, the functions included in each part, each means, each step, etc. can be rearranged so as not to be logically contradictory, and it is possible to combine multiple means or steps into one, or to divide them. It is. Furthermore, the configurations shown in the embodiments described above may be combined as appropriate.

A program, an information processing method, and an information processing device according to the present disclosure can be utilized as a program, an information processing method, and an information processing device that generate reference data from sound source data of a song.

100 Server 110 Server communication section 111 Server transmission section 112 Server reception section 120 Server storage section 121 Sound source data storage section 122 Lyrics data storage section 123 Reference data storage section 130 Server control section 131 Sound source data acquisition section 132 Lyrics data acquisition section 133 Singing section Specification unit 134 Reference data generation unit 135 Accompaniment part acquisition unit 136 Singing voice acquisition unit 137 Evaluation unit 141 Singing voice extraction unit 142 Text conversion unit 143 Collation unit 150 Singing voice separation extraction unit 200 Terminal 210 Terminal communication unit 211 Terminal transmission unit 212 Terminal Receiving unit 220 Terminal storage unit 230 Terminal control unit 231 Playback processing unit 232 Display processing unit 233 Singer voice acquisition unit 240 Input/output unit 250 Display unit 260 Microphone 270 Speaker 280 Camera 300 Network 400 Communication system

Claims (6)

A program to be executed on a computer of an information processing device,
a sound source data acquisition step of acquiring sound source data including a melody part of the song;
a singing voice separation and extraction step of separating and extracting first singing voice data of a first singer and second singing voice data of a second singer different from the first singer from the sound source data;
The first reference data of the part of the melody part sung by the first singer is generated based on the first singing voice data, and the first reference data of the part of the melody part sung by the first singer is generated based on the second singing voice data. A reference data generation step of generating second reference data of a portion sung by two singers is executed by the computer of the information processing device. The program according to claim 1,
a singing voice acquisition step of acquiring singing voice data of a singer;
An evaluation step of comparing and evaluating the singing voice data acquired in the singing voice acquisition step with either the first reference data or the second reference data is further executed by the computer of the information processing device. The program according to claim 1,
First accompaniment data in which a portion of the melody part sung by the first singer is canceled from the sound source data, and a portion of the melody part sung by the second singer is canceled from the sound source data. The computer of the information processing device further executes the step of generating the second accompaniment data. 4. The program according to claim 3,
a singing voice acquisition step of acquiring singing voice data of a singer;
When the first accompaniment data is played back, the singing sound data acquired in the singing sound acquisition step is evaluated by comparing it with the first reference data, and when the second accompaniment data is played back, the singing sound data is evaluated by comparing it with the first reference data. An evaluation step of comparing and evaluating the singing voice data acquired in the acquisition step with the second reference data is further executed by the computer of the information processing device. An information processing method executed by a computer of an information processing device, the method comprising:
a sound source data acquisition step of acquiring sound source data including a melody part of the song;
a singing voice separation and extraction step of separating and extracting first singing voice data of a first singer and second singing voice data of a second singer different from the first singer from the sound source data;
The first reference data of the part of the melody part sung by the first singer is generated based on the first singing voice data, and the first reference data of the part of the melody part sung by the first singer is generated based on the second singing voice data. and a reference data generation step of generating second reference data of a portion sung by two singers. An information processing device,
a sound source data acquisition unit that acquires sound source data including a melody part of a song;
a singing voice separation extraction unit that separates and extracts first singing voice data of a first singer and second singing voice data of a second singer different from the first singer from the sound source data;
The first reference data of the part of the melody part sung by the first singer is generated based on the first singing voice data, and the first reference data of the part of the melody part sung by the first singer is generated based on the second singing voice data. and a reference data generation unit that generates second reference data of a portion sung by two singers.

Images