WO2016017623A1

WO2016017623A1 - Reference display device, reference display method, and program

Info

Publication number: WO2016017623A1
Application number: PCT/JP2015/071347
Authority: WO
Inventors: 紀行畑
Original assignee: ヤマハ株式会社
Priority date: 2014-07-28
Filing date: 2015-07-28
Publication date: 2016-02-04
Also published as: TW201610980A; JP2016031395A

Abstract

Provided are a reference display device and program for displaying in a more intuitive and easier-to-understand manner. A CPU (11) displays guide images from a 3D perspective in which the planar direction corresponds to a musical scale, and the depth direction corresponds to time. Guide images corresponding to reference data are displayed on the screen. A corresponding character is displayed in each of the guide images. Obtained points corresponding to score results are displayed in the upper section of the screen. The CPU (11) causes the guide images and the background to scroll in a manner such that the character images move in the depth direction along the guide images, or in other words, along the musical interval for a singer.

Description

Reference display device, reference display method and program

The present invention relates to a display device, and more particularly to a device for displaying a model (reference).

Conventionally, a karaoke apparatus displays lyrics and a model pitch on a display unit (see, for example, Patent Document 1). The pitch is displayed as a so-called piano roll. A piano roll is a linear image on the screen where the vertical axis represents the scale (the piano keyboard is vertical) and the horizontal axis corresponds to the time. Is displayed. Thereby, the singer can grasp | ascertain visually the timing and pitch which sing.

Japanese Unexamined Patent Publication No. 2004-205817

However, conventional piano rolls are not always intuitive and easy to understand. For example, when performing a duet singing by a plurality of people, even if an attempt is made to display each singer's piano roll on the same screen, the linear images overlap when the pitches overlap, making it difficult to see.

Therefore, an object of the present invention is to provide a reference display device, a reference display method, and a program that perform a more intuitive and easy-to-understand display.

The reference display device of the present invention includes a display unit and an image processing unit that displays a guide image indicating a sound generation timing, a pitch, and a sound generation length on the display unit based on the reference data.

Then, the image processing unit displays the guide image from a three-dimensional viewpoint in which the depth direction corresponds to the time axis and the plane direction corresponds to the pitch. The three-dimensional viewpoint includes a subjective viewpoint that is a display mode imitating the user's own visual field. An aspect in which an image (character image or user image) corresponding to the user himself / herself is displayed and the character image or the like is displayed from behind is also equivalent to a three-dimensional viewpoint.

As a result, a guide image serving as a model is displayed in a three-dimensional space including the subjective viewpoint of the user (singer, speaker, performer, etc.). The user can more intuitively understand the sound generation timing and pitch of each sound, and can sing while having fun.

The image processing unit displays a user image corresponding to the reference data (a photograph taken by the user, a character image, etc.) at a position corresponding to the current sounding timing, and the image corresponding to the user follows the guide image. The guide image may be scrolled so as to move. In this case, the user can feel as if the character or the like is moved in accordance with his / her pronunciation, and can enjoy singing, language learning, or playing.

Further, when performing a duet song, for example, the reference display device displays the reference data for each singer as a plurality of guide images side by side from the three-dimensional viewpoint. Since the present invention displays the guide images side by side from a three-dimensional viewpoint, the guide images do not overlap even on the same screen, and high visibility can be obtained.

It is also possible to display a character corresponding to itself and a character corresponding to another singer in parallel. In this case, since the other singers are displayed in the three-dimensional space with depth, the user can more feel the atmosphere of singing with other singers.

The reference display device includes input means for inputting a user's pronunciation result, and scoring means for scoring by comparing the user's pronunciation result with the reference data, and the image processing unit includes the scoring It is preferable to display the scoring result of the means together with the guide image.

In this case, since the display image changes according to the result of the user's own singing or the like, the user can perform singing or the like while having fun.

The guide image includes an elongated linear image extending in the depth direction or a plate-like image having a width in the plane direction and extending in the depth direction.

The reference display device or the reference display method of the present invention can display more intuitively and easily.

It is the block diagram which showed the structure of the karaoke system. It is the block diagram which showed the structure of the karaoke apparatus. It is the block diagram which showed the structure of the center. It is a figure which shows the structure of the various data containing reference data. It is a figure which shows the example of a display of a reference. It is a figure which shows the example of a display of a reference. It is a figure which shows the example of a display of a reference. It is a flowchart which shows operation | movement of a karaoke apparatus. It is a display mode of a reference concerning another example. It is a display mode of a reference concerning another example. It is a display mode of a reference concerning another example. It is the block diagram which showed the minimum structure of the reference display apparatus.

FIG. 1 is a diagram showing a configuration of a karaoke system provided with a reference display device of the present invention. The karaoke system includes a center (server) 1 connected via a network 2 such as the Internet and a plurality of karaoke stores 3.

Each karaoke store 3 is provided with a relay device 5 such as a router connected to the network 2 and a plurality of karaoke devices 7 connected to the network 2 via the relay device 5. The repeater 5 is installed in a management room of a karaoke store. A plurality of karaoke apparatuses 7 are installed in each private room (karaoke box). Each karaoke device 7 is provided with a remote controller 9.

The karaoke device 7 can communicate with other karaoke devices 7 via the relay 5 and the network 2. A karaoke system can communicate between karaoke apparatuses 7 installed in different places, and can perform a duet among a plurality of singers.

FIG. 2 is a block diagram showing the configuration of the karaoke apparatus. FIG. 3 is a block diagram showing the configuration of the center 1.

The karaoke device 7 corresponds to the reference display device of the present invention. The karaoke device 7 includes a CPU 11, a RAM 12, an HDD 13, a network interface (I / F) 14, an LCD (touch panel) 15, a microphone 16, an A / D converter 17, a sound source 18, a mixer (effector) 19, and a sound system (SS) 20. , A speaker 21, an MPEG decoder 22, a display processing unit 23, a monitor 24, an operation unit 25, and a transmission / reception unit 26.

The CPU 11 that controls the operation of the entire apparatus includes a RAM 12, an HDD 13, a network interface (I / F) 14, an LCD (touch panel) 15, an A / D converter 17, a sound source 18, a mixer (effector) 19, and a decoder 22 such as an MPEG. The display processing unit 23, the operation unit 25, and the transmission / reception unit 26 are connected.

The HDD 13 stores an operation program for the CPU 11. In the RAM 12 as a work memory, an area to be read for executing the operation program of the CPU 11, an area to read music data to play karaoke music, an area to read reference data such as a guide melody, a reservation list, a scoring result, etc. An area for temporarily storing the data is set.

Further, the HDD 13 stores music data for playing karaoke music. Further, the HDD 13 also stores video data for displaying a background video on the monitor 24. The video data stores both moving images and still images. The video data includes a character image to be described later. The music data and the video data are regularly distributed from the center 1 and updated.

The center 1 includes a CPU 31, a RAM 32, an HDD 33, and a network interface (I / F) 34, as shown in FIG. The HDD 33 stores an operation program for the CPU 31. In the RAM 32 which is a work memory, an area to be read for executing the operation program of the CPU 31 is set. The CPU 31 is a control unit that comprehensively controls the center 1. CPU31 performs the connection process which connects each karaoke apparatus 7, when performing a duet via a network. Moreover, the distribution process of the song data memorize | stored in HDD33 is performed.

The HDD 33 stores music data and video data for distribution. Further, the HDD 33 stores information on each singer such as a name. For example, when each singer operates a karaoke apparatus and inputs the name of a specific singer, the information on the singer is transmitted to the center 1. CPU31 searches the name which concerns on the said singer from the singer information memorize | stored in HDD33. CPU31 connects the karaoke apparatus which input the name of the singer, and the karaoke apparatus which the searched singer uses. Thereby, duet singing via a network is realized.

The HDD 33 further stores song data and scoring data. The singing data is, for example, recorded singing sound when each singer has sung in the past. The scoring data indicates the scoring results at the time of singing. These singing data and scoring data are distributed when each singer makes a request. Thereby, each singer can perform a duet with the singing sound when he sang in the past, for example. The singing data and the scoring data may be sung by a singer or a voice actor, for example. Thereby, each singer can perform a duet with a favorite singer or a voice actor.

Returning to FIG. 2, the karaoke apparatus 7 will be described. The CPU 11 is a control unit that comprehensively controls the karaoke apparatus, and functionally incorporates a sequencer to perform karaoke performance. The CPU 11 performs audio signal generation processing, video signal generation processing, scoring processing, and guide image display processing. Thereby, the CPU 11 functions as an image processing unit in the present invention.

The touch panel 15 and the operation unit 25 are provided on the front surface of the karaoke apparatus. The CPU 11 displays an image corresponding to the operation information on the touch panel 15 based on the operation information input from the touch panel 15 to realize a GUI. The remote controller 9 also realizes the same GUI. The CPU 11 performs various operations based on operation information input from the remote controller 9 via the touch panel 15, the operation unit 25, or the transmission / reception unit 26.

Next, a configuration for performing karaoke performance will be described. As described above, the CPU 11 functionally includes a sequencer. The CPU 11 reads music data corresponding to the music number of the reserved music registered in the reserved list in the RAM 12 from the HDD 13, and performs a karaoke performance with the sequencer.

For example, as shown in FIG. 4, the music data includes a header in which a music number is written, a musical sound track in which performance MIDI data is written, a guide melody track in which MIDI data for guide melody is written, and lyrics It consists of a lyrics track in which MIDI data is written, a chorus track in which back chorus playback timing and audio data to be played are written, a breath position track indicating the timing of breathing, a technique position track indicating the timing of the singing technique, and the like. ing. The guide melody track, breath position track, and technique position track correspond to the reference data of the present invention. Reference data is model data for a singer to use as a reference for singing, and includes information indicating the sounding timing, pitch, and sounding length of each sound. Note that the format of the music data is not limited to this example. For example, existing music data without a breath position track and a technique position track may be prepared, and the breath position track and the technique position track may be prepared as separate data. In this case, it is not necessary to prepare new music data including the breath position track and the technique position track. However, song identification information such as a song number is described in the data of the breath position track and the technique position track. When reading the music data, the CPU 11 reads the corresponding breath position track and technique position track, and performs a sequence operation. Also, the format of the reference data is not limited to the MIDI format as described above. For example, the reference data indicating the breath position may be text data indicating the timing of the breath position (elapsed time from the beginning of the music). In addition, when the reference data is voice data (for example, recorded singing sound), the pitch is extracted from the voice data to extract the pitch, and the sound is generated from the timing and length at which the pitch is extracted. It is also possible to extract timing and pronunciation length. It is also possible to detect the silent section by detecting the volume (power), and when there is a silent section between each sound, the timing at which the silent section is extracted can be extracted as the timing of the breath position It is. In addition, when the pitch fluctuates regularly within a predetermined period, the timing (technical position) at which the singing technique is performed is extracted by determining that “vibrato” is being performed for the institution. It is also possible.

The musical sound track stores information indicating the type, timing, pitch (key), strength, length, localization (pan), sound effect (effect), etc. of the musical instrument that generates the musical sound. In the guide melody track, information such as the sounding start timing of each sound corresponding to the model song and the length of the sounding are recorded.

The sequencer controls the sound source 18 based on the data of the musical tone track, and generates the musical tone of the karaoke song.

Also, the sequencer plays back chorus audio data (compressed audio data such as MP3 attached to music data) at the timing specified by the chorus track. Further, the sequencer synthesizes the character pattern of the lyrics in synchronism with the progress of the song based on the lyrics track, converts the character pattern into a video signal, and inputs it to the display processing unit 23.

The sound source 18 forms a musical sound signal (digital audio signal) according to data (note event data) input from the CPU 11 by processing of the sequencer. The formed tone signal is input to the mixer 19.

The mixer 19 generates an acoustic effect such as an echo on the musical sound signal generated by the sound source 18, the chorus sound, and the singing voice signal of the singer input from the microphone (singing voice input means) 16 via the A / D converter 17. And mixing these signals.

In addition, when karaoke devices 7 installed at different places communicate with each other and perform a duet, a singing voice signal is transmitted from another karaoke device. The singing voice signal received from the other karaoke apparatus is also input to the mixer 19 and mixed with the singing voice signal input from the microphone 16 of the own apparatus.

Each mixed digital audio signal is input to the sound system 20. The sound system 20 includes a D / A converter and a power amplifier, converts an input digital signal into an analog signal, amplifies it, and emits the sound from a speaker (musical sound generating means) 21. The effect that the mixer 19 gives to each audio signal and the balance of mixing are controlled by the CPU 11.

The CPU 11 reads the video data stored in the HDD 13 and reproduces the background video and the like in synchronism with the generation of musical sounds and the generation of the lyrics telop by the sequencer. The video data of the moving image is encoded in the MPEG format.

Further, the CPU 11 can also download picture data representing a singer or video data such as a character from the center 1 and input it to the display processing unit 23. The photograph representing the singer can be taken on the spot with a camera (not shown) provided on the karaoke device or the remote control 9, or with a camera provided on a mobile terminal owned by the user. is there.

The CPU 11 inputs the read video data of the background video to the decoder 22. The decoder 22 converts the input data such as MPEG into a video signal and inputs it to the display processing unit 23. In addition to the video signal of the background video, the display processing unit 23 also receives the video signal of the guide image and the video signal of the character image based on the guide melody track, together with the character pattern of the lyrics telop.

In the conventional piano roll, the vertical axis is the scale (the piano keyboard is vertical), and the horizontal axis is the time corresponding to the sound start timing and the length of the sound. The image was displayed. However, in this embodiment, as shown in FIG. 5A, the guide image is displayed from a three-dimensional viewpoint in which the depth direction corresponds to the time axis and the planar direction (in this example, the vertical direction) corresponds to the musical scale. The three-dimensional viewpoint is a display mode that imitates the user's own visual field. As shown in FIG. 5A, an aspect in which an image (character image or the like) corresponding to the user himself / herself is displayed and the character image or the like is displayed from behind is also equivalent to a three-dimensional viewpoint.

The large spherical image 502 displayed overlapping the guide image in the figure is the sound generation timing of each sound in the guide melody track. As a result, the user can grasp that the sound is generated at the timing when the spherical image 502 is displayed. The small spherical image 503 corresponds to the timing of scoring, and a score is given if the pitches of the singing sound and the guide melody match. When a score is given, a star-shaped image 505 is displayed so as to flow, and the score is added to the score column 504 displayed at the upper right in the figure. In addition, when the pitches are continuously matched, for example, “25 combo” is displayed as shown in the upper part of the figure, thereby grasping the number of pitches that can be matched continuously. be able to.

The guide image referred to in the present invention is not limited to the long and narrow line as shown in FIG. 5B, but as shown in the example of FIG. 5A, a plate shape having a certain width in the left-right or vertical direction. In which the image extends in one direction (in this example, the depth direction).

In the present embodiment, the guide images corresponding to the reference data (in FIG. 5A, the guide image 501A, the guide image 501B, and the guide image 501C) are displayed so as to extend in the depth direction. Further, in each guide image 501A, guide image 501B, and guide image 501C, the corresponding character image 101A, character image 101B, and character image 101C are displayed. The score corresponding to the scoring result is displayed at the top of the screen.

First, the CPU 11 generates a guide image based on information on the sound generation start timing and the sound generation length of each sound included in the guide melody track. Then, the CPU 11 smoothly connects the guide images for each sound. The inclination of the connection part of each sound is displayed as an image having the same inclination uniformly, for example, by connecting with a time length corresponding to a sixteenth note. However, there are actually different ways of singing for each song, and the manner of changing the pitch is not uniform. Therefore, it is preferable to display with a different inclination for each connection portion of each sound. In this case, the reference data may include information that specifies an inclination according to a change in the pitch of the connected portion of each sound. The guide image may be read and displayed as image data (corresponding to the guide image) prepared for each song in advance.

After that, the CPU 11 disperses the guide image at the breathing timing indicated by the breath position track. For example, in the conventional objective viewpoint, as shown in FIG. 5B, since there is a breathing timing between the pronunciation of “Akai” and the pronunciation timing of the beginning of “Hanaga”, “Akai” Are separated from the guide image of “Hanaga”.

Then, the CPU 11 displays the guide image from a three-dimensional viewpoint as shown in FIG. 5A. Thereby, the singer can grasp the pronunciation timing and pitch of each sound more intuitively. Note that it is not essential in the present invention to connect the guide images smoothly. However, by smoothly connecting the guide images, the singer can visually grasp how to connect each sound and the timing of breathing. In addition, since the CPU 11 makes the guide image discrete at the breathing timing indicated by the breath position track (because the reference data includes information indicating the breathing timing), the CPU 11 clearly distinguishes between the silent section and the breathing section. This allows the user to know the exact breathing position.

Also, the CPU 11 displays a character at the current singing position. In this example, the character image 101A is displayed at the lower center of the screen, and the guide image and background are displayed so that the character image 101A moves in the depth direction along the guide image (that is, along the pitch of the singer). Scroll. When the singer sings the pitch by mistake, the character image 101A is displayed at a position away from the guide image, and the singer can intuitively understand how far the pitch has deviated. it can.

Further, the CPU 11 displays a score according to the scoring result. Scoring is performed by comparing the singer's singing voice with the guide melody track. Scoring is performed by comparing the pitch of the singing voice and the guide melody for each note of the guide melody track. That is, a high score is given when the pitch of the singing voice matches the pitch of the guide melody track for a predetermined time or longer (within the allowable range). The timing of pitch change is also taken into account. Further, points are added based on the presence or absence of singing techniques such as timing of pitch change, vibrato, inflection, and squealing (moving gently from a low pitch).

In the scoring process, whether or not the singer performed breathing at the breathing timing included in the breath position track may be added. Whether or not breathing has been performed depends on whether no sound is input from the microphone 16 within a predetermined time including the breathing timing (the input level is less than a predetermined threshold) or when a breathing sound is collected. When the voice is input from the microphone 16 (the input level is equal to or higher than the predetermined threshold value), it is determined that breathing is not performed. Whether or not the breathing sound has been collected is determined by comparing with the waveform of the breathing sound, for example, by pattern matching or the like. Note that the CPU 11 may perform display such that the character image falls from the ground (where the guide image is interrupted) when no breathing is detected.

Also, in the scoring process of this embodiment, it is preferable to give a higher score when the same technique is detected at the timing of each technique included in the technique position track.

The scoring process may be performed at each karaoke apparatus, but may be performed at the center 1 (or another server). Moreover, when performing duet with another karaoke apparatus via a network, you may perform a scoring process with one karaoke apparatus which processes typically.

As described above, the singer can enjoy the karaoke like a game because the character image 101A moves according to his / her singing and the scoring result of his / her singing can be confirmed by scoring. Can do.

In the example of FIG. 5A, when performing a duet singing, a character image 101A corresponding to itself and a character image 101B and character image 101C corresponding to other singers are displayed side by side. The music data for duet music includes a plurality of reference data (guide melody tracks) for a plurality of singers. Therefore, the CPU 11 displays guide images corresponding to each guide melody track side by side, and displays a character image corresponding to each guide image. Thereby, the user can feel more the atmosphere of singing with other singers. Moreover, if a score is added according to the scoring result of each singer and it is set as an aspect, it can feel that each singer is earning a score while cooperating, and a sense of unity can be obtained more.

It should be noted that a sensor that can detect the attitude of the microphone 16 such as a gyro sensor or an acceleration sensor may be provided, and the image of the three-dimensional viewpoint may be changed according to the attitude of the microphone 16. For example, in the example of FIG. 5A, the character image 101A is displayed near the left and right centers, but when the singer points the microphone 16 to the right, the character image and the guide image on the screen move to the left. Then, display is performed such that the character image 101C approaches the left and right centers. As a result, when the singer turns to the right, the display of the three-dimensional viewpoint is also turned to the right, and a more realistic and intuitive display mode is obtained.

When the singer selects music data that is not for a duet song (when the same melody is sung by different singers), the CPU 11 displays, for example, the same guide images side by side, and characters corresponding to the respective guide images. Display an image.

In the example of FIG. 5A, the vertical direction is displayed with a three-dimensional viewpoint corresponding to the musical scale, but for example, as shown in FIG. 6, the horizontal direction may be displayed with a three-dimensional viewpoint corresponding to the musical scale. . In this case, the scale is lower in the left direction and higher in the right direction.

In the present embodiment, an example in which the guide image is displayed from a three-dimensional viewpoint has been described. However, for example, as illustrated in FIG. 8, the guide image may be displayed from an objective viewpoint. In the example of FIG. 8, the left-right direction corresponds to the time axis, and the up-down direction corresponds to the musical scale. In this example, the guide image corresponds to the ground image, and the ground image is interrupted at the breath connection timing. Then, the guide image and the background are scrolled so that the character image 101A moves along the guide image (ground image). At the breath connection timing, since the ground is interrupted, the character image 101A falls from the ground when the breath connection (state in which no sound is input from the microphone 16) is not detected. Also in this example, the singing result is displayed on the screen. Therefore, also in this example, the singer can enjoy karaoke like a game.

Next, the operation of the karaoke system will be described with reference to a flowchart. FIG. 7 is a flowchart showing the operation of the karaoke system.

First, the CPU 11 receives a request for music from a singer (s11). Further, the CPU 11 determines whether or not it is a request for a duet song via the network (s12). If the request is duet duet via the network, a connection request is made to the center 1 (s13). At this time, the CPU 11 displays an image prompting whether or not to perform a duet with a singer of another karaoke apparatus connected via the network on the monitor 24, and accepts a duet song via the network. For example, when a singer inputs the name of a specific user using the touch panel 15, the operation unit 25, or the remote control 9, information related to the name is transmitted to the center 1.

The center 1 receives the connection request (s51), reads the singer information stored in the HDD 33, and performs connection processing with the karaoke apparatus of the corresponding singer (s52). Thereby, each karaoke apparatus is connected with another karaoke apparatus (s14), and a duet song is attained by transmitting / receiving singing data of each singer.

Next, the CPU 11 of the karaoke apparatus 7 reads the requested music data (s15) and performs a drawing process (s16). That is, the CPU 11 reads the video data stored in the HDD 13, generates a background video, and synthesizes a character image with the background video. Furthermore, the CPU 11 generates a guide image based on information on the sound generation start timing and the sound generation length of each sound included in the guide melody track. The CPU 11 reads the breath timing information from the breath position track, smoothly connects the guide images of each sound, and displays the guide images in a discrete manner at the breath timing indicated by the breath position track. The guide image is synthesized with a background video in correspondence with each character image. In the case of a duet song, a plurality of guide images and character images are displayed side by side.

The CPU 11 preferably displays the guide image related to the phoneme of the prompt sound and the guide image after the phoneme of the prompt sound in a discrete manner. The prompt sound is represented by “t” or “t” in Japanese kana notation, and is silent between the following sound. The CPU 11 extracts the prompt sound from the lyrics track, and disperses the guide image at the timing of emitting the extracted prompt sound. For example, if the lyrics are “if you use”, there is a prompt sound after “ka”, so the guide image of “ka” and the subsequent guide image of “ta” are displayed separately. In addition, an image indicating that there is a phoneme related to the prompting sound (in this example, a square image with “tsu”) may be further displayed.

Further, it is preferable that the CPU 11 reads the technique position track and displays the singing technique on the guide image. For example, in the example of FIG. 5A, the beginning is a portion where a technique of scrambling (singing while lifting from a pitch lower than the reference pitch) is performed. Therefore, an image 506 is displayed that is reminiscent of a rising pitch like “no”. Further, as shown in FIG. 6, for the vibrato section, in addition to the guide image of the wave guide, a wave guide image 507 such as “m” is separately displayed. Thereby, the singer can grasp | ascertain easily what kind of singing technique is performed at which timing.

Note that the vibrato section may be displayed by changing the guide image to a wavy line. Further, when the “position” information is included in the technique position track, the head position of the sound corresponding to the “time” information (in this example, the “ka” sound) is delayed. This makes it easier for the singer to intuitively understand the “tame” that intentionally delays the singing of the specific sound (“ka” sound in this example). If the technique position track includes “Kobushi” information, the guide image is temporarily raised at a position corresponding to the timing of the “Kobushi”. Thereby, it is also possible to obtain a guide image corresponding to “Kobushi”, which is a singing technique for changing the voice color of a specific sound so as to beat in the middle of pronunciation. When the technique position track includes “fall” information, the “fall” singing technique is performed by changing the guide image to a lower pitch from the position corresponding to the timing of “fall”. It is also possible to use a guide image corresponding to.

Further, the CPU 11 may display an image (for example, an image such as “V”) for prompting breathing together with the guide image based on information indicating breathing timing indicated by the breath position track.

Further, the CPU 11 may read information indicating the volume of each sound included in the guide melody track, and perform a process of changing to a guide image corresponding to the volume of each sound. For example, the thickness of the line is changed according to the volume, or the color of the line is changed according to the volume.

When different lyrics are continuously generated with the same pitch, if the guide images are connected, it may be difficult for the user to grasp when to pronounce the next lyrics. Therefore, as shown in FIG. 5A, the CPU 11 displays, for example, a spherical image superimposed on the guide image at the sound generation timing of each sound in the guide melody track. Thereby, the user can grasp | ascertain that sounding is performed at the timing when the said spherical image is displayed. In the example of FIG. 5A, a small spherical image is displayed at the beat timing, and a large spherical image is displayed at the sounding timing.

Further, in the drawing process, the CPU 11 reads the lyrics track and displays the lyrics image in association with each guide image.

Next, the CPU 11 performs a scoring process as the karaoke performance progresses (s17). The scoring process is performed by comparison with the reference data as described above.

The CPU 11 transmits the scoring result in the scoring process and the singing data related to the singing sound input from the microphone 16 to the center 1 (s18). The center 1 receives the scoring results and singing data from each karaoke device 7 and transmits the singing data to another corresponding karaoke device 7 (s53). Thereby, the singing voice in each karaoke device 7 is transmitted to the other karaoke devices 7. In each karaoke apparatus 7, its own singing voice and the singing voice of the duet partner are mixed and emitted from the speaker 21. Further, the center 1 aggregates the scoring results received from each karaoke device 7 and then transmits it to each karaoke device 7. Thereby, in each karaoke apparatus 7, the scoring result which totaled its own scoring result and the duet partner's scoring result can be obtained, for example, the score according to the scoring result after summation is displayed.

The above processing is performed until the end of the music performance. The CPU 11 of the karaoke device 7 determines whether or not the performance of the music has ended (s19), and scrolls the guide image and moves the character image according to the progress of the music until the performance of the music is completed. Moreover, the score according to the scoring result is displayed. Similarly, the CPU 31 of the center 1 determines whether or not the performance of the music has ended (s54), and transmits and receives song data and scoring results from each karaoke device 7 until the performance of the music is completed.

In the present embodiment, the karaoke device 7 is used to perform the karaoke performance and display the guide image from a three-dimensional viewpoint. The reference display device of the present invention can be realized also by using a speaker and a display and a display unit. Note that the music data and the reference data need not be stored in the display device, and may be downloaded from the server each time and used.

Moreover, although the example which displays the guide melody in karaoke as a guide image was shown in this embodiment, for example, the change of the pitch of the model of a wind instrument performance is displayed as a guide image, and it is the same also as an aspect which a guide image interrupts at a breathing timing. The effect is obtained. Further, for example, in language learning, the same effect can be obtained by displaying a guide image indicating the pronunciation timing and length of the model and interrupting the guide image at the breath timing and the prompt sound.

Although FIG. 5 shows a mode in which a character image corresponding to the user himself and a character image corresponding to another user are respectively displayed, displaying a plurality of guide images and a plurality of character images is not limited to the present invention. Is not essential. For example, as shown in FIG. 9A, only the guide image may be displayed from a three-dimensional viewpoint in which the depth direction corresponds to the time axis and the plane direction corresponds to the pitch without displaying the character image. Although the lyrics are also displayed in the example of FIG. 9A, the display of the lyrics is not essential in the present invention. Also in this case, as shown in FIG. 9B, since the guide image extending in the depth direction is displayed so as to approach the user as time elapses, the user can intuitively know the sound generation timing and pitch of each sound. Singing and playing musical instruments while having fun.

As shown in FIG. 10, the reference display device of the present invention includes a monitor 24 that is a display unit and a CPU 11 that functions as an image processing unit that performs a guide image display process, and the CPU 11 is stored in the HDD 13. The guide image based on the music data (which is an example of the reference data of the present invention) may be displayed in a three-dimensional viewpoint. Other hardware configurations are not essential elements in the present invention.

As described above, the reference data does not need to be stored in the HDD 13 and may be downloaded and used from the outside (for example, a server) each time. In addition, the decoder 22, the display processing unit 23, and the RAM 12 may be incorporated in the CPU 11 as a part of the function of the CPU 11.

This application claims priority based on Japanese Patent Application No. 2014-152480 filed on July 28, 2014, and is incorporated herein by reference.

DESCRIPTION OF SYMBOLS 1 ... Center 2 ... Network 3 ... Karaoke store 5 ... Relay machine 7 ... Karaoke apparatus 9 ... Remote control 11 ... CPU
12 ... RAM
13 ... HDD
DESCRIPTION OF SYMBOLS 15 ... Touch panel 16 ... Microphone 17 ... A / D converter 18 ... Sound source 19 ... Mixer 20 ... Sound system 21 ... Speaker 22 ... Decoder 23 ... Display processing part 24 ... Monitor 25 ... Operation part 26 ... Transmission / reception part 31 ... CPU
32 ... RAM
33 ... HDD
101A, 101B, 101C ...

character images

501A, 501B, 501C ... guide images

Claims

A display unit;
Based on the reference data, an image processing unit that displays a guide image indicating the sound generation timing, pitch, and sound generation length on the display unit;
A reference display device comprising:
The image processing unit is a reference display device that displays the guide image from a three-dimensional viewpoint in which a depth direction corresponds to a time axis and a planar direction corresponds to a pitch.
The reference display device according to claim 1, wherein the image processing unit displays different reference data side by side as the plurality of guide images from the three-dimensional viewpoint.
The image processing unit displays a user image corresponding to the reference data at a position corresponding to a current pronunciation timing, and scrolls the guide image so that the user image moves along the guide image. 3. The reference display device according to 1 or 2.
An input means for inputting a user's pronunciation result;
Scoring means for scoring by comparing the user's pronunciation results with the reference data;
The reference display device according to claim 1, wherein the image processing unit displays a scoring result of the scoring unit together with the guide image.
5. The reference display device according to claim 1, wherein the image processing unit displays the guide image from a three-dimensional viewpoint in which a vertical direction or a horizontal direction in the display unit corresponds to a pitch.
The reference display device according to any one of claims 1 to 5, wherein the guide image includes an elongated linear image extending in a depth direction, or a plate-like image having a width in a plane direction and extending in the depth direction.
A reference display method in an information processing apparatus including a display unit,
Based on the reference data, a guide image showing the sound generation timing, pitch, and sound generation length is displayed on the display unit,
A reference display method in which the guide image is displayed from a three-dimensional viewpoint in which a depth direction corresponds to a time axis and a planar direction corresponds to a pitch.
In an information processing device equipped with a display
A program for executing an image processing step for displaying a guide image indicating a sound generation timing, a pitch, and a sound generation length on the display unit based on reference data,
The image processing step is a program for displaying the guide image from a three-dimensional viewpoint in which the depth direction corresponds to the time axis and the plane direction corresponds to the pitch.