JP2011013446A - Musical score display device, musical score display method, computer program, and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a musical score display device capable of automatically changing pages at desired timing without performing specific operation, regardless of musical instruments performed at the same time and the number of players.SOLUTION: The musical score display device includes: a storage function for storing a series of page images of a musical score together with a musical score name and the order of the page image; an image taking-in function (S422) for outputting a series of pieces of image data in which a subject is imaged, in real time; a phrase data storage function for storing the motion feature of a player when a certain phrase is played, by associating it with a page image corresponding to the musical score name, as a phrase data; and a page changing function (S426, S430) which displays the page image of next order following the displayed page image, when the motion feature of the player extracted from the output of the image taking-in function matches the phrase data which are associated with the displayed page image, when the page image of the musical score is displayed on the display device.

Description

  The present invention relates to a score display apparatus that displays a musical score used when playing a musical instrument, and relates to a score display apparatus having a page turning function for automatically switching pages to be displayed.

  When performing while looking at a musical score that spans multiple pages, it is necessary to turn the musical score as the performance progresses. In concerts with keyboard instruments, there are cases where a staff in charge of turning pages is set up separately from the performers. However, the page is usually turned by the performer's own hand at a timing that does not interfere with the performance. It is good if the timing of turning pages is rest, but if it is not, it needs to be done quickly during the performance, which is very troublesome for the performer.

  In order to solve the problem, there are a device for switching a musical score to the next page by depressing a pedal switch, and a score display device for changing a score display by a key operation if it is an electronic score. However, operating a device that is not related to performance for page switching, on the contrary, increases the burden on the performer. In order to solve these problems, the following techniques are disclosed.

  Japanese Patent Application Laid-Open No. 2004-228688 discloses a music score display device equipped with a television camera that captures an image of a performer from the television camera and changes a musical score page to be displayed when a predetermined body sign is detected from the captured image. Is disclosed.

  Patent Document 2 stores score image data generated by photographing a score for a plurality of pages in a score display apparatus, and pitch data generated by a performance operation matches a pre-registered feature phrase. In some cases, a technique for automatically displaying the next page is disclosed. In Patent Document 2, the characteristic phrase to be registered is a part of the music. The number of matches between the pitch group detected from the performance and the pitch group of the registered feature phrase is counted, and if the two match, the progression of the song has reached the registered feature phrase and automatically follows the score. Display the page.

JP 2005-37751 A JP-A-2005-208156

  The above technique still has the following problems.

  In the technique of Patent Document 1, there is a problem that when a musical score page to be displayed is changed, a body sign unrelated to performance must be performed. The performance during performance varies depending on the performer. Therefore, even for a simple body sign, learning the body sign and putting out the sign during performance at the desired page turning timing has a problem that it is burdensome for the performer. Furthermore, the body sign must be changed between when going to the next page and when going back to the previous page. This technique is therefore impractical.

  The technique of Patent Document 2 is effective when pitch data can be obtained directly from a performance operation such as an electronic keyboard instrument. However, when playing other musical instruments, there is a problem that it is difficult to accurately detect pitch data. For example, when a plurality of performers perform simultaneously, sounds other than the performers who use the apparatus are detected at the same time. Therefore, it is extremely difficult to detect only the pitch of the device user.

  As described above, there is a need for a score display apparatus that can be used in a wide variety of musical instruments and can perform a page turning operation in a manner that does not burden the performer.

  Therefore, an object of the present invention is to provide a musical score display device capable of automatically switching pages at a timing desired by a user without performing a special operation during the performance, regardless of the musical instrument and the number of people playing at the same time. Is to provide.

  A score display apparatus according to a first aspect of the present invention includes a display device, a storage unit for storing a series of page images of a score together with a score name, an order of the series of page images of the score, and a subject. Image capture means for outputting a series of captured image data in real time, and the characteristics of the player's movement when a player plays a phrase in association with the score name and the page image corresponding to the score name Phrase data storage means for storing as phrase data and a series of page images of the score corresponding to the designated score name are read from the storage means, and a predetermined page image of the read page images is displayed on the display device. The musical score display means for displaying, and when the page image of the musical score is displayed on the display device, the characteristics of the player's movement extracted from the output of the image capturing means are displayed on the display device. Phrase determination means for outputting a determination signal that takes the first value when it matches the phrase data associated with the page image of the musical score shown, and the second value otherwise, and the determination signal In response to the first value being reached, page change means for displaying on the display device the page image in the order next to the page image displayed on the display device.

  The storage means stores a series of page images of the score together with the name of the score and the order of the series of page images of the score. The image capturing means outputs a series of image data obtained by imaging the subject in real time. The phrase data storage means stores the characteristics of the player's movement when the player performs a certain phrase as phrase data in association with the score name and the page image corresponding to the score name. The score display means reads out a series of page images of the score corresponding to the designated score name from the storage means, and displays a predetermined page image of the read page images on the display device. The phrase determination unit associates the feature of the player's movement extracted from the output of the image capturing unit with the page image of the score displayed on the display device when the page image of the score is displayed on the display device. A judgment signal is output that takes the first value when the phrase data matches the given phrase data, and takes the second value otherwise. In response to the determination signal having reached the first value, the page changing means displays the page image in the next order of the page images displayed on the display device on the display device.

  In this musical score display apparatus, a performer plays a phrase (specific phrase) of a portion of a musical score page to be displayed and captures image data at that time. The score display device extracts the characteristics of the player's movement from the captured image data, and stores it as phrase data in association with the score name of the score including the specific phrase and the page image corresponding to the score name. When performing a performance using the score display device, a page image of the score is displayed on the display device. The musical score display device is configured such that when the player's movement characteristic extracted from the output of the image capturing means matches the phrase data stored in association with the page image displayed on the display device, that is, the player selects the page. When it is determined that the movement of the phrase to be changed is the same as when the phrase is played, the page image in the next order of the page image is displayed on the display device.

  In this musical score display device, when the page of the musical score is changed, the performer need only perform in the same manner as when the specific phrase is registered, and does not need to perform a special operation. In addition, since the characteristic data of the player's movement is used for determining the page change, it is not affected by the sound of the musical instrument playing at the same time. As a result, it is possible to provide a musical score display device capable of automatically switching pages at a timing desired by a user without performing a special operation during the performance, regardless of the musical instruments and the number of people playing at the same time. .

  Preferably, the score display apparatus further includes an operation mode designating unit for designating any one of the first and second operation modes. When the first operation mode is specified by the operation mode specifying unit, the phrase data storage unit is configured to display the player's movement characteristics extracted from the output of the image capturing unit, the specified score name, Means for storing as phrase data in association with the designated page image of the page images corresponding to the score name is included. The phrase determining means is the player's extracted from the output of the image capturing means when the second operation mode is specified by the operation mode specifying means and the page image of the score is displayed on the display device. It includes determination means for determining whether or not the feature of movement matches phrase data associated with the page image of the score displayed on the display device and outputting a determination signal.

  The operation mode designating unit designates one of the first and second operation modes. The means for memorizing, when the first operation mode is specified by the operation mode specifying means, the player's movement characteristics extracted from the output of the image capturing means, the specified score name, It is stored as phrase data in association with the designated page image of the page images corresponding to the musical score name. The determining means is the movement of the player extracted from the output of the image capturing means when the second operation mode is specified by the operation mode specifying means and the page image of the score is displayed on the display device. Is determined to match the phrase data associated with the page image of the score displayed on the display device, and a determination signal is output.

  The score display device associates the player's movement characteristics extracted from the output of the image capturing means with a specified score name and a specified page image of the page images corresponding to the score name. Whether the operation mode when stored as phrase data and the player's movement characteristics extracted from the output of the image capture means match the phrase data associated with the page image of the score displayed on the display device And an operation mode when determining whether or not. As described above, the musical score display apparatus has two operation modes, so that it is possible to surely switch the operation of storing phrase data using the output of the image capturing means and determining the phrase.

  Preferably, the musical score display device further includes a sound detection means for detecting sound. The image capturing means includes means for starting imaging of the subject and outputting a series of image data of the subject in real time in response to the sound detecting means detecting the sound.

  In this musical score display apparatus, in response to the player actually performing the part for which the page change is desired and the sound detecting means detects the sound. The imaging of the subject is started and a series of image data of the subject is output in real time. By outputting image data in response to the detection of the sound of the performance sound, it is possible to accurately detect the start and end of the phrase to be played, and to obtain more accurately the characteristics of the player's movement when playing the phrase it can. As a result, the phrase data is less likely to include extra movement features other than during performance, and the accuracy in determining whether the player's movement features match the phrase data can be improved.

  Preferably, the musical score display device further includes a time measuring means for outputting a time expiration signal when a predetermined time has elapsed from the start of the time measurement. In response to the determination signal having reached the first value, the page changing means is configured to cause the time measuring means to start measuring time, and in response to the time expiration signal from the time measuring means, Means for displaying an image on a display device.

  The score display device starts timing from the time when it is determined that the player's movement characteristics match the phrase data, and in response to a signal output when a predetermined time elapses, the next page image is displayed. Is displayed on the display device. Thus, for example, if the phrase for changing the page is not in an appropriate place in the score to be displayed, the portion a little before the place to be changed is stored as phrase data, and after a predetermined time has elapsed since it was determined that they match You can change the page of the score. As a result, the display page can be changed in accordance with the performance flow.

  Preferably, the phrase data storage means indicates the characteristics of the player's movement when the performer plays a phrase, the score name, the page image corresponding to the score name, and, if any, the page image. Means for storing as phrase data in association with next page specifying information for specifying a page image to be displayed is included. The phrase determination unit associates the feature of the player's movement extracted from the output of the image capturing unit with the page image of the score displayed on the display device when the page image of the score is displayed on the display device. Determining means for determining whether or not it matches any of the phrase data received and outputting a determination signal; and outputting information specifying phrase data determined by the determining means to match the characteristics of the player's movement Means. In response to the determination signal having become the first value, the page changing means selects the page image specified by the next page image specifying information for the phrase data specified by the information output by the means for outputting. Means for displaying on the display device;

  The means for memorizing displays the characteristics of the player's movement when the player performs a certain phrase, the score name, the page image corresponding to the score name, and if present, the page image. It is stored as phrase data in association with the next page specifying information for specifying the page image to be processed. When the score page image is displayed on the display device, the judging means associates the characteristics of the player's movement extracted from the output of the image capturing means with the score page image displayed on the display device. It is determined whether or not it matches any of the phrase data, and a determination signal is output. The means for outputting outputs information specifying phrase data determined by the determining means to match the characteristics of the player's movement. The means for displaying the page specified by the next page image specifying information for the phrase data specified by the information output by the means for outputting in response to the determination signal having become the first value. The image is displayed on the display device.

  When there is next page specifying information for specifying a page image to be displayed next to the page image, the score displaying apparatus can store the page data in association with the phrase data. In addition, the score display device can determine whether it matches any of the phrase data associated with the page image of the score displayed on the display device, and outputs information specifying the phrase data determined to match. be able to. When the phrase data specified by the output specifying information is stored in association with the information specifying the next page, the page is displayed.

  In this way, by determining whether it matches any of the phrase data registered in association with the page image, from the characteristics of the player's movement when playing different phrases stored for the page, the display device The page to be displayed can be changed. Further, by storing the phrase data in association with the information of the page image to be displayed next, the page on which the phrase data is displayed after the performance can be changed. As a result, it is possible to deal with a musical score in which a page image to be displayed next changes due to a repetition symbol or the like described in the musical score.

  Preferably, the page changing unit further has no next page image specifying information for the phrase data specified by the information output by the outputting unit in response to the determination signal having become the first value. In addition, for the page image displayed on the display device, means for displaying on the display device the page image specified by the order of the page images is included.

  When there is no next page image specifying information in the phrase data, the page image to be displayed is changed according to the order of a series of page images of the score. As a result, the page can be changed even when the next page image specifying information is not registered.

  Preferably, the score display device specifies the phrase data by means for outputting each phrase data stored in association with each page image of the score while displaying the page image corresponding to the phrase data. Counting means for counting the number of times the information to be output is output is further included. The means for storing further includes the number of times counted by the counting means for each phrase data associated with the page image displayed on the display device, and information for specifying the page image to be displayed next. Remember the set further. The means for displaying is combined with the number of times counted by the counting means of the phrase data specified by the information output by the means for outputting in response to the determination signal having become the first value. Means for displaying a page image specified by the information being displayed on the display device.

  There are several types of repetition symbols in the score. Depending on the musical score, the place where the phrase is played following the phrase and the score page to be displayed may differ depending on the number of times the phrase is played. The number of times the information specifying the phrase data is output by the counting means, and for each phrase data associated with the page image, the counted number and the information specifying the page image to be displayed next By setting, it is possible to deal with a score having a complicated structure including a repetition symbol.

  A score display method according to a second aspect of the present invention includes a display device, a storage unit for storing a series of page images of a score together with a score name, a display order of the series of page images of the score, and a subject. Image capturing means for outputting a series of image data obtained by capturing images in real time, the characteristics of the player's movement when a player plays a phrase, the score name, and the page image corresponding to the score name The phrase data storage means for storing the phrase data in association with each other, and a series of page images of the score corresponding to the designated score name are read from the storage means, and a predetermined page image is read out of the read page images. The score display means for displaying on the display device, and the feature of the player's movement extracted from the output of the image capture means when the page image of the score is displayed on the display device is displayed. A phrase determination means for outputting a determination signal that takes a first value when it coincides with the phrase data associated with the page image of the musical score displayed on the screen; otherwise, a determination In a score display apparatus, comprising: a page changing means for displaying a page image in the next order of the page images displayed on the display device in response to the signal having become the first value. In the score display method, the score display means reads a series of page images of a score corresponding to a designated score name from the storage means, and displays a predetermined page image of the read page images on the display device. The step and the phrase judging means display on the display device the characteristics of the player's movement extracted from the output of the image capturing means when the page image of the score is displayed on the display device. A step of outputting a determination signal that takes the first value when it matches the phrase data associated with the page image of the musical score being recorded, and the second value otherwise, and the page changing means, Responsive to the first value being displayed, displaying a page image in the next order of the page images displayed on the display device on the display device.

  When the computer program according to the third aspect of the present invention is executed by a computer to which a display device and image capturing means for outputting a series of image data obtained by imaging a subject in real time are connected, the computer A storage means for storing a series of page images of a score together with the name of the score and the order of the sequence of page images of the score, and the characteristics of the movement of the player when the player plays a phrase, Phrase data storage means for storing as a phrase data in association with a score name and a page image corresponding to the score name, and reading a series of page images of a score corresponding to the designated score name from the storage means A score display means for displaying a predetermined page image on the display device, and a page image of the score is displayed on the display device. When the player's movement characteristic extracted from the output of the image capturing means matches the phrase data associated with the page image of the score displayed on the display device, the first value is set. In response to the phrase determination means for outputting a determination signal for each of the second values and the determination signal having reached the first value, the next of the page image displayed on the display device It functions as a page changing means for displaying the sequential page images on the display device.

  A recording medium according to the fourth aspect of the present invention is a computer-readable recording medium that records the above-described computer program.

  In this musical score display device, the characteristics of the movement when the performer actually plays the phrase of the part for which the page change is desired are stored in advance in association with the corresponding page image, and the characteristic of the movement of the performer is displayed. When it is determined that the motion feature associated with the page image of the musical score displayed on the apparatus matches the page feature, the page is changed. The performer only needs to perform the same operation as when the specific phrase is registered when the page of the score is changed, and does not need to perform a special operation. In addition, since the characteristic data of the player's movement is used for determining the page change, it is not affected by the sound of the musical instrument playing at the same time. As a result, it is possible to provide a musical score display device capable of automatically switching pages at a timing desired by a user without performing a special operation during the performance, regardless of the musical instruments and the number of people playing at the same time. .

It is a block diagram showing the structure of the score display apparatus 30 which concerns on the 1st Embodiment of this invention. 1 is an external view of a score display device 30. FIG. It is a flowchart which shows the control structure of the program for implement | achieving the operation | movement of phrase registration mode which CPU32 of the score display apparatus 30 which concerns on 1st Embodiment performs. It is a figure showing the structural example of phrase data. It is a flowchart which shows the control structure of the program for implement | achieving operation | movement of the performance mode which CPU32 of the score display apparatus 30 performs. It is a flowchart which shows the control structure of the program for implement | achieving the operation | movement of phrase data collation processing in the program shown in FIG. It is a setting example of phrase data in the musical score data stored in the musical score display apparatus in the second embodiment. It is a change destination table 250 for setting the number of performances of phrase data and a page to be displayed after the performance. It is a figure showing the frequency | count of performance of phrase data, the page to display, and state transition. It is a flowchart which shows the control structure of the program for implement | achieving the operation | movement of phrase registration mode which CPU32 of the score display apparatus which concerns on 2nd Embodiment performs. It is a flowchart which shows the control structure of the program for implement | achieving operation | movement of the performance mode which CPU32 of the score display apparatus in 2nd Embodiment performs.

  Hereinafter, specific embodiments of the present invention will be described. In the following description and drawings, the same parts are denoted by the same reference numerals and names. Their functions are also the same. Therefore, detailed description thereof will not be repeated.

[First Embodiment]
-Constitution-
Referring to FIG. 1, a score display device 30 executes a predetermined program and controls the overall operation of the score display device 30, a CPU (Central Processing Unit) 32, a bus 50 connected to the CPU 32, Both are ROM (Read-Only Memory) 34 for storing a program to be executed by CPU 32, RAM (Random Access Memory) 36 that operates as a temporary storage unit when executing a program, connected to bus 50, An operation unit 38 for receiving an operation from the player, a display unit 40 for displaying an operation screen for controlling the score data and the score display device, a microphone 42 for detecting a sound played by the performer, and various images. Camera 44 for shooting and capturing as image data, various information Storage unit 48 for storing, and external storage media including external memory interface 46 for taking and musical score data.

  The ROM 34 is an electrically rewritable nonvolatile memory such as an EEPROM (Electrically Erasable and Programmable Read Only Memory) and a flash memory. The storage unit 48 includes an electrically rewritable nonvolatile memory such as a magnetic disk, an EEPROM, and a flash memory.

  In the present embodiment, display unit 40 includes an LCD (Liquid Crystal Display).

  The storage unit 48 stores a musical score data storage unit 52 for storing musical score data and a musical score data stored in association with the musical score data stored in the musical score data storage unit 52 when the player is playing a specific phrase. A phrase data storage unit 54 for storing phrase data including a series of images of the performers.

  The score data stored in the score data storage unit 52 is image data for each page of a score composed of one or a plurality of pages. In addition to the method of capturing the musical score data via the external storage medium by the external memory interface 46, it is also possible to read and capture the image using the camera 44. The score data storage unit 52 can store a plurality of score data, and the score data to be displayed is selected by the player's operation.

  The phrase data stored in the phrase data storage unit 54 is stored in association with each musical score data page, and stores a series of movements of the performer before turning each page, which is taken using the camera 44. It is data. The configuration and setting of phrase data will be described later.

  Referring to FIG. 2, operation unit 38 has a selection button 66 for selecting an item displayed on the screen, a determination button 64, a start button 62 for starting various operations, and for stopping the operation. And an end button 60.

  Referring to FIG. 2, the camera 44 and the microphone 42 are installed near the center of the lower portion of the display unit 40 in order to detect the shooting and performance sound of the performer. The image display device 30 itself is configured by a thin body, and can be used on a general-purpose music stand instead of a paper score.

  On the display unit 40, musical score data selected by executing a program to be described later is displayed page by page.

  The score display device 30 in the present embodiment has two operation modes: a performance mode and a phrase registration mode. The performance mode is a mode in which the musical score data selected by the performer is displayed on the display unit 40, and the displayed page is automatically switched according to the progress of the performance. The phrase data registration mode is a mode for registering a phrase for switching the page on which the selected musical score data is displayed. In order for a performer to use automatic page switching, it is necessary to register page switching phrase data in advance in the phrase registration mode.

  In the score display device 30 according to the present embodiment, one phrase data for switching pages can be registered for each page to be displayed.

  With reference to FIG. 3, a control structure of a program executed when the phrase registration mode is selected in the score display device 30 will be described. This program is executed following the step 202 of displaying the score data names stored in the storage unit 48 as a list on the display unit 40 (hereinafter simply referred to as “S”) 202 and the score data to be displayed. S204 which receives a selection operation from the user, and is executed following S204. The information of the selected musical score data is read out from the musical score data storage unit 52. The musical score data page which is executed following S206 and is selected. S208 for displaying the image data on the display unit 40.

  In S206, image data of the selected score data and page information of the score data are read. In S208, image data is displayed and page information of the image data to be displayed is stored at the same time. When S208 is executed following S206, the first page of the score data is displayed and 1 is stored as display page information.

  This program is further executed following S208, and it is determined whether or not the display page is changed by the user, and there is a change operation in S210 and S210 that branches the control flow according to the determination result ( (YES) is executed, the page information to be displayed is set, the control is advanced to S208, and the process is executed when it is determined in S210 that there is no change operation (NO), and the start button 62 is displayed. S214 that determines whether or not the button is pressed and branches the control flow according to the determination result. If it is determined in S214 that the start button 62 has not been pressed (NO), the control returns to S210.

  In S210, when the selection button 66 is pressed, it is determined that a display page change operation has been performed. In S212, the page information of the selected page is stored, and in S208, the selected page is displayed on the display unit 40.

  This program is further executed when it is determined in S214 that the start button 62 has been pressed (YES), and is executed subsequent to S216 for starting the camera 44 and S216, and the image data taken by the camera 44 is temporarily stored. S218 for erasing the contents of the stored image buffer and S220, which is executed subsequent to S218, determines whether or not the microphone 42 has detected a sound played by the performer, and branches the control flow according to the determination result S220. This is executed when it is determined that the sound to be played in S220 is detected (YES), and is executed subsequent to S222 and S222 for buffering image data captured by the camera 44 in the image buffer. It is determined whether or not the sound played by the person is detected, and the flow of control is branched according to the determination result, S224 and S2 4 is executed when the sound to be played is determined not detected (NO), the including a S226 that the camera 44 is finished buffering image data to be photographed. If it is determined in S220 that the performance sound is not detected (NO), the control returns to S220 and waits for the performance sound to be detected. If it is determined in S224 that a performance sound has been detected (YES), the control returns to S222.

  This program is further executed following S226, and it is determined whether or not an operation of registering a phrase for which performance has been completed as phrase data is received, and the operation is branched according to S228 for branching the control flow according to the determination result. S230 is executed when it is determined to have been received (YES), and phrase data for switching the currently displayed page is created and registered from image data captured using the camera 44, and after execution of S230 or S228. This is executed when it is determined that the operation has not been received (NO), and a screen for selecting whether or not to end phrase data is displayed on the display unit 40, and whether or not the end of phrase data registration has been selected is displayed. The operation of the camera 44 is executed when it is determined that the control flow is branched and the registration end is selected (YES) in S232. S234 to be stopped, executed after S234, a display screen for selecting whether or not to register the phrase data of another page is displayed on the display unit 40, and it is determined whether or not the registration of another page has been selected. S236 for branching the flow of control according to the above, and when it is determined in S236 that registration of another page is not selected (NO), a screen for selecting whether to end the phrase registration mode is displayed. And S238 for branching the control flow depending on whether or not the mode end is selected. If it is determined in S232 that the registration end is not selected (NO), that is, if it is selected to continue the registration of the phrase data, the control returns to S218. If it is determined in S236 that registration of another page has been selected (YES), the control returns to S208. If it is determined in S238 that termination of the phrase registration mode has not been selected (NO), the control returns to S202. If it is determined in S238 that the phrase registration mode end has been selected (YES), this program is ended.

  Regarding the registration of the phrase data in S230, the phrase data in the present embodiment is extracted from the buffered performance image data as an image having a change with respect to the immediately preceding image as a feature image. The time interval between feature images is stored. In the program of FIG. 3, the registration start is selected in S214, and the performance image data shot and buffered by the camera 44 between the time when the performance sound is detected and the time when the performance sound is not detected is a feature. An image is extracted. Then, the extracted feature images and the time interval information between the feature images are collected and registered as phrase data for the score display page.

  In this musical score display device, the phrase registration mode and the performance mode can be used in different places (for example, a practice room and a stage). Therefore, in the feature image, it is necessary to extract an image of only the performer excluding the background from the photographed image. As a method for extracting the performer's image, for example, there are the following methods. First, a captured performance image is divided into blocks arranged in a matrix, and each block is classified into a block with a large movement between frames and a small block. Since blocks with small movement between frames can be considered as background parts, a group of blocks with large movement is extracted as the player's image. A feature image is determined by comparing the extracted image with the previous and subsequent images. Since there may be a difference in the clothes of the performer, collation of phrase data to be described later is performed without considering color information.

  In S230, it is possible to set a standby time after the phrase performance. As an example of setting the waiting time, for example, when there is no phrase suitable for page change for the performer at the last part of the page to be displayed, a phrase a little before is registered as phrase data, and the waiting time is set at the same time. After the phrase data is played, the page display is changed after the set standby time has elapsed. As a result, it is possible to perform a page turning operation that matches the progress of the performance. The setting of the waiting time is arbitrary. If the standby time is not set, a default value (for example, 0 seconds) is applied. Information on the waiting time is included in the phrase data.

  With reference to FIG. 4, the phrase data of the 1st page and the 2nd page of the score data composed of 3 pages will be described.

Let n be a variable representing the page number displayed on the display unit 40. The phrase data of the nth page is the next from the number _{Mn of} feature images included in the phrase data, the feature image data _Pn [m] of the mth image (m = 1 to _Mn ) and the feature image of the mth image. The time interval T _n [m] until the (m + 1th) feature image appears can be expressed as shown in FIG.

FIG. 4A is a diagram illustrating a configuration example of phrase data of the first page (n = 1) of the score data. In this example, the phrase data on the first page includes three feature images (M ₁ = 3). FIG. 4B is a diagram illustrating a configuration example of phrase data of the second page (n = 2) of the score data. The phrase data on the second page includes four feature images (M ₂ = 4).

The feature image in the phrase data is extracted from the relationship between successive image data in a series of buffered image data. Therefore, the number of feature images M _n , the feature image P _n [m], and the time interval T _n [m] (m = 1 to M _n ) between feature images vary from page to page.

Referring to FIG. 4A, the first feature image P ₁ [1] is recorded, and after the time interval T ₁ [1] has elapsed, the second feature image P ₁ [2] is detected and stored. After recording the feature image P ₁ [2], P ₁ [3], which becomes the last feature image after the elapse of the time interval T ₁ [2], is detected and stored. T ₁ [3] represents a waiting time until the page change operation is performed after the phrase data is played. This waiting time is set by the user in S230 of the program shown in FIG. If the waiting time is not set by the user, a default value is set.

FIG. 4B is similar to FIG. However, in FIG. 4B, since the number of feature images M ₂ = 4, T ₂ [4] represents the standby time.

In FIG. 4, the relationship between the feature image and the time interval is shown in time series for explanation. The actual phrase data includes the feature image number M _n , the value of the time interval T _n [m] (m = 1 to M _n ), and the feature image P _n [m] (m = 1 to _n ) for the variable n representing the page. M _n ) image data is stored.

  Referring to FIG. 5, in the score display device 30, this program executed when the performance mode is selected displays a list of score data names stored in the storage unit 48 on the display unit 40, and S 402. It is executed subsequent to S402, it is determined whether or not a selection operation for the musical score data to be displayed has been received, and it is determined that the selection operation has been received (YES) in S404 for branching the control flow according to the determination result. S406 is executed, and the information of the selected musical score data is read from the musical score data storage unit 52, and the phrase data relating to the selected musical score data is read from the phrase data storage unit 54, respectively. S408 for displaying the page image data of the musical score data displayed on the display unit 40.

  In S406, the image data of the selected score data and the page information of the score data are read as the score data information. In S408, image data is displayed and page information of the image data to be displayed is stored. When S408 is executed subsequent to S406, the first page of the score data is displayed, and 1 is stored as display page information.

  This program is further executed following S408, and it is determined whether or not the display page is changed by the user, and there is a change operation in S410 and S410 that branches the control flow according to the determination result ( (YES) is executed, the page information to be displayed is set, and the control is advanced to S408. The process is executed when it is determined that there is no change operation in S410 (NO), and the start button 62 is displayed. S414 that branches the flow of control depending on whether or not the button is pressed. If the start button 62 is not pressed in S414 (NO), the control returns to S410.

  In S410, when the selection button 66 is pressed, it is determined that a display page change operation has been performed. In S412, the page information of the selected page is stored, and in S408, the selected page is displayed on the display unit 40.

  This program is further executed when it is determined in S414 that the start button 62 has been pressed (YES), and is executed subsequent to S416 for starting the camera 44 and S416, and the image data taken by the camera 44 is temporarily stored. S418 for erasing the contents of the stored image buffer, executed after S418, it is determined whether or not the end button 60 has been pressed, S420 for branching the control flow according to the determination result, and the end button in S420 S422 is executed when it is determined that 60 is not pressed (NO), and the performance image data captured by the camera 44 is buffered, and the performance image buffered in S422 is executed following S422. S424 for performing a collation process between the data and the registered phrase data, and subsequent to S424, S42 is executed. It is determined whether or not the performance image data matches the registered phrase data in the phrase data matching process in S, and S 426 branches the flow of control according to the determination result, and it is determined that they match in S 426 (YES). The page image data of the musical score displayed on the display unit 40, which is executed following S428 and S428, which is executed when the display screen is kept waiting for the time set for the corresponding phrase data, is displayed on the next page. And S430 that advances the control to S418. If it is determined in S426 that they do not match (NO), the control returns to S420.

  When the phrase data collation process of S424 is executed, one of the collation results in which the performance image data and the phrase data match or does not match is output. Details of the phrase data matching process will be described later.

  This program is further executed when it is determined in S420 that the end button 60 has been pressed (YES), and is executed following S432 for stopping the operation of the camera 44 and S432, and whether or not to end the performance mode. And S434 for branching the flow of control depending on whether or not the end of the performance mode is selected. If it is determined in S434 that termination is not selected (NO), the control proceeds to S402. If it is determined in S434 that termination has been selected (YES), the program ends.

  In the phrase data collation process, the feature image series registered in the phrase data of each page is collated with the performance image data buffered when the performance mode is executed. The performance image data is collated, and when similar image data appears in the same order at a time interval considering a predetermined error range ΔT in the time interval between feature images, it is determined that the phrase data match. Specifically, the latest buffered performance image data is compared with the final feature image of the phrase data related to the displayed page to determine whether they are similar. If it is determined that there is a similar feature image, it is determined whether there is an image similar to the previous feature image in the phrase data in the buffered performance image data. When there is similar performance image data, the difference between the time when the performance image data is buffered and the time when the latest performance image data is buffered is measured. If the measured time difference is within the time interval considering the predetermined error range ΔT in the time interval between the feature images, it is determined that a part of the phrase data is being played. This collation determination process is repeated upside down the buffered image, and when the first characteristic image of the phrase data is matched, it is determined that the phrase played by the performer matches the phrase data registered in advance.

Referring to FIG. 6, the program executed in S424 in FIG. 5 is executed subsequent to S502 for acquiring page information displayed on display unit 40 and S502, and is buffered in S422 in FIG. The latest performance image data is compared with the final feature image P _n [M _n ] of the phrase data associated with the page of the page information acquired in S502. It is determined whether or not the image data and the final feature image P _n [M _n ] are similar, and S506 for branching the control flow according to the determination result and S506 determines that they are similar (YES). Is executed, and includes S508 for setting a value of M _n −1 to the variable m representing the order of the feature images in the phrase data.

This program is executed subsequent to S508, and in the buffered performance image data, the performance image data immediately before the performance image data referred to in the determination process of whether or not the performance images are similar is acquired. S512 is executed subsequent to S510, and the performance image data acquired in S510 is compared with the feature image P _n [m]. S512 is executed subsequent to S512, and the performance image data acquired and the feature image P _n [ m] is determined to be similar, and is executed when it is determined in S514 that branches the control flow according to the determination result and in S514 that it is not similar (NO). The time when the performance image data was buffered when determined to be similar in the similarity determination processing, and the performance image data acquired in S510 are buffered. It is determined whether or not the time interval with the determined time exceeds the value of T _n [m] + ΔT, and S524 for branching the control flow according to the determination result and S514 are similar (YES) When the performance image data determined to be similar in the similarity determination process in S514 is buffered and determined to be similar to the one before the similarity determination process S516 for determining whether or not the time interval with the time when the performance image data is buffered is smaller than T _n [m] −ΔT, and branching the control flow according to the determination result.

There are various methods for determining whether or not the performance image data and the characteristic image P _n [m] (m = 1 to M _n ) are similar in S506 and S514. For example, a player's silhouette is created from the image data, and a binary image is created with the silhouette value 1 and the background value 0. The outline of the silhouette extracted from this binary image is called feature data. By comparing the feature data of the performance image data captured by the camera with the feature data created from the feature image data stored as phrase data, it is possible to determine whether the two images are similar.

  Examples of a method for creating a silhouette from image data include the following. For example, assume that the background density is brighter than the density of each part of the person, such as when a person walks in front of a white wall. First, a histogram of the density of performance image data to be photographed is created. Then, a bimodal distribution of the density distribution of the person and the density distribution of the background is obtained. Based on the distribution, a density value a that is a boundary between the person and the background is set, a point where the density value of the pixel of the original image of the performance image data is darker than a silhouette (value 1), and a bright point as the background (value 0). And In this way, a value is assigned to the pixel density value of the image to create a silhouette image.

  In addition, a technique in a general character recognition device can be used for a part for extracting and collating feature data from a binary image. Specifically, a technique is used in which a binary image is created from an input image, character image feature extraction is performed, matching is performed with feature data registered in a dictionary, and similarity is calculated. Similarity determination of images can be similarly performed if the silhouette portion is considered as a character.

  Here, a determination method using a silhouette is given as a determination method for similarity, but when this method is used, the feature data is not stored as the phrase data, but the feature data is stored. Just keep it.

  However, since there are various methods for determining whether or not they are similar, the method is not limited to the method exemplified here.

This program is further executed when it is determined in S524 that the time interval exceeds the value of T _n [m] + ΔT (YES), and it is determined that the phrase data does not match the currently played phrase, and control is performed. The process proceeds to S426 in FIG. 5 and is executed when it is determined in S516 that the time interval is not a value smaller than T _n [m] −ΔT (NO), and the value m−1 is set to the variable m S518. Then, it is executed subsequent to S518, and it is determined whether or not the variable m satisfies m = 0, and the control flow is branched according to the determination result, and S520 is determined to satisfy m = 0 (YES). Is executed, it is determined that the phrases are matched, and the control is advanced to S426 in FIG. 5. If it is determined in S506 that they are not similar (NO), the control proceeds to S526. If it is determined in S516 that the time interval is smaller than T _n [m] −ΔT (YES), the control proceeds to S526. If it is determined in S524 that the time interval does not exceed the value of T _n [m] + ΔT (NO), or if it is determined in S520 that m = 0 is not satisfied (NO), the control proceeds to S510.

―Operation―
The musical score display apparatus 30 according to the first embodiment operates as follows.

<Phrase registration mode>
Before registering the phrase data, the image data of the score is registered in the score display device 30. As the image data of the score, the image data stored in the external storage medium is stored in the storage unit 48 via the external memory interface 46, or the score is read using the camera 44, and the read image data is stored in the storage unit 48. Remember me. The stored image data group is registered with the title of the score.

  When the phrase registration mode is selected, the score display device 30 displays a list of titles of score data stored in the storage unit 48 (S202). When a score title to register phrase data is received from this list (S204), the score data is read and the first page of the selected score data is displayed on the display unit 40 (S206, S208). In the phrase registration mode, the phrase data for turning the page displayed on the display unit 40 is registered. Therefore, when a display page change operation is received from the user (YES in S210), the score display device 30 changes the page information of the page to be displayed (S212) and changes the page to be displayed.

  When the start button 62 is pressed while the page for registering phrase data is displayed on the display unit 40 (YES in S214), the score display device 30 activates the camera 44 (S216) and clears the image buffer (S216). S218). Thereafter, the user performs a performance of the portion registered as phrase data, and starts buffering of the image data in response to the microphone 42 detecting the performance sound (YES in S220, S222). When the user stops the performance and no performance sound is detected, the buffering is terminated (NO in S224, S226).

  After receiving the operation of registering the played phrase as phrase data after playing the phrase (YES in S228), the phrase data relating to the currently displayed page is created from the image data captured using the camera 44. (S230). The phrase data created at this time includes a feature image group extracted from the buffered performance image data, time interval information between feature images, and a waiting time set by the user.

  When the end of registration of phrase data for the page to be displayed is selected (YES in S232), the camera 44 stops (S234), and it is selected whether to register the phrase data of another page of the score data (S236). When registering the phrase data of another page, the page for registering the phrase data is displayed, and the start button 62 is pressed to register the phrase data. When the registration of the phrase data necessary for turning the pages is completed, the user selects the end of the mode (YES in S238) and ends the phrase registration mode.

<Performance mode>
When the performance mode is selected, the score display device 30 displays a list of titles of score data stored in the storage unit 48 (S402). When the score data is selected from this list (YES in S404), the selected score data and the phrase data registered in association with the score data are read from the storage unit 48, and the first page of the selected score data is read. Is displayed on the display unit 40 (S406, S408). When a page change operation is received from the user (YES in S410), the page information of the page to be displayed is changed (S412), and the page displayed on the display unit 40 is changed.

  When the start button 62 is pressed (YES in S414), the score displaying apparatus 30 activates the camera 44 (S416) and clears the image buffer (S418). Thereafter, the musical score display device 30 buffers performance image data photographed by the camera 44 until the end button 60 is pressed (S422), and is registered in the page of the musical score to be displayed with the buffered performance image data. The collation process with the phrase data is performed (S424), and it is repeatedly determined whether the phrases match (S426).

In the phrase data collation process, the displayed page information is acquired (S502), the latest performance image data to be buffered, and the final feature image P _n [M _n ] of the phrase data of the displayed page Are compared to determine whether both images are similar (S504, S506). If it is determined that they are similar (YES in S506), M _n −1 is set to the variable m representing the order of the feature images in the phrase data (S508), and is referred to in the determination process of whether they are similar from the buffer. The previous performance image data of the performed performance image data is acquired (S510), and the acquired performance image data is compared with the feature image P _n [m] (S512). As a result of comparing the images, if it is determined that the two images are not similar (NO in S514), the time when the performance image data determined to be similar last time was buffered, and the acquired image is buffered. It is determined whether or not the time interval with the determined time exceeds T _n [m] + ΔT (S524). If it is determined that the time interval T _n [m] + ΔT has not been exceeded (NO in S524), the performance image data immediately before the referenced performance image data is acquired from the buffer, and the characteristic image P is again acquired. Comparison with _n [m] is performed. If it is determined that the time interval T _n [m] + ΔT has been exceeded, it is determined that the phrases do not match (S526), and the control proceeds to S424.

As a result of comparing the images, if it is determined that the two images are similar (YES in S514), the time when the performance image data determined to be similar to the previous time was buffered, and the acquired image is buffered. It is determined whether or not the time interval from the set time is smaller than T _n [m] −ΔT (S516). If it is determined that the time interval is smaller than T _n [m] −ΔT (YES in S516), it is determined that the phrases do not match, and the control proceeds to S424. When it is determined that the time interval is not smaller than T _n [m] −ΔT (NO in S516), a value of m−1 is set to the variable m (S518), and whether or not the variable m satisfies m = 0. Is determined (S520). When it is determined that m = 0 is not satisfied, the characteristic image P _n [m] is compared with the buffered performance image data, and the above processing is repeated to determine whether the two images are similar. . When it is determined that m = 0 is satisfied, that is, when the performance image data appears in accordance with the order of the feature images included in the registered phrase data and the time interval, it is determined that the phrases match. (S522), control proceeds to S424.

  If it is determined in the phrase matching process that the phrases do not match, the performance image data is buffered until the end button 60 is pressed, and the phrase data matching process is repeated. If it is determined in the phrase matching process that the phrases match, the display page is left as it is for the time set in the phrase data (S428), and after the set time, the next page of the displayed page is displayed (S430). ). After the display page is changed, the image buffer is once cleared, and the phrase data matching process for the displayed page is repeated until the end button 60 is pressed.

  As described above, in the score display device 30, in the phrase registration mode, the performance image is buffered while the performance sound of the performer is detected, the characteristic image is extracted from the performance image, and the characteristic image is extracted. And the time interval between feature images are stored as phrase data for turning pages. On the other hand, in the performance mode when the performance is actually performed, the player's movement (performance image data) during the performance is continuously taken by the camera, and it is determined whether the image matches the registered phrase data. If it is determined that they match, a page change operation is performed. Even for a newly displayed page, it is determined whether or not the player's movement matches with the phrase data registered on the page.

  Thus, in the phrase registration mode, the feature image is extracted from the performance image during sound detection, so that an image when the performer is actually playing a page turning phrase can be obtained accurately. On the other hand, when it is determined whether the phrase data matches with the registered phrase data in the performance mode, only the buffered image is referred to, and it is determined based on whether it appears in the same order as the registered phrase data image. Since the page turning operation is determined based on the performer's image, it is not affected by the sound of the musical instrument playing at the same time. As a result, it is possible to provide a musical score display device capable of automatically switching pages at a timing desired by a user without performing a special operation during the performance, regardless of the musical instruments and the number of people playing at the same time. .

  The display unit 40 is not limited to the LCD. Any thin display device such as an organic EL (Electro-Luminescence) may be used. Further, a touch panel may be provided so as to be superimposed on the display unit 40 and used as the operation unit 38.

[Second Embodiment]
In the score display apparatus according to the second embodiment, a plurality of phrase data can be registered for a page to be displayed. The page displayed after the phrase performance can be arbitrarily set according to the number of performances of each phrase data. The score display apparatus according to the second embodiment has the same hardware configuration as the score display apparatus 30 of the first embodiment, but the program executed by the CPU 32 is different.

-Constitution-
Hereinafter, the configuration of the score display apparatus according to the present embodiment will be described using a specific score as an example.

  Referring to FIG. 7, the musical score data is composed of three pages, phrase data 11 (denoted as “B”) on the first page, phrase data 21 (denoted as “D”) and phrase on the second page. Data 22 (denoted as “E”) includes phrase data 31 (denoted as “G”) on the third page. In order to explain the progress of the score, the part from the beginning of the first page of the score data is A, the part from the beginning of the second page is C, and the part from the beginning of the third page is F.

  In the musical score data example shown in FIG. 7, the phrase data 21 on the second page is a repeat part. When the phrase data 21 appears for the first time, the phrase data 21 returns to the beginning (A) of the score, and when it appears for the second time, the phrase data 21 proceeds directly without returning. The phrase data 11 displays the second page up to the third performance. When the phrase data 22 appears for the first time, the third page is displayed, and the page is not changed after the second time. When the phrase data 31 appears for the first time, the phrase data 31 returns to the beginning (A) of the score.

  The order of progression in accordance with this musical score data example is “A → B → C → D → A → B → C → D → E → F → G → A → B → C → D → E”. The display order of the pages at this time is “1 → 2 → 1 → 2 → 3 → 1 → 2”.

  In the present embodiment, a change destination table 250 as shown in FIG. 8 is used. Referring to FIG. 8, change destination table 250 records a change source page, phrase data, and a display page for each number of performances of each phrase data. In the score display device of the second embodiment, the number of performances of phrase data and the page to be displayed at that time are preset in the change destination table 250. If the number of performances is not set in the change destination table 250, the page to be displayed is not changed even if the phrase is played. In the setting example of FIG. 8, the phrase data 21 on the second page displays the next page when the number of performances is the first. When the phrase data 21 is played for the second time, the page is not changed because the change destination table 250 is not set.

  The same phrase data is not registered on the same page. When the same phrase data is used for the page, for example, in FIG. 8, when the same data is used for the phrase data 21 and the phrase data 22, it is considered as one phrase data, and the page change destination is set by the number of performances. It is because it can cope. If the phrase data to be registered is on another page, there is no problem even if the same phrase data is used.

  Referring to FIG. 9, P is a page, F is phrase data, and numbers below “/” indicate the number of performances of the phrase. Phrase data 11 included in the first page indicates that the second page is displayed after the phrase is played from 1 to 3 times. Phrase data 21 included in the second page indicates that the first page is displayed after the first phrase performance. The phrase data 22 included in the second page indicates that the third page is displayed after the first phrase performance. Phrase data 31 included in the third page indicates that the first page is displayed after the first phrase performance.

  The score display apparatus in the second embodiment has two operation modes, a performance mode and a phrase registration mode, in the same way as the score display apparatus 30 in the first embodiment. However, in the phrase registration mode of the score display apparatus according to the second embodiment, different phrase data can be registered for the displayed page. When the phrase data is registered in the musical score display apparatus according to the second embodiment, the page to be displayed next is set according to the number of performances of the phrase.

  With reference to FIG. 10, in the score display apparatus according to the second embodiment, a control structure of a program executed when the phrase registration mode is selected and score data to be registered is selected from the user will be described. . This program includes S602 for reading the information of the selected score data from the score data storage unit 52, and S604 that is executed following S602 and displays the page image data of the selected score data on the display unit 40. .

  In S602, the image data of the selected score data and the page information of the score data are read. In S604, image data is displayed and page information of the image data to be displayed is stored. When S604 is executed subsequent to S602, the first page of the score data is displayed and 1 is stored as display page information.

  This program is further executed following S604, and it is determined whether or not the display page is changed by the user, and there is a change operation in S606 and S606 that branches the control flow according to the determination result ( (YES) is executed, the page information to be displayed is set, the control is advanced to S604, and when it is determined that there is no change operation in S606 (NO), the start button 62 is displayed. S610 which determines whether it was pushed and branches the flow of control according to a determination result is included. If it is determined in S610 that the start button 62 has not been pressed (NO), the control returns to S606.

  In S606, when the selection button 66 is pressed, it is determined that a display page change operation has been performed. In S608, the page information of the selected page is stored, and in S604, the selected page is displayed on the display unit 40.

  This program is further executed when it is determined in S610 that the start button 62 has been pressed (YES), S612 for starting the camera 44, and a phrase that is executed following S612 and registered in the displayed page. S614 that sets the variable a representing the number of data to 0, S614, which is executed subsequent to S614, and which erases the contents of the image buffer, is executed following S616, and the microphone 42 detects the sound played by the performer. S622 for branching the flow of control according to the determination result, and when it is determined that the sound played in S622 is detected (YES), the image data captured by the camera 44 is acquired. S620 to be buffered and executed following S620 to determine whether or not the microphone 42 has detected a sound played by the performer, S622 for branching the control flow according to the fixed result, and S624, which is executed when it is determined that the performance sound is not detected (NO) in S622 and ends the buffering of the image data captured by the camera 44. Including. If it is determined in S618 that the performance sound is not detected (NO), the control returns to S618 and waits for the performance sound to be detected. If it is determined in S622 that a performance sound is detected (YES), the control returns to S620.

  This program is further executed following S624, and it is determined whether or not an operation of registering a played phrase as phrase data has been received, and the operation has been received in S626 for branching the control flow according to the determination result. (YES) is executed, S628 is executed to set a + 1 to the variable a, and S628 is executed, and phrase data is created from the image data captured using the camera 44 and is currently displayed. S630 to be registered as the a-th phrase data of the current page and S630, which is executed subsequent to S630 and receives phrase data information settings from the performer for the phrase data registered in S630, or after the execution of S632, or This is executed when it is determined in S626 that no operation has been received (NO), and another phrase registration operation from the performer. S636 that determines whether or not the operation has been received and branches the control flow according to the determination result, and S638 that is executed when it is determined that the operation has not been received (NO) in S636 and stops the operation of the camera 44. And S640, which is executed subsequent to S638, determines whether or not an operation for registering phrase data of another page has been received from the performer, and branches the flow of control according to the determination result. If it is determined in S636 that an operation has been received (YES), the control proceeds to S616. If it is determined in S640 that a phrase data registration operation for another page has been received (YES), the control proceeds to S604. If it is determined in S640 that the operation for registering the phrase data of another page has not been received (NO), this program is terminated.

  Regarding the phrase data registration in S630, the phrase data used in the second embodiment has the same configuration as the phrase data in the first embodiment. In the program of FIG. 10, from the performance image data captured and buffered by the camera 44 from the time when the start button 62 is pressed in S612 until the performance sound is detected until the performance sound is not detected. A feature image is extracted. Then, the extracted feature image group, the time interval information between the feature images, and the number of feature images are registered as the a-th phrase data for the score display page.

  The phrase data information set in S632 includes the standby time after the performance and the page information of the musical score to be displayed corresponding to the number of performances of the phrase data to be registered. If the standby time is not set, a default value (for example, 0 seconds) is applied.

  With reference to FIG. 11, the control structure of the program executed when the performance mode is selected in the score display apparatus according to the second embodiment will be described. This program is executed following S702 for displaying the score data names stored in the storage unit 48 as a list on the display unit 40 and S702, and it is determined whether or not a score data selection operation has been received from the displayed list. Then, the flow of control is branched according to the determination result, and when it is determined that the selection operation is received (YES) in S704, the information of the selected score data is obtained from the score data storage unit 52. S706 for reading out the phrase data relating to the selected musical score data from the phrase data storage unit 54, and S708, which is executed subsequent to S706 and displays the page image data of the selected musical score data on the display unit 40.

  In S706, the image data of the selected score data and the page information of the score data are read out as the score data information. In S708, image data is displayed and page information of the image data to be displayed is stored at the same time. When S708 is executed following S706, the first page of the score data is displayed, and 1 is stored as display page information.

  This program is further executed following S708, and it is determined whether or not a display page changing operation by the user has been received, and the control flow is branched according to the determination result, and the changing operation is received in S710. This is executed when it is determined as (YES), sets page information to be displayed, and advances to S708, and is executed when it is determined that no change operation is received at S710 (NO), and a start button S714 for branching the flow of control depending on whether or not 62 is pressed. If it is determined in S714 that the start button 62 has not been pressed (NO), the control returns to S710.

  In S710, when the selection button 66 is pressed, it is determined that a display page change operation has been received. In S712, the page information of the selected page is stored, and in S708, the selected page is displayed on the display unit 40.

  This program is further executed when it is determined in S714 that the start button 62 has been pressed (YES), and is executed subsequent to S716 for starting the camera 44, S716, and the phrase data registered in the score data. Subsequent initialization is performed by substituting 0 for each count value, S718, which is executed subsequent to S718, which is executed after S720 for deleting the contents of the image buffer, and whether or not the end button 60 has been pressed. Determination and branching of the flow of control according to the determination result, and buffering of performance image data that is executed when it is determined in S722 that the end button 60 has not been pressed (NO) and photographed by the camera 44 Is executed after S724 and S724, and the phrase data is recorded using the performance image data buffered in S724. S726 that performs the data matching process, and is executed subsequent to S726. In the phrase data matching process in S726, it is determined whether or not there is a matching phrase between the performance image data and the registered phrase data, and according to the determination result. S728 for branching the control flow, executed when it is determined that there is matching phrase data in S728 (YES), and S730 is added to the count value of the phrase data determined to match, and S730 is followed Executed, referring to the change destination table 250, branches the control flow depending on whether or not the page change destination corresponding to the count number is registered for the corresponding phrase data, and the page change destination is registered in S732 Is executed (YES) and is set in the corresponding phrase data. And S734 to wait as it is only a display screen time, is executed following S734, it displays the image data of a page registered to the destination table 250, and a S736 advancing control to S720. If it is determined in S728 that there is no matching phrase (NO), the control returns to S722. If it is determined in S732 that the page change destination corresponding to the count number is not registered (NO), the control returns to S722.

  In S726, the program shown in the flowchart shown in FIG. 6 is executed to perform phrase data collation processing. Referring to FIG. 6, in the second embodiment, after executing S522 or S526 of the program shown in FIG. 6, control returns to S728 in FIG. Also, the phrase data matching process in S726 performs a matching process on all phrase data registered in the page displayed on the display unit 40, and detects the presence or absence of matching phrase data and the matching phrase data name. . As described above, the same phrase data does not exist in one page. For this reason, in S726, collation processing using the program shown in FIG. 6 is simultaneously performed on all phrase data registered on the page to collate phrase data and detect matching phrase data names.

  This program is further executed when it is determined in S722 that the end button 60 has been pressed (YES), and is executed subsequent to S738 for stopping the operation of the camera 44 and S738, and whether or not to end the performance mode. And S740 for determining whether or not the end of the performance mode has been selected by the user and branching the control flow according to the determination result. If it is determined in S740 that termination is not selected (NO), the control proceeds to S702. If it is determined in S740 that the end is selected (YES), the program ends.

―Operation―
The musical score display apparatus according to the second embodiment operates as follows.

<Phrase registration mode>
Before registering the phrase data, register the musical score image data. The registration of musical score image data is the same as that of the musical score display apparatus 30 of the first embodiment, and therefore, description thereof will not be repeated.

  When the phrase registration mode is selected and the selection of the score title to register the phrase data is received, the score display apparatus in the second embodiment reads the score data, and the first page of the score data selected on the display unit 40 Is displayed (S602, S604). If a display page change operation is received from the user (YES in S606), the page information of the page to be displayed is changed (S608), and the page displayed on the display unit 40 is changed.

  When registering phrase data, the start button 62 is pressed when a desired page is displayed on the display unit 40 (YES in S610). Then, the camera 44 is activated (S612), a value of 0 is set to the variable a indicating the number of phrase data registered in the page to be displayed (S614), and the image buffer is cleared (S616).

  Thereafter, the user performs the performance of the portion registered as phrase data. While the performance sound continues to be detected by the microphone 42, the image data is buffered (YES in S618, S620). When the user stops the performance and no performance sound is detected, the buffering is terminated (NO in S622, S624).

  After playing the phrase, if the user receives an operation of registering the played phrase as phrase data (YES in S626), 1 is added to the variable a (S628), and the a-th phrase data of the page to be displayed is Phrase data is created and registered from the image data that played the phrase (S630). At this time, the created phrase data includes a feature image group extracted from the buffered performance image data and time interval information between the feature images. The musical score display apparatus according to the second embodiment further receives setting information regarding phrase data to be registered from the user (S632). The setting information related to the phrase data includes the standby time after the performance and the page information of the musical score to be displayed next corresponding to the number of performances of the phrase data to be registered.

  After the phrase data is registered, or when the user selects not to register the data in order to retrieve the phrase data that has been played (NO in S626), whether or not to register another phrase data is selected. When registering another phrase data (YES in S636), the image buffer is cleared and the above operation is repeated.

  If it is selected not to register another phrase data (NO in S636), the camera 44 is stopped (S638), and whether to register another page of phrase data is selected (S640). When registering the phrase data of another page, the page for registering the phrase data is displayed, and the start button 62 is pressed to register the phrase data. When the registration of the phrase data is completed and no other page is registered (NO in S640), the phrase registration mode is terminated.

<Performance mode>
When the performance mode is selected, the score display apparatus according to the second embodiment displays a list of titles of score data stored in the storage unit 48 (S702). When the score data is selected from this list (YES in S704), the selected score data and the phrase data registered in association with the score data are read from the storage unit 48, and the first page of the selected score data is read. Is displayed on the display unit 40 (S706, S708). When a page change operation is received from the user (YES in S710), the page information of the page to be displayed is changed (S712), and the page displayed on the display unit 40 is changed.

  When the start button 62 is pressed (YES in S714), the score displaying apparatus in the second embodiment activates the camera 44 (S716). The score display apparatus of the second embodiment performs initialization by substituting 0 for each performance count value of the phrase data registered in the score data (S718), and clears the image buffer (S720). Thereafter, the musical score display apparatus according to the second embodiment buffers the performance image data captured by the camera 44 until the end button 60 is pressed (S724), and displays the buffered performance image data and the page to be displayed. A collation process is performed with all phrase data registered in association with (S726), and it is determined whether or not there is a matching phrase (S728).

  As a result of the phrase matching process, if it is determined that there is no matching phrase, the performance image data is continuously buffered until the end button is pressed, and the phrase matching process is repeated. When it is determined by the phrase matching process that there is a matching phrase, 1 is added to the performance count value of the phrase data determined to match (S730), the phrase data determined to match by referring to the change destination table 250 It is determined whether or not a page change destination is registered for the number of performances (S732). If the change destination is not registered in the change destination table 250, the display page is not changed, the performance image data is buffered, and the phrase data matching process is repeated.

  When the change destination is registered in the change destination table 250, it waits for the time set in the phrase data from when it is determined that the phrases match (S734), and the registered page is displayed (S736). . After changing the display to the registered page, clear the image buffer. Thereafter, the phrase collation process is repeated with the phrase data registered in the displayed page and the performance image data newly buffered.

  In the performance mode, the performance image data is buffered until the end button 60 is pressed, and the phrase data matching process is repeated. When the end button 60 is pressed, the operation of the camera 44 is stopped (S738). Thereafter, when the end of the performance mode is selected (S738), the performance mode is ended.

  As described above, the musical score display apparatus according to the second embodiment registers one or a plurality of phrase data in the phrase data for one page in the phrase registration mode. For each phrase data, a page to be displayed next is set for each number of times the phrase data is played. In the performance mode, the number of performances of the phrase data is counted, and the page to be displayed is changed according to the display settings for each number of performances set for the phrase data.

  In this way, by registering one or more phrase data for one page and changing the page to be displayed next according to the number of performances, even for a musical score with a repeat (repeated) symbol, Pages can be displayed in the order of performance. As a result, it is possible to realize a musical score display device that can automatically switch pages at a timing desired by the user and can display musical scores in the order of performance.

  Also, in the musical scores shown in FIGS. 7 to 9, when the musical scores are displayed in the order of performance, the page transition is “1 → 2 → 1 → 2 → 3 → 1 → 2”. In the case of such a musical score example, it can be considered as a musical score having a 7-page structure on the assumption that a plurality of the same pages exist. When considered as a 7-page musical score, it is necessary to set phrase data for each page. When displaying musical scores in the order shown in FIGS. 7 to 9, it is necessary to register six phrase data. When the page to be displayed is changed according to the number of times the phrase data is played as in the second embodiment, in the above example, the page display is performed with four phrase data and registration of the change destination for each phrase data. Can be realized. Since the phrase data is composed of a feature image group and time interval information, the amount of information used per phrase data is large. By changing the display image according to the number of performances as in the second embodiment, it is possible to deal with a complicated display order with a small number of registered phrase data. As a result, it is possible to realize a musical score display apparatus that can set the display order of musical scores with a smaller amount of information.

  The embodiment disclosed herein is merely an example, and the present invention is not limited to the above-described embodiment. The scope of the present invention is indicated by each of the claims after taking into account the description of the detailed description of the invention, and all modifications within the meaning and scope equivalent to the wording described therein are intended. Including.

30 Musical score display device 32 CPU
40 Display unit 42 Microphone 44 Camera 48 Storage unit 52 Musical score data storage unit 54 Phrase data storage unit 60 End button 62 Start button 66 Selection button 250 Change destination table

Claims (10)

A display device;
Storage means for storing a series of page images of a score together with a score name and an order of the series of page images of the score;
Image capturing means for outputting a series of image data obtained by imaging a subject in real time;
Phrase data storage means for storing, as a phrase data in association with a page name corresponding to the score name, a name of the score when the performer performs a certain phrase,
A score display means for reading a series of page images of a score corresponding to a designated score name from the storage means, and displaying a predetermined page image on the display device among the read page images;
When the musical score page image is displayed on the display device, the player's movement feature extracted from the output of the image capturing means is associated with the musical score page image displayed on the display device. Phrase determination means for outputting a determination signal that takes a first value when the phrase data matches the other phrase data, and a second value otherwise.
In response to the determination signal becoming the first value, page change means for displaying a page image in the next order of the page images displayed on the display device on the display device. , Sheet music display device. An operation mode designating unit for designating one of the first and second operation modes;
The phrase data storage means is designated with the player's movement characteristics extracted from the output of the image capture means when the first operation mode is designated by the action mode designation means. Means for storing the musical score name and the phrase data in association with a designated page image of the page images corresponding to the musical score name;
The phrase determining means is extracted from the output of the image capturing means when the second operation mode is specified by the operation mode specifying means and a score page image is displayed on the display device. Determining means for determining whether or not the characteristic of the player's movement matches the phrase data associated with the page image of the score displayed on the display device and outputting the determination signal; The musical score display device according to claim 1, further comprising: A voice detecting means for detecting voice;
The image capturing means includes means for starting imaging of a subject and outputting a series of image data of the subject in real time in response to the sound detecting means detecting sound. The musical score display device according to claim 1 or 2. It further includes a time measuring means for outputting a time expiration signal when a predetermined time has elapsed from the start of the time measurement,
The page changing means includes
Means for causing the timing means to start timing in response to the determination signal becoming the first value;
4. The musical score display according to claim 1, further comprising means for displaying the next sequential page image on the display device in response to the time expiration signal from the time measuring means. apparatus. The phrase data storage means displays the characteristics of the performer's movement when the performer plays a phrase, the score name, a page image corresponding to the score name, and, if any, the page image. Means for storing as phrase data in association with next page specifying information for specifying a page image to be included;
The phrase determination means is
When the musical score page image is displayed on the display device, the player's movement feature extracted from the output of the image capturing means is associated with the musical score page image displayed on the display device. Determination means for determining whether the phrase data matches any of the phrase data and outputting the determination signal;
Means for outputting information identifying phrase data determined by the determining means to match the characteristics of the player's movement,
The page changing means is specified by the next page image specifying information for the phrase data specified by the information output by the means for outputting in response to the determination signal becoming the first value. The score display apparatus according to claim 1, further comprising means for displaying a page image on the display apparatus.   When the page changing means further has no next page image specifying information for the phrase data specified by the information output by the means for outputting in response to the determination signal becoming the first value. The score display device according to claim 5, further comprising means for displaying, on the display device, a page image specified by the order of the page images for the page image displayed on the display device. For each phrase data stored in association with each page image of the score, information for specifying the phrase data is output by the means for outputting while displaying the page image corresponding to the phrase data. And further includes a counting means for counting the number of times,
The means for storing further specifies the number of times counted by the counting means and the page image to be displayed next for each phrase data associated with the page image displayed on the display device. Remember more pairs with information,
The means for displaying counts the phrase data specified by the information output by the means for outputting in response to the determination signal becoming the first value by the counting means. The score display apparatus according to claim 5 or 6, comprising means for displaying a page image specified by the information combined with the number of times displayed on the display apparatus. A display device;
Storage means for storing a series of page images of a score together with a score name and a display order of the series of page images of the score;
Image capturing means for outputting a series of image data obtained by imaging a subject in real time;
Phrase data storage means for storing, as a phrase data in association with a page name corresponding to the score name, a name of the score when the performer performs a certain phrase,
A score display means for reading a series of page images of a score corresponding to a designated score name from the storage means, and displaying a predetermined page image on the display device among the read page images;
When the page image of the score is displayed on the display device, the feature of the player's movement extracted from the output of the image capturing means is associated with the page image of the score displayed on the display device. Phrase determination means for outputting a determination signal that takes a first value when the phrase data matches the other phrase data, and a second value otherwise.
In response to the determination signal becoming the first value, page change means for displaying a page image in the next order of the page images displayed on the display device on the display device. A score display method in a score display device,
The score display means reads a series of page images of a score corresponding to a designated score name from the storage means, and displays a predetermined page image of the read page images on the display device;
When the phrase judging means displays a page image of a score on the display device, the player's movement characteristics extracted from the output of the image capturing means are displayed on the display device. Outputting a determination signal that takes a first value when it matches the phrase data associated with the page image, and a second value otherwise.
The page changing means displaying, on the display device, the page image in the next order of the page images displayed on the display device in response to the determination signal becoming the first value; Music score display method including A display device;
When executed by a computer to which image capturing means for outputting a series of image data obtained by imaging a subject in real time is connected,
Storage means for storing a series of page images of a score together with a score name and an order of the series of page images of the score;
Phrase data storage means for storing, as a phrase data in association with a page name corresponding to the score name, a name of the score when the performer performs a certain phrase,
A score display means for reading a series of page images of a score corresponding to a designated score name from the storage means, and displaying a predetermined page image on the display device among the read page images;
When the musical score page image is displayed on the display device, the player's movement feature extracted from the output of the image capturing means is associated with the musical score page image displayed on the display device. Phrase determination means for outputting a determination signal that takes a first value when the phrase data matches the other phrase data, and a second value otherwise.
In response to the determination signal having reached the first value, the page signal in the order next to the page image displayed on the display device is caused to function as a page changing means for displaying on the display device. , Computer program. A computer-readable recording medium on which the computer program according to claim 9 is recorded.

Images