JP5999689B2 - Performance system and program - Google Patents

Description

  The present invention relates to a performance system and a program, and more particularly to a performance system and a program for performing a performance using a drawn staff.

  Japanese Patent Laying-Open No. 2008-123181 (Patent Document 1) discloses a musical score recognition apparatus that analyzes a musical score image and automatically generates MIDI data. Here, since the score recognition apparatus of Patent Document 1 includes an interface for transmitting and receiving MIDI data to and from an external MIDI device, if the score recognition device is connected to the external MIDI device, it is based on the scanned score. MIDI data can be directly output to an external MIDI device for performance.

  However, the score recognition device of Patent Document 1 generates MIDI data by batch processing (batch processing) in units of musical scores or paragraphs. The score recognition device generates MIDI data from a score, and the external MIDI device Performing based on the generated MIDI data is essentially irrelevant.

JP 2008-123181 A

  An object of this invention is to provide the novel performance system and program which perform a performance using a staff score.

  As a result of reexamining the music notation that has been used as a music description format, the present inventor has come up with a new performance form using the music notation, and has reached the present invention.

  That is, according to the present invention, a performance system for performing a performance using a drawn staff score, a performance device equipped with a photographing means, and an image analysis for analyzing an image of the staff score photographed by the photographing means A pitch determining unit that determines a pitch of a sound represented by a note drawn on the staff based on an analysis result of the image analysis unit, and generates pronunciation data based on the determined pitch A sound generation data generation unit, a sound generation control unit that outputs a sound signal based on the sound generation data, a performance GUI is generated based on the analysis result of the image analysis unit, and a performance operation is received via the performance GUI A performance GUI unit, and the performance GUI unit generates a performance GUI that superimposes and displays a sound line extending in the Y-axis direction on a background image representing the staff. The sound generation data generation unit includes: While serial sound line is superimposed on the note of the image on the musical staff, performance system for outputting the sound data to the sound control unit is provided.

  As described above, according to the present invention, a novel performance system for performing a performance using a staff is provided. According to the performance system of the present invention, a simple printed score for music beginners can be reproduced, so that an interactive music learning experience using a number of score materials published so far can be provided to the user. Can do.

The functional block diagram of the performance system of this embodiment. The figure which shows the implementation form of the performance system of this embodiment. The figure which shows the staff score drawn with the pencil on paper. The figure which shows the device for performance in this embodiment. The flowchart which shows a real-time performance process. The flowchart which shows an image analysis process. The conceptual diagram for demonstrating an image analysis process. The conceptual diagram for demonstrating an image analysis process. The conceptual diagram for demonstrating the GUI screen for performance. The conceptual diagram for demonstrating the process which determines a sound volume. The conceptual diagram for demonstrating the process which adds an effect. The conceptual diagram for demonstrating an audio | voice setting process.

  Hereinafter, the present invention will be described with reference to embodiments shown in the drawings, but the present invention is not limited to the embodiments shown in the drawings. In the drawings referred to below, the same reference numerals are used for common elements, and the description thereof is omitted as appropriate.

  FIG. 1 shows a functional block diagram of a performance system 1000 according to an embodiment of the present invention. The performance system 1000 of the present embodiment includes a performance device 100, an information processing apparatus 200 that executes predetermined information processing based on input from the performance device 100, and a sound signal output from the information processing apparatus 200 as a sound wave. And a display device 400 that displays image data output from the information processing device 200.

  The performance device 100 according to the present embodiment is configured to include a photographing unit 110 and a mode switch 120. The photographing means 110 can be configured as a digital camera equipped with a CCD image sensor or a CMOS image sensor, and the performance device 100 receives an image photographed by the photographing means 110 and a switch signal of the mode change switch 120 as an information processing device. 200.

  The information processing device 200 is a computer device including a CPU, a RAM, a ROM, an external storage device such as a hard disk and a flash memory, and an appropriate communication interface such as a network interface and a wireless interface. Object-oriented programming languages such as C, C ++, Visual C ++, Visual Basic, Java (registered trademark) under the control of an appropriate operating system (OS) such as registered trademark (registered trademark), UNIX (registered trademark), LINUX (registered trademark) By executing a predetermined application program described by the above, an image analysis unit 211, an OCR unit 212, a pitch determination unit 214, a volume determination unit 216, an effect unit 218, a performance GU Part 220, tone color setting section 224, sound data generation unit 226, the sound generation control unit 228, which functions as a mode switching unit 230.

  The image analysis unit 211 and the OCR unit 212 execute predetermined image analysis processing based on an image input from the performance device 100 via the image input I / F 210. Each of the pitch determination unit 214, the sound volume determination unit 216, the effect unit 218, and the performance GUI unit 220 executes the following processing based on the analysis result of the image analysis unit 211.

  In other words, the pitch determination unit 214 determines the pitch of the sound represented by the notes described in the staff, and the volume determination unit 216 determines the volume of the sound represented by the notes. The effect unit 218 adds a predetermined effect to the sound represented by the note, and the performance GUI unit 220 generates a performance GUI for accepting a performance operation and displays the performance GUI on the display device 400.

  The sound generation data generation unit 226 is a functional unit for generating sound generation data referred to as MIDI data based on the pitch determined by the pitch determination unit 214, and is operated by the user via the performance GUI screen. The sound generation data generated in response to the input is output to the sound generation control unit 228.

  The sound generation control unit 228 converts the sound generation data input from the sound generation data generation unit 226 into an audio signal such as WAV reflecting the sound volume determined by the sound volume determination unit 216 and a predetermined effect applied by the effect unit 218, The sound is output to the sound generation means 300 referred to as a speaker device.

  Here, the components of the performance device 100, the information processing device 200, the sound generation means 300, and the display device 400 shown in FIG. 1 may be physically separated from each other, and are integrated in an appropriate combination. May be. For example, the functions of the information processing device 200, the sound generation means 300, and the display device 400 may be integrated into one device (for example, a desktop PC, a notebook PC, a smartphone, a tablet terminal, a PDA, etc.), or a performance device 100, the information processing device 200, the sound generation means 300, and the display device 400 may all be integrated into one housing. When collected in one housing, the performance system 1000 can be referred to as a performance device. Note that the physically separated components can be connected via a local connection or a network such as the Internet or a LAN regardless of a wired or wireless system.

  Furthermore, the display device 400 according to the present embodiment may be any device that can display an image. Depending on the implementation of the performance system 1000, a desktop display device, a head-mounted display, a projector Appropriate means such as an apparatus and a small liquid crystal panel can be employed.

  The basic configuration of the performance system 1000 according to this embodiment has been described above. Next, the performance system 1000 will be described based on a more specific mounting form. In the following description, FIG. 1 will be referred to as appropriate.

  FIG. 2 illustrates an implementation of the performance system 1000. A performance system 1000 illustrated in FIG. 2 includes a performance device 100 and an output device 500 that a user holds and operates like a pen. Here, the output device 500 should be referred to as a device in which the functions of the information processing device 200, the sound generation means 300, and the display device 400 described above are integrated.

  In performing the performance, the user first draws a musical score on an arbitrary medium such as paper as shown in FIG. FIG. 3A shows a staff notation drawn with a pencil on a paper surface. As shown in FIG. 3A, in the improvisational performance using the performance system 1000, it is assumed that a simplified note (simple note consisting only of note heads) is described on the staff. This is because, in the improvisational performance in the present embodiment, since it is assumed that the user dynamically determines the length (sound value) of the sound, it is sufficient that the note represents only the pitch. And by using such a simple note, the amount of data required for the staff score recognition process can be reduced, so that the staff score recognition processing speed is improved and the real-time performance is improved.

  When the user finishes writing the staff, he activates the performance device 100 and the output device 500, and traces the staff in a state where the performance device 100 is in contact with the paper surface as shown in FIG. 2B. Move from left to right. At this time, the user moves the performance device 100 at a desired speed while viewing the performance GUI screen 20 displayed on the display screen of the output device 500. Then, when the performance device 100 reaches the top of the note recorded on the staff, a sound corresponding to the note is generated from the sound generation means of the output device 500.

  4 shows a front view, a top view, a bottom view, a side view, and a side cross-sectional view of the performance device 100 shown in FIG. As shown in FIG. 4, the performance device 100 includes a columnar part 132 and a substantially hemispherical reading unit 134 connected to one end of the columnar part 132. Two toggle switches 122 and 123 for switching modes are disposed on the outer peripheral surface of the cylindrical portion 132, and when the toggle switches 122 and 123 are pressed, the output device 500 is switched to the real-time performance mode and the recording performance mode. The recorded contents are reset when the toggle switch 124 is pressed.

  The substantially hemispherical reading unit 134 has a hollow shape, and two notch portions 136 and 136 facing each other are formed on the outer periphery thereof so as to guide the reading direction to the user. Here, the two notches 136 and 136 are formed so as to face each other in the X-axis direction of the camera coordinate system of the CCD camera 110, and the user reads the staff notation in FIGS. 2B and 3. As shown in (b), the reading device 134 is brought into contact with the paper surface so that all of the five drawing lines can be accommodated in the width direction of the cutout portion 136, and the performance device 100 is held approximately on the X axis while maintaining this state. Move to trace the direction.

  Further, the performance device 100 is equipped with a CCD camera 110 facing the inside of the reading unit 134, and an LED 137 for illuminating the inside of the reading unit 134 is disposed. In a state where the reading unit 134 is in contact with the paper surface, the CCD camera 110 faces the staff on the paper surface. Then, in this state, as shown in FIG. 3B, the CCD camera 110 sets the angle of view so that the staff and at least one musical note described on the staff can be photographed simultaneously. In addition, as an alternative means for the CCD camera 110 and the LED 137, an infrared camera and an infrared irradiation means may be used. The image output of the CCD camera 110 and the ON / OFF signals of the toggle switches 122, 123, and 124 (mode switch 120) are transferred to the output device 500 in real time via the wireless communication I / F 140.

  Although the performance device 100 has been described above, it should be noted that the shape and structure shown in FIG. 4 are merely examples, and various design changes are possible.

  In this embodiment, in response to the toggle switch 122 of the performance device 100 being pressed by the user, the mode switching unit 230 switches the operation mode of the information processing apparatus 200 to the real-time performance mode. Hereinafter, real-time performance processing executed in the information processing apparatus 200 will be described with reference to the flowchart shown in FIG.

  In the real-time performance mode, first, the image analysis unit 211 executes image analysis processing (step 100). Hereinafter, image analysis processing executed by the image analysis unit 211 will be described based on the flowchart shown in FIG.

  First, in step 101, a digital image of a handwritten score transferred from the performance device 100 in real time is acquired. FIG. 7A illustrates a digital image of the acquired handwritten score. The image shown in FIG. 7A includes five drawing lines extending in the X-axis direction of the image coordinate system and two simple musical notes (hereinafter simply referred to as musical notes).

  Next, in step 102, binarization processing is executed on the acquired digital image. Note that step 102 may appropriately include image processing (noise removal, smoothing, edge enhancement, thinning, background normalization, etc.) used to increase the accuracy of image recognition. FIG. 7B shows a binary image obtained by binarizing the digital image shown in FIG.

  Next, in step 103, a process for dividing the binarized image in the X-axis direction of the image coordinate system is executed. Specifically, the binarized image is divided into an area where only the staff is present (hereinafter referred to as the first area) and the remaining area (hereinafter referred to as the second area). As a result, the area where the musical notes are drawn on the staff is divided as the second area. FIG. 7C shows a state where the binarized image is divided into regions. In FIG. 7C, the first region and the second region are indicated by encircled numerals 1 and 2, respectively (the same applies to the following drawings). However, in step 103, as an exception, an area where the staff is not drawn, such as both ends of the staff, or an area where all the staffs cannot be recognized due to faint or broken lines, It may be divided as the second area regardless of whether it is drawn.

  Next, based on the result of step 103, it is determined in step 104 whether or not the first area exists. If the first area exists (step 104, Yes), the process proceeds to step 105 as it is. On the other hand, if the first region does not exist (step 104, No), the process proceeds to the timbre setting process (step 300) and then proceeds to step 105. The tone color setting process will be described later.

  In step 105, five estimated lines are defined in the second region by performing linear interpolation based on the coordinate information of the staff existing in the first region. Specifically, as shown in an enlarged view in FIG. 8A, the first region located on the left side of the second region of interest with respect to the drawing lines in the two first regions adjacent to the left and right of the second region. A line segment S connecting the right end pixel R of the inner drawing line and the left end pixel L of the drawing line in the first region located on the right side of the second region of interest is estimated as a staff in the second region. At this time, the five estimated lines are marked as the first to fifth lines in order from the bottom to the top of the image.

  Next, in step 106, a pixel area of a note drawn in the second area is defined using a technique such as a labeling process, and then, as shown in an enlarged view in FIG. And the coordinates of the representative pixel D of the note. FIG. 8B defines a reference line B extending in the Y-axis direction so as to bisect the second region, and a pixel corresponding to the midpoint of a line segment connecting two edge pixels existing on the reference line B. In this example, the representative pixel D may be determined in accordance with an appropriate rule, for example, the center of gravity of the note pixel region may be the representative pixel D.

  Finally, in step 107, for each second region, the coordinate information of the five estimated lines defined in the previous step 105 and various coordinate information (edge pixels and representative pixels) relating to the musical note obtained in the previous step 106 Is stored in the temporary storage buffer as musical score image information (step 107), and the process ends. Note that, in the case where a note is not drawn in the second area, the musical score image information related to the second area is held as a blank.

  Returning to FIG. 5 again, the description will be continued. When the image analysis process (step 100) ends, the process proceeds to step 201, and the performance GUI unit 220 determines whether or not the sounding line to be displayed on the performance GUI screen is on the second area. Here, the performance GUI screen generated and provided by the performance GUI unit 220 will be described.

  FIG. 9 is a conceptual diagram for explaining a performance GUI screen in the present embodiment. In FIG. 9, for ease of understanding, the performance GUI screen 20 is displayed superimposed on the handwritten score (hereinafter the same as in FIGS. 10 and 11). The performance GUI screen in this embodiment is generated by the performance GUI unit 220 based on the analysis result of the image analysis unit 211. The performance GUI screen 20 shown in FIG. 9 displays an image (for example, color-coded display) in which the first area and the second area of the binarized image of the image transferred from the performance device 100 are displayed so as to be distinguishable. And a straight line P extending in the Y-axis direction of the image coordinate system and traversing an appropriate position in the center of the background image is fixedly displayed. In the present embodiment, this straight line P is defined as a sound generation line P.

  In response to the user tracing the handwritten score from left to right with the performance device 100, the background image displayed on the performance GUI screen 20 changes from moment to moment. If the performance GUI screen 20 is in the state shown in FIG. 9A in step 201, the sounding line P is not on the second region (step 201, No), and the process returns to step 100. The image analysis process is executed again based on the latest image transferred from the performance device 100.

  Subsequently, as a result of the user moving the performance device 100 to the right side, as shown in FIG. 9B, when the sound generation line P is superimposed on the second area (step 201, Yes), the processing is performed. Proceeding to step 202, it is determined whether or not there is a note in the second region where the sound generation line P is superimposed. This determination can be made based on whether or not the score image information is held in the temporary storage buffer for the second area. In short, it is determined whether or not the sounding line P displayed on the performance GUI screen is superimposed on the note image through the above-described step 201 and step 202.

  In the state shown in FIG. 9B, the note Q is present in the second region where the sounding line P is superimposed (step 202, Yes), so that the process proceeds to step 203, and the second stored in the temporary storage buffer. Based on the score image information of the two areas, the pitch determination unit 214 determines the pitch of the sound represented by the note Q. For example, when the musical score image information shown in FIG. 8C is held for the second region where the note Q exists, an appropriate algorithm is based on the mutual positional relationship between the representative pixel D and the first to third lines. Use to determine the pitch. Note that the method for determining the pitch represented by the note from the image of the musical score is known, and in this embodiment, the pitch is not limited to the above-described procedure, and the pitch is calculated using an appropriate algorithm including a known method. Can be determined. Hereinafter, the description will be continued assuming that the pitch “so” is determined in step 203.

  Subsequently, the process proceeds to step 204, where the volume determination unit 216 determines the volume of the sound represented by the note Q based on the same musical score image information. FIG. 10A is a conceptual diagram for explaining the process of determining the volume. In determining the volume, the volume is determined based on the width of the note Q in the Y-axis direction. FIG. 10A shows the pixel distance between two edge pixels of the note Q existing on the reference line B extending in the Y-axis direction so as to bisect the second region as the width of the note Q in the Y-axis direction. An example in which the volume is determined based on the width is shown. In addition to this, a line passing through the representative pixel D of the note Q extending in the Y-axis direction is defined as a reference line B, and the pixel distance between two edge pixels of the note Q existing on the reference line B is defined as the Y of the note Q. It can also be an axial width.

  Subsequently, the process proceeds to step 205, where sound generation is started with the pitch and volume determined in the above-described procedure. Specifically, the sound generation data generation unit 226 generates sound generation data (MIDI data) based on the timbre set by the timbre setting unit 224 and the pitch “seo” determined by the pitch determination unit 214, thereby generating sound control To the unit 228. The sound generation control unit 228 converts the sound generation data of the pitch “So” input from the sound generation data generation unit 226 into a sound signal such as WAV based on the sound volume determined by the sound volume determination unit 216 and sends it to the sound generation unit 300. Output.

  After the sound generation is started, the performance GUI unit 220 instructs the sound generation data generation unit 226 to continue sound generation as long as the sound generation line P is superimposed on the second area. In response to this, the sound generation data generation unit 226 continues to output sound generation data to the sound generation control unit 228. Further, during this time, in response to a predetermined movement of the performance device 100, the processing of Step 206 or Step 207, which will be described later, is executed, and the nature of the sound to be sounded dynamically changes. Hereinafter, this point will be described.

  In step 206, in response to the performance device 100 rotating about the longitudinal direction while maintaining the state where the sound generation line P is superimposed on the second region, the volume determination unit 216 represents the note Q. Change the volume of the sound. The processing executed in step 206 is essentially the same as the processing executed by the volume determination unit 216 in the previous step 204. The volume determination unit 216 constantly monitors the width of the note Q in the Y-axis direction, The volume is dynamically reset based on the width information. In general, since a note (note head) is often described as an ellipse having an inclination with respect to a staff, the performance device 100 is rotated to the right from the state shown on the left side of FIG. In the state shown on the left side of FIG. 10B, the pixel distance changes from W1 to W2 as shown on the right side of FIG. 10B (W2> W1). In response to this, the volume determination unit 216 increases the set volume of the sound represented by the note Q.

In step 207, in response to the performance device 100 moving in the musical score vertical direction (that is, the Y-axis direction of the image coordinate system) while maintaining the state where the sound generation line P is superimposed on the second region, the effect unit 218 adds a pitch vent effect to the pronunciation. Specifically, as shown in FIG. 11 (b) from the initial state shown in FIG. 11 (a), in response to the user moving the performance device 100 in the vertical direction of the musical score, the effect unit 218 generates a sound. The absolute value of the difference between the Y coordinate value [Y ₁ ] of the representative pixel D of the note Q at the time of start and the Y coordinate value [Y ₂ ] of the representative pixel D of the note Q after movement is detected as a movement amount, The pitch of the sound is changed steplessly from the pitch “seo” according to the amount of movement (that is, the frequency is incremented from the pitch “seo” based on the frequency). Although the description has been given based on the pitch vent effect here, the effect applied by the effect unit 218 is not limited to this, and an arbitrary effect (for example, reverb or chorus) is selected according to the amount of movement in the musical score vertical direction. Etc.) may be set.

  In subsequent step 208, the performance GUI unit 220 determines again whether or not the sounding line P is on the second area. If the sounding line P is still on the second area, the procedure after step 206 is performed. repeat. On the other hand, as a result of the user moving the camera further to the right side of the musical score, when the state shown in FIG. 9C is obtained, the performance GUI unit 220 determines that the sounding line P has deviated from the second area (step 208, No). ), The performance GUI unit 220 instructs the sound generation data generation unit 226 to stop sound generation. In response to this, the sound generation data generation unit 226 stops outputting the sound generation data to the sound generation control unit 228, and the sound generation ends (step 209). Thereafter, the process returns to the image analysis process in step 100 again, and the series of processes described above is repeated.

  Next, the tone color setting process in this embodiment will be described. The tone color setting unit 224 in the present embodiment manages an arbitrary character and a predetermined tone color in association with each other. For example, the tone color setting unit 224 can link the character string “pf” to the tone color of “piano”, or link the character string “dr” to the tone color of “drum”.

  For example, as shown in FIG. 12A, after the user has handwritten a character string “pf” and a character string “dr” using an appropriate margin of the handwritten musical score, the user reads the reading unit of the performance device 100. Returning to FIG. 6, description will be made assuming that 134 is in contact with the position where the character string “pf” is drawn. In this case, since there is no first area (area where only the staff is present) in the acquired image (step 104, No), the process proceeds to step 300, and the timbre setting process is executed. The timbre setting process will be described below based on the flowchart shown in FIG.

  First, in step 301, an image of the handwritten character “pf” photographed by the performance device 100 is acquired. Subsequently, in step 302, the OCR process of the image of the handwritten character “pf” acquired by the OCR unit 212 is executed to recognize the handwritten character. Finally, in step 302, the tone color setting unit 224 sets the piano tone color associated with the recognized character “pf” in the pronunciation data generation unit 226.

  Although the real-time performance mode of the present embodiment has been described above, in this embodiment, the mode switching unit 230 is operated by the information processing apparatus 200 in response to the toggle switch 124 of the performance device 100 being pressed by the user. Switch to the recording performance mode. In the recording performance mode, while the user scans the staff, the pronunciation data generation unit 226 performs the pitch and volume determined by the pitch determination unit 214 and the volume determination unit 216, and the period of stay in the second region of the pronunciation line P. MIDI data is generated and stored based on the sound value determined accordingly, and the performance is reproduced in response to a request from the user.

  Although the present invention has been described with the embodiment, the present invention is not limited to the above-described embodiment.

  For example, in the above-described embodiment, the aspect of performing using the hand-drawn staff notation has been described. However, the performance system of the present invention is not limited to the hand-drawn staff notation, but the printed staff notation and the printed staff Needless to say, any musical notation such as a musical notation in which handwritten notes are drawn is applicable. The medium on which the staff is drawn is not limited to paper, and any medium may be used.

  In the performance GUI screen of the above-described embodiment, the sound line is defined, and the straight line representing the sound line is superimposed on the background image. However, the graphic representing the sound line is displayed by the user. Anything can be used as long as the pronunciation line can be intuitively recognized. For example, a graphic such as an arrow may be employed instead of the straight line described above.

  In the above-described embodiment, the timbre setting process is automatically executed based on the content of the acquired image. However, a dedicated switch for the timbre setting process is provided, and the timbre is set in response to pressing of the switch. A setting process may be executed. In the above-described embodiment, the aspect in which the timbre is set using the OCR function has been described. However, in the present invention, predetermined setting items such as an effect, a rhythm box, a sound range, and a tempo are used using the OCR function. It may be configured to set.

  In the present invention, a plurality of colors and timbres are associated with each other in the timbre setting unit, and the color image obtained by the 3CCD camera mounted on the performance device is analyzed, so that the musical note is changed according to the coloring of the musical note. It can also be configured to set the timbre of the sound to be represented. Further, the sound volume determination unit may be configured to determine the sound volume based on the area (number of pixels) of the pixel area of the note, and may sound at a constant sound volume regardless of the size and shape of the note. Furthermore, an appropriate vibration actuator may be mounted on the performance device, and a tactile feedback configuration may be employed in which a vibration of about 10 ms is given to the user when a note is generated or when OCR recognition is successful. In addition, it is included in the scope of the present invention as long as the effects and effects of the present invention are exhibited within the scope of embodiments that can be considered by those skilled in the art.

  Each function of the above-described embodiment can be realized by a device-executable program written in an object-oriented programming language such as C, C ++, C #, Java (registered trademark), and the program of this embodiment is a hard disk device. , CD-ROM, MO, DVD, flexible disk, EEPROM, EPROM and the like can be stored and distributed in a device-readable recording medium, and can be transmitted via a network in a format that other devices can.

  Finally, application development of the performance system of the present invention will be described.

(Use as music education content)
In Japan, even though compulsory education requires reading the staff notation, there is still the reality that many people still become adults without being able to read the score. This is because learning so far only begins to theoretically learn the format of the staff notation, so that the visual information of the staff and the sound it represents (auditory information) will last in the student's head. The cause is that they are not connected.

  In this regard, if the performance system of the present invention is introduced as a content for music education, the student can immediately confirm the musical score handwritten as he / she thinks as a sound, so notation → pronunciation → notation → You will be able to read the score naturally as you play through the pronunciation cycle.

(Proposal of new performance)
In recent years, a performance form called laptop music in which a performance is performed using a laptop computer has attracted attention. In laptop music, live performances are performed using only operating devices such as laptop computers and samplers, but the appearance of the player's operation itself is far from an attractive performance. Audiences of live performances have a great appeal not only to the sound played by the performer, but also to the performance performance of the performer. In this regard, laptop music is inferior to playing classical instruments. I do not get.

  On the other hand, if the performance system of the present invention is applied to laptop music, it is possible to show the human performance to the audience that is similar to the performance of conventional classical instruments while maintaining the original advantages of laptop music. become. For example, the following performance may be realized.

  The performer improvisely draws a staff on the stage and traces the staff that has been written with the performance device of the present invention. The situation is projected on a large screen. At this time, the performance GUI screen is also displayed on the large screen. The performer can increase or decrease the volume and pitch by moving the performance device by increasing or decreasing the speed of tracing the stave to give the melody more or less, reversing the direction of the trace to produce a reverse playback sound, or moving the performance device. Nuances such as shaking, tremolo, vibrato, and pitch bent are improvised in the performance, and while the audience watches, new scores are added one after another, and each score is loop-sequenced with different instrument sounds. And so on. Thus, if the performance system of the present invention is applied to laptop music, it will be possible to attract the audience both visually and auditorily.

20 ... GUI screen for performance 100 ... Device for performance 110 ... Photography means (CCD camera)
DESCRIPTION OF SYMBOLS 120 ... Mode change switch 122,123,124 ... Toggle switch 132 ... Cylindrical part 134 ... Reading part 136 ... Notch part 137 ... LED
140 ... Wireless communication I / F
200 ... Information processing device 210 ... Image input I / F
211 ... Image analysis unit 212 ... OCR unit 214 ... Pitch determination unit 216 ... Volume determination unit 218 ... Effect unit 220 ... Performance GUI unit 224 ... Tone setting unit 226 ... Sound generation data generation unit 228 ... Sound generation control unit 230 ... Mode switching Part 300 ... Sound generation means 400 ... Display device 500 ... Output device 1000 ... Performance system

Claims (8)

A performance system for performing a performance using a drawn staff notation,
A performance device equipped with a photographing means;
An image analysis unit for analyzing an image of the staff score photographed by the photographing means;
A pitch determination unit that determines a pitch of a sound represented by a note drawn on the staff based on the analysis result of the image analysis unit;
A pronunciation data generation unit that generates the pronunciation data based on the determined pitch;
A sound generation control unit that outputs a sound signal based on the sound generation data;
A performance GUI unit that generates a performance GUI based on the analysis result of the image analysis unit and receives a performance operation via the performance GUI;
Including
The performance GUI unit generates a performance GUI that superimposes and displays a straight line extending in the Y-axis direction as a pronunciation line on the background image representing the staff.
The sound generation data generation unit outputs the sound generation data to the sound generation control unit while the sound generation line is superimposed on the image of the note on the staff .
An effect unit that adds a predetermined effect in an amount corresponding to the amount of movement in the Y-axis direction to the sound represented by the note in response to the photographing unit being moved in the Y-axis direction;
Performance system.   The performance system according to claim 1, wherein the staff notation is drawn by handwriting, and the notes are described by simple notes with only note heads.   A volume determination unit that determines a volume of a sound represented by the note based on a width of the note in the Y-axis direction, and the sound generation control unit outputs the audio signal based on the determined volume. The performance system according to claim 1 or 2. The performance system according to claim 1, wherein the predetermined effect is a pitch vent effect. An OCR unit that performs character recognition on an image photographed by the photographing unit;
It further includes a timbre setting unit for managing an arbitrary character and a predetermined timbre in association with each other,
The performance system according to any one of claims 1 to 4 , wherein the timbre setting unit sets a predetermined timbre associated with a character recognized by the OCR unit as a timbre of a sound represented by the note. A computer-executable program for causing a computer to perform a performance using a drawn staff.
Analyzing an image of a musical score taken by a performance device equipped with a photographing means;
Determining a pitch of a sound represented by a note drawn on the staff based on the image analysis result of the staff;
Generating pronunciation data based on the determined pitch;
Generating a performance GUI for displaying a straight line extending in the Y-axis direction as a pronunciation line on a background image representing the staff based on the image analysis result of the staff;
Outputting an audio signal based on the pronunciation data while the pronunciation line is superimposed on the image of the note on the staff .
Performing a step of applying a predetermined effect in an amount corresponding to the amount of movement in the Y-axis direction to the sound represented by the note in response to the photographing unit being moved in the Y-axis direction. Program to let you. The step of analyzing the image of the staff is as follows:
Binarizing the stave image to generate a binarized image;
Dividing the binarized image into a first region where only a staff is present and a second region where a note is drawn on the staff in the X-axis direction of the image coordinate system ; Including
The step of generating the performance GUI includes:
For the binarized image, the displayed image obtained by dividing the first region and the second region so as to be distinguishable is used as the background image, and a straight line extending across the center of the background image is fixedly displayed as the sound generation line. The program according to claim 6 , comprising steps. In the step of the area division, tone when said first area does not exist, the imaging means performs character recognition on the image captured is, the sound representing a predetermined tone which are linked to the recognized character is the note The program according to claim 7 , set as