JP4957329B2 - Performance information display device and program - Google Patents

Description

  The present invention relates to a performance information display device and a program suitable for use in displaying a score or the like on a portable information terminal.

  Conventionally, various techniques for displaying a musical score or the like on a display such as an electronic musical instrument are known. For example, Patent Document 1 discloses a technique for displaying a score composed of a plurality of parts in a window on a display. Here, when the user selects a desired rectangular portion in the score with a mouse or the like, the portion is enlarged and displayed in another window. Patent Document 2 discloses a technique for emphasizing a specific part of a musical score by changing display shades, line spacing of a staff notation, and the like.

JP 07-271361 A JP 2004-309580 A

By the way, in recent years, there has been a demand for editing performance information, automatic performance, and the like while displaying a musical score or the like on the portable information terminal due to improvement in performance of the portable information terminal. However, many displays provided in the portable information terminal have a small display area and a low resolution, and there is a problem that visibility is lowered when a score is displayed as it is.
The present invention has been made in view of the above-described circumstances, and can display a musical score information required by a user in a relatively small and low-resolution display in an easy-to-see manner, and further enhance the degree of freedom in display and can be enjoyed visually. An object is to provide an information display device and a program.

In order to solve the above problems, the present invention is characterized by having the following configuration. The parentheses are examples.
2. The performance information display device according to claim 1, wherein the performance information is a storage means (20) for storing performance information including object specification information (logical score information), wherein the object specification information is a display for displaying the performance information. For each of a plurality of objects constituting the screen, the storage means (20) for specifying the object and the layer to which the object belongs, and a virtual three-dimensional space consisting of time, height, and depth, Surface designating means (2, 10, SP10) for designating a curved surface (S (t, d)) for arranging each layer, a virtual viewpoint for viewing each layer in the three-dimensional space, and viewpoint designating means (2,10, SP14) for specifying the orientation and, a display means for displaying a virtual image viewed from said viewpoint (8, SP40), on the three-dimensional space, virtual A light source coordinate a is the light source coordinates (P _L), a light source specifying means for specifying the focal coordinates (P _B) is a coordinate the optical axis toward the light source, wherein each object and the light source coordinates (P _L) a straight line connecting the display density setting means (2,10, SP28) for setting the display density of each object according to the angle (theta) formed between the optical axis and having a.
Further, in the configuration according to claim 2, in the performance information display device according to claim 1, the curved surface designating means (2, 10, SP10) is a curved surface currently applied (S (t, d)). And a means for deforming.
Furthermore, in the configuration according to claim 3, the performance information display device according to claim 1 or 2, further comprising depth changing means for changing the depth of each layer in the three-dimensional space. To do .
Et al is, in the configuration of claim 4, wherein, in the performance information display device according to any one of claims 1 to 4, the temporal position acquisition means for acquiring a temporal position of the automatic performance (t) ( SP16), “position and direction of the viewpoint”, “position and direction of the light source”, or “position of the object on the layer” are moved according to the temporal position (t). It is characterized by further having a moving means.
Furthermore, in the structure of Claim 5, in the performance information display apparatus in any one of Claim 1 thru | or 4 , a display magnification is set with respect to the image displayed on the said display means (8). The apparatus further comprises means.
Further, in the sixth aspect of the program, the processor, and instructions (SP16) for reading the playing information including object identification information (logical score data) from the storage means (20), wherein, the object specific information all SANYO identifying the layer in which the object and the object belongs for each of a plurality of objects constituting a display screen for displaying the performance information, time, virtual three-dimensional space of height and depth A procedure (SP10) for designating a curved surface (S (t, d)) for arranging each layer, a virtual viewpoint for viewing each layer in the three-dimensional space, and its orientation the procedure for specifying (SP14), and the procedure (SP40) for displaying a virtual image viewed from the viewpoint, on the three-dimensional space, light source coordinates and the coordinates of the virtual light source And instructions that specify the focal coordinates are coordinates that the optical axis of the light source is directed, said the straight line connecting the respective object with the light source coordinates, the display density of each object in accordance with the angle between the optical axis It is a program for executing the setting procedure .

Thus, according to the present invention, for each of a plurality of objects constituting a display screen for displaying performance information, the layer to which the object belongs is specified based on the performance information, and the curved surface in the virtual three-dimensional space is identified. Since these layers can be arranged and a virtual video viewing these layers from the viewpoint can be displayed, the musical score information required by the user can be easily displayed and the degree of freedom in display can be increased. Furthermore, specifying a focus coordinate is a coordinate that the optical axis is directed in the source coordinates and the light source are coordinates of a virtual light source on the three-dimensional space, linear with the optical axis connecting the respective object and the light source coordinate since setting the table 示濃of each object according to the angle of bets forms, it can be given a three-dimensional shaded feeling becomes more visible.

1． Hardware Configuration of Embodiment Next, the hardware configuration of a portable information terminal according to an embodiment of the present invention will be described with reference to FIG.
In FIG. 1, reference numeral 2 denotes an operation unit, which includes various switches. A detection circuit 4 detects the state of the operation unit 2. Reference numeral 6 denotes a touch panel, on which various images are displayed by the display circuit 8. Further, the operation state with respect to the touch panel 6 is detected by the detection circuit 10. An audio interface unit 12 outputs an audio signal via a headphone (not shown) and the like, and inputs an audio signal via a microphone (not shown). A communication interface unit 14 inputs and outputs data with an external device.

  A CPU 22 controls each unit in the portable information terminal via the bus 16 according to a program stored in the flash memory 20. Reference numeral 18 denotes a RAM, which is used as a work memory for the CPU 22. A timer 24 measures the current time and generates a timer interrupt to the CPU 22 as necessary. Reference numeral 26 denotes a recording medium slot, which reads / writes data from / to a recording medium such as a memory card inserted therein.

2． Example premise theory
2.1. Outline of Processing Next, various assumption theories in this embodiment will be described. First, an outline of the processing will be described.
In this embodiment, a score is displayed in a three-dimensional manner on the touch panel 6 of the portable information terminal. First, assuming a three-dimensional space called “original coordinate system” having a T axis (time), an H axis (height), and a D axis (depth), as shown in FIG. 4 (a), a plurality of layers L ₁ , Assume that L ₂ ,..., L _n are arranged in the space. Here, each layer _{L 1, L 2, ......,} L n is generally part Pt _1, Pt ₂ in score, ..., each corresponding to Pt _n.

In addition, an “object” such as a musical note is arranged in each layer L _i (where i = 1, 2,..., N). An example is shown in FIG. In FIG. 3, the position of an object Q on the layer L _i is expressed as “Q (t _q , h _q )” based on the positions on the T axis (time) and H axis (height). . The coordinate system of the three-dimensional model is generally three-dimensional. However, in the present invention, each layer is a plane having no thickness. Therefore, the position of the object on each layer is expressed by orthogonal two-dimensional coordinates. Here, the time position t _q corresponds to the position in the horizontal direction on the score, and is represented, for example, with the origin O as the starting point of the music and the bars and beats as units. The time position t _q may be expressed in units of minutes and seconds.

Next, assuming a three-dimensional space called “world coordinate system” having an X _W axis, a Y _W axis, and a Z _W axis, a curved surface S (t, d) is formed in the space as shown in FIG. Is placed. Here, the curved surface S (t, d) is a mapping of the TD plane in the original coordinate system, and the mapping of each layer L _i (where i = 1, 2,..., N) is the curved surface S (t, d, d) is placed on top. Next, a method for obtaining the mapping of each object to the world coordinate system in the original coordinate system will be described. First, the depth values of the layer L _i and d _i in FIG. 5 (a). In the world coordinate system, the mapping of the straight line formed by “d = d _i ” on the TD plane becomes a curve L _S (t) on the curved surface S (t, d) as shown in FIG.

Accordingly, the map of the point (t _q , d _i ) on the TD plane in FIG. 5A is the point P _S (t _q , d _i ) on the curve L _S (t). Here, the position of the point P _S (t _q , d _i ) in the world coordinate system is (x _S , y _S , z _S ). Assuming that the layer L _i is parallel to the Y _W axis in the world coordinate system, the mapping of the object Q (t _q , d _i , h _q ) in FIG. 5A to the world coordinate system is as shown in FIG. becomes P _W points shown in), the coordinates will _{"(x W, y W, z} W) = (x S, y S + h q, z S) ".

Next, as shown in FIG. 9A, an arbitrary viewpoint O _e is defined in the world coordinate system, and the viewpoint coordinate system having the viewpoint O _e as the origin and having the X _e axis, the Y _e axis, and the Z _e axis is defined. Suppose. In this viewpoint coordinate system, based on the image viewed each layer L _i from the view point O _e, an image is obtained which displays the final touch panel 6.

2.2. Data structure of performance information Next, the data structure of performance information in the present embodiment will be described with reference to FIG. For example, the SMF (standard MIDI format) format is adopted as the performance information of the present embodiment. In FIG. 2, the performance information is composed of one header chunk 30 and a plurality of track chunks 32,. The header chunk 30 stores the number of tracks, time unit, format type, and the like. Each track chunk 32 includes timing data 34,..., 34 and event data 36,.
Here, the timing data 34,..., 34 is a timing for reproducing one or a plurality of event data 36,..., 36 immediately following it, and the event data 36, ..., 36 are MIDI events, system exclusive messages. Data such as meta events.

In the present embodiment, the score is displayed in synchronization with the automatic performance. The display content in the score is specified by the logical score information described by the system exclusive message or the meta event. The track chunk 32 described above, ..., 32, each part Pt _1, Pt _2, ......, is provided in association with the Pt _n. In the present embodiment, the above-described layers L ₁ , L ₂ ,..., L _n are formed for each of these parts.

  The “part” includes “lyric part” and “chord progression part”. Therefore, there are parts that do not involve MIDI events at all and are not involved in pronunciation. In addition, each part is not necessarily a part that is represented by one stage in a staff score. For example, in the case of a piano part or the like, there is a case where one part is displayed as a “large staff” composed of two stages of a treble clef and a treble clef. Further, the part related to “piano” may be divided into “right-hand part” and “left-hand part”, and the tracks may be divided in advance so that they are displayed on different layers.

2.3. Layer Construction Processing Next, layer data for specifying the contents of each layer is configured based on the logical score information in the performance information described above. First, the logical score information represents notes, rests, bar lines, and the like, and one or a plurality of “objects” are formed based on each logical score information. Then, objects included in one track are arranged on one layer. As described above, in FIG. 3, an object Q is represented as “Q (t _q , h _q )” based on the position on the T axis (time) and H axis (height) in the original coordinate system. The position on the layer is expressed.

The height h _q represents the position in the vertical direction on the score in the object coordinate system. For example, for a part with a staff score, the first line is the origin, and a high pitch is a large value and a low pitch is a small value corresponding to the pitch of the note information. Similarly, symbols other than notes (for example, crescendo symbols, fermata symbols, treble clefs, staffs, etc.) are represented by their H and T coordinates. Here, for convenience of explanation, the position of the object Q (t _q , h _q ) is expressed by the coordinates of the center of the note head, but in reality, the outline of the note head's outer edge, the stem, Information such as tail coordinates is also included in the object.

2.4. Modeling Conversion to Original Coordinate System Each layer data constructed by the layer construction process described above is arranged in an original coordinate system that is a three-dimensional orthogonal coordinate system. Here, each layer L _1, L _2, ......, relative to L _n, the depth d _1, d _2, ......, is given d _n. That is, each layer _Li is arranged on the TD plane so as to be parallel to the T axis and the H axis and to be orthogonal to the D axis. As a result, each object on the layer L _i is represented by “Q (t _q , d _i , h _q )”.

In the initial state, depth values d _i are assigned at equal intervals to the layer number i. That is, the depth value d _i is equal to “d ₁ × i”. However, the depth value d _i can be freely changed in the original coordinate system in accordance with subsequent user operations. That is, the user may perform an operation (for example, an operation of selecting and dragging a layer, for example) while observing a final display while imagining a world coordinate system described later. The operation amount of the operator is reflected in the display content as an operation amount in the world coordinate system, and the operation amount is further reflected in the depth value of the original coordinate system.

For example, moving from the position shown the layer L _i by the solid line A in the world coordinate system shown in FIG. 4 (b) to the position shown in broken line B, in the original coordinate system of FIG. 4 (c), the layer L _i is the solid line A ' To the position indicated by the broken line B ′. Further, the positions of arbitrary two layers L _i and L _k can be interchanged by the user performing a predetermined operation. As shown in FIG. 4D, when such an operation is executed in the world coordinate system, the operation is also reflected in the original coordinate system as shown in FIG.

2.5. Designation of Curved Surface S (t, d) Next, the process of specifying the curved surface S (t, d) in the world coordinate system will be described with reference to FIGS. First, the curved surface S (t, d) is set to coincide with the X _W Z _W plane in the initial state. Accordingly, the layers in the world coordinate system are arranged in the same manner as in the original coordinate system as shown in FIG. Here, the user can deform the curved surface S (t, d) as necessary.

First, when the user performs a predetermined operation for deforming the curved surface S (t, d) in the operator unit 2, the touch panel 6 has a plurality of control points P _ij (where i = 0, 1, 2; j = 0, 1, 2) is displayed. In the initial state, the control points P _ij are arranged in square grids at equal intervals, as shown in FIG. 6 (b). Here, the position of each control point P _ij can be moved to another position on the world coordinate system, for example, as shown in FIG.

Next, the curved surface S (t, d) in the world coordinate system is obtained by interpolating and smoothing these control points _Pij . Here, as a specific process of smoothing, there are a number of methods such as a Bezier curved surface, and any of them may be adopted. In the present embodiment, a curved surface S (t, d) is represented by a B-spline expressed by the following equation. Is identified.

The B-spline curved surface is defined by n × m control points P _ij and knot sequences in the t direction (n + p) and the d direction (m + q) as in the above equation. Here, N _{i, p} (t) and N _{j, q} (d) in the equation are B-spline basis functions, p is a rank in the t direction, and q is a rank in the d direction. “P-1” is the order in the T direction, and “q-1” is the order in the D direction. The “B-spline basis function” is well known and will not be described. From the above equation, a mapping in the world coordinate system to an arbitrary point (t, d) on the TD plane of the original coordinate system, that is, the coordinates of one point on the curved surface S (t, d) is obtained.

Here, FIG. 7A shows an example in which each layer is rearranged on the deformed curved surface S (t, d). Further, the curved surface S (t, d) may be accompanied by compression in the X _W axis and Z _W axis directions as shown in FIG. 7B, or as shown in FIG. 7C. In addition, deformation in the X _W axis and Z _W axis directions may be included. In addition, when moving the control point _Pij , a plurality of control points may be selected and dragged, or a coordinate curve that is a curve passing through the plurality of control points may be selected and dragged. Also good.

Further, the plurality of control points P _ij may be rotated around the X _W axis, the Y _W axis, the Z _W axis, and other axes determined by the user. The position of the control point P _ij is not limited to being specified by dragging or the like, and may be specified by inputting parameters such as coordinate values and angles, for example. Further, when changing the coordinates of the control points P _ij, rather than dragging the control point P _ij itself, when selecting and dragging layers and objects thereon that constitute the music (such as notes), in conjunction with this Thus, the coordinates of the control point P _ij may be determined so that the curved surface S (t, d) is deformed.

2.6. Specifying the orientation of a layer in the world coordinate system In the above-mentioned "Summary of processing", it is assumed that each layer in the world coordinate system is parallel to the _YW axis as shown in Fig. 8 (a). The orientation of the layer is not limited to this, and the user can arbitrarily specify the position on each point on the curved surface S (t, d). The “direction of each layer” is “the direction of the H axis in each layer when an object in the original coordinate system is mapped to the world coordinate system”. For example, as shown in FIG. 8 (b), H axis in each layer may also be parallel to the Y _e axis of the viewpoint coordinate system, curved S (t as shown in FIG. 8 (c), It may be in the normal direction of d).

Next, details of a process for applying the direction of the H axis to an arbitrary object Q (t _q , d _i , h _q ) included in the layer _Li will be described. First, in FIG. 8 (a) or (b), a point P _S (t _q , d _i ) on the curved surface S (t, d) and corresponding to the time position t _q and the depth value d _i in the world coordinate system. Coordinates are obtained. The point P _S is expressed as “P _S (x _S , y _S , z _S )” using the obtained coordinates x _S , y _S , z _S.

Next, a point moved by a height h _q in the H-axis direction on the layer L _i from this point P _S (x _S , y _S , z _S ) is defined as P _W, and coordinates x _W , y _W, the point P _W by z _W is referred to as _{"P W (x W, y W} , z W) ". Here, when the direction of the H axis in the layer L _i is the same as the direction of the Y _W-axis, the coordinates of the point P _W becomes as the following equation.
x _W = x _S ,
y _W = y _S + h _q ,
z _W = z _S.

Next, as an example in the case of arbitrarily setting the direction of the H axis, a method for determining the point P _W when the direction H axis corresponds to the Y _e axis of the viewpoint coordinate system. In FIG. 8B, the Y _e axis can be arbitrarily determined in a direction that is not parallel to the X _W Z _W plane. The X _e axis is determined to be orthogonal to the Y _e axis and parallel to the X _W Z _W plane. The Z _e axis is determined so as to be orthogonal to both the X _e axis and the Y _e axis. Further, the angle of the X _W axis and X _e axis is alpha, the angle Y _W-axis and Y _e axis and beta. In this case, the coordinates of the point P _W are as follows:
x _W = x _S + h _q sinα sinβ,
y _W = x _S + h _q cosβ,
z _W = z _S -h _q cosαsinβ.

2.7. Conversion to the modified world coordinate system The following processing may be performed using the object coordinates converted to the world coordinate system as they are, but the world coordinate system is appropriately affine transformed so that it can be handled easily in the subsequent processing. It is preferable to use the modified world coordinate system and perform the subsequent processing using the modified world coordinate system. Therefore, also in this embodiment, the mapping of each object with respect to the modified world coordinate system is obtained and used for the subsequent processing.

First, it is assumed that an object in the world coordinate system is arranged at a point P _W (x _W , y _W , z _W ). Assuming that the coordinates of the point P _W after the affine transformation are “P _A (x _A , y _A , z _A )”, the coordinates of the point P _A are generally expressed by the following equations.
x _A = A ₁₁ x _W + A ₁₂ y _W + A ₁₃ z _W + A ₁₄ ,
y _A = A ₂₁ x _W + A ₂₂ y _W + A ₂₃ z _W + A ₂₄ ,
z _A = A ₃₁ x _W + A ₃₂ y _W + A ₃₃ z _W + A ₃₄ .

Among the above affine transformations, in this embodiment, enlargement / reduction, parallel movement, rotation, and the like are effective in displaying the score easily. Examples of "scaling", and A _11-fold scale in X _W axis direction, Y _W-axis direction by reducing A _22-fold expansion, if reduced A _33-fold expansion in Z _W-axis direction, the following equation It ’s good to make some conversions.
x _A = A ₁₁ x _W , y _A = A ₂₂ y _W , z _A = A ₃₃ z _W.

Further, examples of "parallel movement", A ₁₄ moves the X _W axis direction, A ₂₄ to move to the Y _W-axis direction, when A ₃₄ moves the Z _W-axis direction, may be performed the following conversion .
x _A = x _W + A ₁₄ , y _A = y _W + A ₂₄ , z _A = z _W + A ₃₄ .
In addition, as an example of “rotation”, when rotating around X _W axis by θ _X, rotating around Y _W axis by θ _Y , or rotating around Z _W axis by θ _Z , respectively, It ’s good to make some conversions.
Around the X _W axis: x _A = x _W , y _A = cos θ _X y _W -sin θ _X z _W , z _A = sin θ _X y _W + cos θ _X z _W.
Around Y _W axis: x _A = cosθ _Y x _W + sinθ _Y z _W , y _A = y _W , z _A = -sinθ _Y y _W + cos θ _Y z _W.
Around the Z _W axis: x _A = cosθ _Z x _W -sinθ _Z y _W , y _A = sinθ _Z y _W + cos θ _Z z _W , z _A = z _W.
Needless to say, the various conversions described above may be combined.
The coordinates P _A (x _A , y _A , z _A ) determined in this way are used as new coordinates P _W in the modified world coordinate system, and the processing described below is executed.

2.8. Assumed viewpoint coordinate three-dimensional world coordinate system after view transformation modifications to system in _{(O W -X W Y W Z} W), the viewpoint P _F shown in FIG. _{9 (a) (x F,} y F, z F) a To do. Viewpoint P _F and _{(x F, y F, z} F) the origin O _e, direction of the sight line, i.e., the origin O _e and the point _{P W (x W, y W} , z W) and a Z _e axis line connecting the The coordinate system of the left-handed system is called the viewpoint coordinate system (O _e −X _e Y _e Z _e ). In this embodiment, X _e axis shall take parallel to X _W Z _W surface.

Here, the angle formed between the X _W axis and X _e axis is alpha, the angle between the Y _W-axis and Y _e axis and beta. At this time, if the coordinates of the object represented in the world coordinate system (O _W -X _W Y _W Z _W ) are P _W (x _W , y _W , z _W ), the coordinates P _W are represented by the viewpoint coordinate system (O _The coordinates P _e (x _e , y _e , z _e ) converted into _e− X _e Y _e Z _e ) are expressed by the following formulas (note that the following formulas are homogeneous coordinates commonly used in image processing): Expressed by the system).

Accordingly, the coordinates P _e (x _e , y _e , z _e ) are obtained by the following expression.
x _e = (x _W -x _F ) cosα + (z _W -z _F ) sinα,
y _e = (x _W -x _F ) sinαsinβ + (y _W -y _F ) cosβ- (z _W -z _F ) cosαsinβ,
z _e = (x _W -x _F ) sinαcosβ- (y _W -y _F ) sinβ- (z _W -z _F ) cosαcosβ.

2.9. Spotlight / Focus Processing In this embodiment, processing such as shading can be performed on an object (part) placed in the world coordinate system. In order to perform a display as focused by a spotlight, a predetermined light source is assumed as shown in FIG. 9B, and a density for displaying each object is assigned according to the amount of light emitted from the light source. The display density can be controlled. In particular, when no shading effect is applied, a uniform density is assigned to all objects.

The shading process may be performed after the object coordinate system is converted to the viewpoint coordinate system, or after other hidden processes such as hidden line processing and hidden surface processing, but in this embodiment, shading is performed in the world coordinate system. Processing is to be performed. In the world coordinate system, the coordinates of the objects constituting the musical score and the like are P _O and the coordinates of the light source are P _L. The light source is a point light source such as a spotlight and has directivity in the focal coordinate P _B direction. A straight line connecting the light source coordinates P _L and the focus coordinates P _B is called an “optical axis”. The light source coordinates P _L and the focus coordinates P _B can be appropriately set by a user operation. Here, display density I _O for the object coordinates P _O is obtained by the following equation.

However, the meaning of each variable in the above equation is as follows.
I _L : intensity of the light source in the optical axis direction (constant or user specified)
θ: Angle formed by the optical axis and the straight line P _O P _L
n: Coefficient indicating directivity strength (may be a constant or specified by the user)
φ: Angle formed by the normal f of each layer and the straight line P _O P _L.
Here, the direction of the normal f of the layer is uniquely determined by the “layer direction” described above. In the example of FIG. 9B, the direction of the normal line f is equal to the Z _W axis because the direction of the H axis is the same as the Y _W axis.
r: distance between the coordinate P _O and coordinate P _L
Kd: diffuse reflection coefficient (constant)

The above equation is based on Lambert Model diffuse reflection. In the above formula, specular reflection, ambient light, atmospheric attenuation, and the like are omitted, but they may be added. The light source is a point light source, but may be a line light source or a surface light source.
As described above, the corresponding display density I _O is assigned to the coordinates P _O of all the objects (parts) in the world coordinate system. Thereby, each object is displayed with the display density I _O of the assigned display. Here, when the display density _IO is large, the object is displayed dark and opaque. When the display density _IO is small, the object is light and displayed in a color close to the background color. Further, when viewed from the viewpoint, the object (part) in the back can be seen through. There may be an implementation in which the coordinates P _L of the light source coincide with the coordinates P _{F of} the viewpoint.

2.10. Updating the Current Position In this embodiment, not only the score is displayed statically, but also the score display is dynamically changed according to the elapsed time. For example, the current position, that is, the current position in the performance information is updated based on the tempo information in the performance information and the elapsed time, and accordingly, the object (part constituting the musical score), the viewpoint coordinates P _F , the light source are the coordinates P _L etc. are updated, as a result, the musical score to be displayed is updated based on the event data in the current position, the tone generating and performance guide, etc. is executed at the same time.

2.10.1. Updating the coordinates of the object The origin O of the original coordinate system described above is the beginning position of the performance information in the initial state, but moves with the current position when the reproduction of the performance information is started. That is, the current position at that time becomes the origin O in the original coordinate system. In other words, in the above description, the position of the object Q on a certain layer in the original coordinate system is (t _q , d _i , h _q ), but the elapsed time from the playback start time of the performance information is t. Then, the coordinates of the object Q are (t _q -t, d _i , h _q ). The state is shown in FIG.

As described above, when the coordinates of the object Q in the original coordinate system are updated, the coordinates of each object Q are also updated in the world coordinate system and the viewpoint coordinate system as shown in FIG. . At this time, by fixing the viewpoint coordinates P _F in the world coordinate system (kept constant the relative positional relationship between the viewpoint coordinate P _F and the origin O _W), in the final display image, always The current position of the performance will be displayed. However, the coordinates of all objects Q are not set to (t _q -t, d _i , h _q ), but clef symbols (g clef etc.), key signatures (# etc.), time signatures (4 The coordinates of / 4) are preferably fixed at (t _q , d _i , h _q ). Also, an offset may be provided as appropriate, for example, the origin is two bars before the current position.

2.10.2. Updating the position of the viewpoint and light source As described above, if the coordinates of the object Q are set to (t _q -t, d _i , h _q ), the object moves according to the change of the current position. The position coordinates are fixed, and the viewpoint coordinates P _F , the light source coordinates P _L, or the focus coordinates P _B are changed according to the current position so that the current performance location is highlighted on the touch panel 6. Also good. Here, as one example, previously described as an example of moving along the view point coordinates P _F to the current position.

First, in the original coordinate system, to determine the depth value d _O as a reference. This depth value d _O is, for example, a depth value d _i assigned to a part to be particularly watched. Next, as shown in FIG. 11 (a), in the curved surface S (t, d) of the world coordinate system, the curve corresponding to the depth value d _O is C (t) = S (t, d _O ). . Here, as a function of time elapsed viewpoint coordinates P _F i.e. current position t, it can be expressed as "P _F (t)". Here, the viewpoint coordinates _{P F (t) = P F} (0) + - and {C (t) C (0 )}. However, P _F (0) is the viewpoint coordinate P _F when “current position t = 0”, and the curve C (0) is a point on the curve C (t) at that timing.

Thus, as shown in FIG. 11 (b), in accordance with the updating of the current position, the viewpoint coordinates P _F is updated in the world coordinate system, the updated content is reflected in the final display. In particular, the present embodiment is characterized in that the part related to the depth value d _O can be specified, so that the part to be watched is displayed easily.

Method of moving the viewpoint coordinates P _F may be another method is one example. For example, instead of the curve C (t), a straight line L (t) obtained by projecting the curve C (t) in an arbitrary direction (for example, the X _W Y _W plane direction) (see FIG. 11A). may move the viewpoint coordinates P _F along. In this case, the viewpoint coordinates _{P F (t) = P F} (0) + - becomes {L (t) L (0 )}.

Further, not only the viewpoint coordinates P _F are moved according to the current position t, but the light source coordinates P _L may be moved as shown in FIG. 11B, and the viewpoint coordinates P _F and the light source coordinates P _L are moved. May be combined. Further, for example, the viewpoint coordinates P _F is fixed in a position such that the overhead of each layer, by moving only the source coordinates P _L, may be displayed in display density determined by the position of each object sources .

2.10.3. Viewpoint, updating also the orientation of the light sources, the coordinates of the object Q, while fixing the viewpoint coordinates P _F and the light source coordinates P _L, to change the line-of-sight direction, focal coordinates P _B in the viewpoint coordinate system according to the current position t It may be. That is, the viewpoint coordinates P _F and the line connecting the point on the curve C (t) at the current time is "line of sight", the angle as shown in FIG. 11 (c) alpha, function of the β current position t alpha (t ), and beta (t), it may be set to so as to overlap and the Z _e axis of the visual axis and the viewpoint coordinate system α and (t) beta and (t).

At the same time, the focal point at the coordinate P _B may be changed according to the current position t so as to follow the curve C (t). Here, the viewpoint coordinates P _F, that a light source coordinates P _L match. In this case, as shown in FIG. 11D, the angle φ formed by the normal line f of the layer and the straight line P _O P _L at the object coordinate P _O , the angle θ formed by the optical axis and the straight line P _O P _L , and the object The distances r between the coordinates P _O and the light source coordinates P _L are φ (t), θ (t), and r (t), which are functions of the current position t, respectively. As a result, the display density _IO assigned to the object is expressed by the following equation.

2.11. Generation of Display Screen Once the coordinates of each object in the viewpoint coordinate system and the display density _IO are determined, the generated image is then displayed on the touch panel 6 through processes such as clipping, perspective conversion, and viewport conversion. The Note that these processes are known in computer graphics, and these known processes may be applied in the present embodiment.
The perspective transformation may be not only perspective projection with perspective, but also parallel projection (also called orthogonal projection or orthographic projection).

3． Next, the operation of this embodiment will be described.
When a predetermined operation is performed on the portable information terminal, the processing programs shown in FIGS. 12 and 13 are activated. In the figure, when the process proceeds to step SP2, a predetermined initialization process is performed. Next, when the process proceeds to step SP4, it is determined whether or not an operation for changing the performance state (stopped or being performed) has been performed in the operator section 2. If "YES" is determined here, the process proceeds to step SP6, and the performance state (stopped or being performed) is changed. That is, if it is “stopped”, it is changed to “playing”, and if it is “playing”, it is changed to “stopped”.

Next, when the process proceeds to step SP8, it is determined whether or not the performance information is “playing”. If "YES" is determined here, the process proceeds to step SP16, the performance position, that is, the current position t is advanced based on the time measurement result of the timer 24, and the event data at the current position t is read out. Next, when the process proceeds to step SP18, the layer position in the original coordinate system is moved according to the current position t, and the coordinates of each object are changed accordingly. Further, the viewpoint coordinates P _F , the light source coordinates P _{L and the} like are moved as necessary. Next, when the process proceeds to step SP19, a sound generation process is performed based on the MIDI event at the current position t. The sound generation process is a process in which the CPU 22 generates a musical sound according to a MIDI event. This is a well-known process, and the process may be applied also in the present embodiment. In this embodiment, the sound generation process is executed by the CPU 22, but a known sound source circuit may be provided and the sound generation process may be performed by the sound source circuit. If the performance state is not “playing”, the processes of steps SP16 to SP19 are skipped.

Next, in steps SP10, SP12, SP14, SP28, SP30, and SP32, it is detected whether or not various display parameter changing operations have been performed by the operator section 2, and if detected, corresponding processing is performed. Executed. Hereinafter, the contents of these processes will be described. First, when an operation for changing the curved surface S (t, d), for example, dragging of the control point _Pij is detected, the process proceeds to step SP20 via step SP10. Here, the curved surface information specifying the curved surface S (t, d) in the world coordinate system is updated. Next, when the process proceeds to step SP22, position information representing the position of each layer on the curved surface S (t, d) is updated according to the updated curved surface information.

When a moving operation for moving the position of any layer is executed, the process proceeds to step SP24 via step SP12, and the position of the layer is changed in the original coordinate system. Further, when the operation for changing the viewpoint coordinates P _F or line-of-sight direction is detected, the process proceeds to step SP26 via step SP14, the viewpoint information for specifying the viewpoint coordinates P _F and line-of-sight direction in the viewpoint coordinate system Updated.

If an operation for changing the focal coordinate P _B is detected, the process proceeds to step SP 34 via step SP 28, and the focal information specifying the focal coordinate P _B in the viewpoint coordinate system is updated. When an operation for changing the display magnification (or setting for wide-angle display) is detected, the process proceeds to step SP36 via step SP30, and the setting of the wide-angle display or the change of the display magnification is performed. When another change operation is detected, the process proceeds to step SP38 via step SP32, and a parameter change process necessary to change the display is performed according to the detected operation.

  As described above, when the display parameter is updated as necessary, the process proceeds to step SP40, and the display process is performed. That is, each object is arranged in the original coordinate system based on the display parameters changed as necessary, and mapping in the world coordinate system modified world coordinate system and the viewpoint coordinate system is sequentially obtained. Next, after spotlight / focus processing is performed as necessary, the generated image is displayed on the touch panel 6 through processing such as clipping, perspective conversion, and viewport conversion. And a process returns to step SP4 and the process after the step SP4 is repeated.

Next, various display examples according to display parameters will be described with reference to FIGS. First, FIG. 14A is a display example in which layers of “3” parts are arranged at equal intervals. For this FIG. 14 (a), the shows a display example of changing the viewpoint coordinates P _F and line of sight in FIG. 14 (b). FIG. 14C is a display example in which the position of the layer in the depth direction is changed with respect to FIG. 14A to change the layer interval. FIG. 14D is a display example of FIG. It is the example of a display which expanded a part. FIG. 15A is a display example when the curved surface S (t, d) is distorted with respect to FIG. FIG. 15B is a display example when the time axis is partly shortened (partial intervals of the control points _{Pij in} FIG. 6 are narrowed) compared to FIG. In the illustrated example, the bar interval of the right side portion of the layer located at the farthest is displayed narrowly.

FIG. 15C is a display example in which a specific portion is focused and spotlight / focus processing is performed. In the example shown in the figure, it is assumed that the focal coordinate P _B is in the second layer from the front, and the display density of the layer in the foreground is lighter. FIG. 4D shows a display example when the layer is constituted by a grand staff.

Four. Modifications The present invention is not limited to the above-described embodiments, and various modifications can be made as follows, for example.
(1) In the above embodiment, the score is displayed by an application program running on the portable information terminal, but only this application program is stored in a recording medium such as a CD-ROM or a memory card and distributed. Alternatively, it may be distributed through a transmission line. Further, the application program is not limited to that executed in the portable information terminal, and it goes without saying that a similar program can be executed in a normal personal computer, electronic musical instrument, or the like.

(2) In the above embodiment, the logical musical score information for specifying the musical score is described by the system exclusive message or the meta event. However, the logical musical score information is automatically generated based on the MIDI event. Also good.

(3) In the above embodiment, the MIDI event of each part is recorded on each track of the performance information, and these tracks are associated with each layer. However, the relationship between the track, the part and the layer is a pair. There is no need to be one. For example, logical score information of a plurality of parts may be included in one track. Thereby, the score of a plurality of parts can be displayed on one layer.

  In addition, one track does not necessarily correspond to one layer. A multi-track score may be displayed on one layer. Conversely, the score of one track may be divided and displayed in a plurality of layers. In short, since it is only necessary to specify the layer that should contain the logical score information for each piece of logical score information, it is possible to specify the layer by various methods other than those exemplified in the above embodiment.

(4) In the above embodiment, the object is generated based on the logic score information. However, the X _W axis, Y _W axis, Z _W axis and the curved surface S (t, d) of the world coordinate system are As an object in the coordinate system, it may be displayed on the touch panel 6 so as to be visible to the user. For example, the X _W axis, Y _W axis, and Z _W axis may be displayed as graphics such as arrows, and the curved surface S (t, d) may be displayed as an opaque or translucent curved surface. Further, it is preferable that the user can specify whether or not to display these objects.

Further, according to circumstances, X _W axis, Y _W-axis, Z _W-axis and the curved surface S (t, d) may automatically determine whether to display the object, such as. For example, when performing an operation to drag the object, such as a note, X _W axis, Y _W-axis, automatically displays Z _W-axis and the curved surface S (t, d) objects, such as, not displayed when otherwise You may do it. In this way, when an operation by the user is performed, a state in which the direction and amount are reflected in the world coordinate system is displayed, so that the user can move the object or curved surface S (t, d) on the world coordinate system. Since the state can be grasped intuitively, operability can be improved.

It is a block diagram of the portable information terminal of one Example of this invention. It is a figure which shows the data structure of the performance information of one Example. It is a figure which shows the structure of a layer. It is operation | movement explanatory drawing of one Example. It is other operation | movement explanatory drawing of one Example. It is other operation | movement explanatory drawing of one Example. It is other operation | movement explanatory drawing of one Example. It is other operation | movement explanatory drawing of one Example. It is other operation | movement explanatory drawing of one Example. It is other operation | movement explanatory drawing of one Example. It is other operation | movement explanatory drawing of one Example. It is a flowchart (1/2) of the processing program of one Example. It is a flowchart (2/2) of the processing program of one Example. 6 is a diagram showing a display example on the touch panel 6. FIG. FIG. 10 is a diagram illustrating another display example on the touch panel 6.

Explanation of symbols

  2: operation unit (curved surface designating means, viewpoint designating means), 4: detection circuit, 6: touch panel, 8: display circuit (display means), 10: detection circuit (curved surface designating means, viewpoint designating means), 12: audio Interface section, 14: Communication interface section, 16: Bus, 18: RAM, 20: Flash memory (storage means), 22: CPU, 24: Timer, 26: Recording media slot, 30: Header chunk, 32,. : Track chunk, 34, ..., 34: Timing data, 36, ..., 36: Event data.

Claims (6)

Storage means for storing performance information including object specifying information, the object specifying information specifying the object and a layer to which the object belongs for each of a plurality of objects constituting a display screen for displaying the performance information. A storage means,
A curved surface designating means for designating a curved surface for arranging each layer in a virtual three-dimensional space consisting of time, height and depth;
Viewpoint designation means for designating a virtual viewpoint for viewing each layer and its direction in the three-dimensional space;
Display means for displaying a virtual image viewed from the viewpoint ;
On the three-dimensional space, a light source coordinates are coordinates of a virtual light source, a light source specifying means for specifying the focal coordinate a coordinate of the optical axis of the light source is directed,
Playing and having a <br/> display density setting means for the setting and the straight line connecting the respective object with the light source coordinates, the display density before Symbol each object in accordance with the angle between the optical axis forms Information display device.   2. The performance information display device according to claim 1, wherein the curved surface designating means includes means for deforming a currently applied curved surface.   3. The performance information display device according to claim 1, further comprising a depth changing means for changing the depth of each layer in the three-dimensional space. A time position acquisition means for acquiring a time position of automatic performance;
It further has a moving means for moving any one of “the position and direction of the viewpoint”, “the position and direction of the light source”, and “the position of the object on the layer” according to the temporal position. The performance information display device according to any one of claims 1 to 3 , wherein Performance information display device according to any one of claims 1 to 4, further comprising a means for setting a display magnification for the image to be displayed on said display means. In the processing equipment,
A procedure for reading performance information including object specifying information from the storage means , and the object specifying information includes the object and a layer to which the object belongs for each of a plurality of objects constituting a display screen for displaying the performance information. all SANYO to identify,
A procedure for designating a curved surface for arranging each layer in a virtual three-dimensional space consisting of time, height and depth;
In the three-dimensional space, a procedure for designating a virtual viewpoint for viewing each layer and its orientation;
A procedure for displaying a virtual image viewed from the viewpoint;
On the three-dimensional space, a light source coordinates are coordinates of a virtual light source, a step of specifying a focus coordinate a coordinate of the optical axis of the light source is directed,
A straight line connecting the each object and the light source coordinates, the procedures for setting the display density before Symbol each object in accordance with the angle between the optical axis
Program for the execution.