JP6586847B2 - Music score image analyzer - Google Patents

Description

  The present invention relates to a technique for photographing a musical score.

  Techniques for photographing books have been proposed. For example, Patent Document 1 discloses a technique for correcting the distortion of a book image taken by a user using an imaging device such as a digital camera.

JP 2013-93704 A

  If the technique disclosed in Patent Document 1 is used, it is possible to correct the distortion of the captured image. However, it is essential that the captured image is originally captured from an appropriate imaging direction in which the distortion of the captured image is reduced. However, the user adjusts the shooting direction of the shooting device while visually checking the distortion of the book displayed on a monitor such as a digital camera, and gives a shooting instruction while maintaining the shooting device in an appropriate shooting direction. It's not really easy to do. In view of the above circumstances, an object of the present invention is to obtain an image in which a score is photographed from an appropriate photographing direction without a user instructing photographing.

  In order to solve the above-described problems, a score image analysis apparatus according to the present invention uses a photographing device to generate a score including a plurality of staves arranged in parallel to each other and score symbols arranged at predetermined positions with respect to each staff. A symbol detection unit that detects a musical score symbol from each score image generated in time series in moving image shooting, a symbol string identification unit that identifies a symbol string that is an array of a plurality of musical score symbols detected by the symbol detection unit, and a score An index calculation unit that calculates a posture index, which is an index of suitability of the shooting direction, according to a symbol string, and a shooting control unit that instructs the shooting device to shoot a still image of a score image when the posture index satisfies a predetermined condition It has. With the above configuration, the shooting control unit instructs the shooting device to take a still image when the posture index, which is an index of the appropriateness of the shooting direction with respect to the score, satisfies a predetermined condition. Therefore, it is possible to acquire a score image photographed in an appropriate photographing direction without the user instructing photographing.

  In a preferred aspect of the present invention, the index calculation unit includes a unit region setting unit that sets a unit region that intersects the symbol string at the position of the musical score symbol, and a ratio included in the unit region in the staff in which the musical score symbol is arranged. And a first index calculation unit that calculates a corresponding first index, and sets an attitude index corresponding to the first index. In the above configuration, a unit region that intersects a symbol string that is an array of a plurality of musical score symbols is set, and a first index corresponding to the proportion of the staff included in the unit region is calculated and reflected in the posture index. Therefore, a posture index that quantitatively evaluates the suitability of the shooting direction with respect to the score can be applied to a still image shooting instruction.

  In a preferred aspect of the present invention, the index calculation unit includes a second index calculation unit that calculates a second index corresponding to the angle of the symbol string, and sets a posture index corresponding to the second index. In the above configuration, the second index corresponding to the angle of the symbol string is calculated and reflected in the posture index. Therefore, a posture index that quantitatively evaluates the suitability of the shooting direction with respect to the score can be applied to a still image shooting instruction.

  In a preferred aspect of the present invention, the posture index is an index that becomes a larger value as the horizontal direction of the shooting range is closer to the horizontal direction of the score, and the predetermined condition is that the posture index has a predetermined threshold value. It is to exceed. In the above configuration, the posture control index becomes a larger value as the horizontal direction of the shooting range is closer to the horizontal direction of the score, and when the posture index exceeds a predetermined threshold, the shooting control unit Instructs the photographing apparatus to capture a still image of a score image. Therefore, it is possible to acquire a score image shot in an appropriate shooting direction close to a state in which the horizontal direction of the shooting range is parallel to the horizontal direction of the score.

It is a lineblock diagram of the musical score image analysis device of a 1st embodiment. It is a display example of a score image when the shooting direction is parallel to the normal direction of the score. It is a display example of a score image when the shooting direction is inclined with respect to the normal direction of the score. It is explanatory drawing of the process of a symbol detection part. It is explanatory drawing of the process of a symbol string specific | specification part. It is explanatory drawing of the process of a unit area | region setting part. It is a flowchart of the process in which a 1st parameter | index calculation part calculates a 1st parameter | index. It is explanatory drawing of a process of a 1st parameter | index calculation part. It is a flowchart of the process in which a 2nd parameter | index calculation part calculates a 2nd parameter | index. It is an enlarged view of a symbol string figure for explaining a vector. It is the example of a display on the display apparatus of a score image. It is a flowchart of the process which the control apparatus of 1st Embodiment analyzes a score image. It is a block diagram of the musical score image analysis apparatus of 2nd Embodiment. It is a flowchart of the process which the control apparatus of 2nd Embodiment analyzes a score image.

<First Embodiment>
FIG. 1 is a configuration diagram of a score image analyzing apparatus 100 according to the first embodiment. The score image analysis apparatus 100 is an apparatus that analyzes an image of a score photographed by a user, and is realized by a portable device such as a smartphone or a tablet. As illustrated in FIG. 1, the score image analysis device 100 according to the first embodiment includes a photographing device 10, an input device 20, a storage device 30, a display device 40, and a control device 50.

  The photographing apparatus 10 is an apparatus for photographing an image, and includes an image sensor that converts incoming light from a subject into an electrical signal. A user photographs a musical score while holding the photographing device 10 in a desired position and direction with respect to the musical score described on a printed matter or the like. The photographing apparatus 10 generates and outputs a score image (hereinafter referred to as a “score image”) 70 within a predetermined range (hereinafter referred to as a “shooting range”). The photographing apparatus 10 according to the first embodiment can perform moving image photographing and still image photographing. Specifically, the photographing apparatus 10 generates a plurality of musical score images 70 in a time series at a predetermined cycle when shooting a moving image, and generates a single musical score image 70 when shooting a still image. As illustrated in FIG. 2, in the following description, the horizontal direction (horizontal direction of the shooting range) of the score image 70 is expressed as the x-axis direction, and the vertical direction (vertical direction of the shooting range) is expressed as the y-axis direction. . In the first embodiment, a configuration in which the photographing apparatus 10 is integrally mounted on the score image analysis apparatus 100 is illustrated.

  As illustrated in FIG. 2, the score includes a set of a plurality of straight lines (hereinafter referred to as “staff”) 62 (1) to 62 (N) and each staff 62 (n) (n = 1 to 1). A plurality (N sets) of musical score symbols 72 (n) arranged at predetermined positions with respect to N) are juxtaposed in the vertical direction. The staff 62 (n) of the first embodiment is a staff score composed of five straight lines representing different pitches. Codes such as notes, rests, and change symbols representing the music are arranged at arbitrary positions with respect to the staff 62 (n). In the first embodiment, a treble clef and a syllabary symbol located near the left end with respect to the musical score staff 62 (n) are exemplified as a musical score symbol 72 (n).

  A user holds and shoots the photographing apparatus 10 (the musical score image analyzing apparatus 100) so that the entire musical score (for one page) is included in the photographing range. Specifically, the user has a state in which the shooting direction is parallel to the normal direction of the score and the x-axis direction of the score image 70 is parallel to the horizontal direction of the score (hereinafter referred to as “reference state”). Adjust the shooting direction so that In the reference state, as illustrated in FIG. 2, a plurality of score symbols 72 (1) to 72 (N) are arranged in parallel in the y-axis direction as indicated by a broken line 15, and a plurality of staffs 62 (n). Are parallel to the x-axis direction. However, in practice, it is not easy for the user to continuously hold the photographing apparatus 10 in the reference state, and the photographing direction is inclined with respect to the normal direction of the musical score, or the photographing range is a musical score method. It can be a state rotated from the reference state with the line direction as an axis (a state in which the x-axis direction of the score image 70 is not parallel to the horizontal direction of the score). As described above, in the state where the shooting direction is inclined (hereinafter referred to as “tilt state”) and the state where the shooting range is rotated (hereinafter referred to as “rotation state”), as illustrated by the broken line 15 in FIG. There may be a case where the arrangement of the plurality of score symbols 72 (1) to 72 (N) is not parallel to the y-axis direction.

  The display device 40 of FIG. 1 is a liquid crystal display panel, for example, and displays an image instructed from the control device 50. For example, a plurality of score images 70 generated by moving image shooting by the shooting device 10 are sequentially displayed (live view display) on the display device 40. The input device 20 is an operation device operated by a user for various instructions to the score image analysis device 100, and includes, for example, a plurality of operators operated by the user. Specifically, the user can instruct the score image analysis apparatus 100 to shoot moving images or still images by appropriately operating the input device 20. A touch panel configured integrally with the display device 40 may be employed as the input device 20.

  The storage device 30 is a known recording medium such as a magnetic recording medium or a semiconductor recording medium, and stores a program executed by the control device 50 and various data used by the control device 50. For example, a score image 70A of a still image generated by still image shooting by the shooting device 10 is stored in the storage device 30.

  The control device 50 is configured by a processing device such as a CPU (Central Processing Unit), for example, and executes a program stored in the storage device 30 to execute a musical score image photographed by the photographing device 10 as illustrated in FIG. 70, a plurality of elements (symbol detection unit 510, symbol string identification unit 520, unit region setting unit 532, first index calculation unit 534, second index calculation unit 544, posture index calculation unit 550, display control unit 560, It functions as a photographing control unit 570). A configuration in which a part of the function of the control device 50 is realized by a dedicated electronic circuit, or a configuration in which the function of the control device 50 is distributed to a plurality of devices may be employed.

  Each element realized by the control device 50 will be described in detail. The processing by each element described below is actually executed sequentially for each score image 70 in parallel with the moving image shooting by the shooting apparatus 10 (details will be described later). Focus on the score image 70.

  The symbol detection unit 510 detects a plurality (N) of musical score symbols 72 (1) to 72 (N) from each musical score image 70 generated by the photographing apparatus 10 in time series during moving image shooting. For the detection of the musical score symbol 72 (n) by the symbol detection unit 510, a known image recognition (for example, pattern recognition) technique can be arbitrarily employed. FIG. 4 is an explanatory diagram of processing of the symbol detection unit 510 for the musical score image 70 of the first embodiment. As illustrated in FIG. 4, the symbol detection unit 510 sets the symbol region 76 (n) so as to include each score symbol 72 (n) detected from the score image 70. The symbol area 76 (n) is a rectangular area composed of a straight line parallel to the x-axis and a straight line parallel to the y-axis.

  The user shoots such that the musical score symbols 72 (1) to 72 (N) are near the left end with respect to the staff 62 (n) and that the entire musical score (one page) is included in the shooting range. Considering this, it is highly possible that the N score symbols 72 (1) to 72 (N) are located in a vertical region located on the left side in the score image 70. Therefore, as illustrated in FIG. 4, the symbol detection unit 510 according to the first embodiment targets within a predetermined range 74 in which the score symbols 72 (1) to 72 (N) are expected to be located in the score image 70. The musical score symbols 72 (1) to 72 (N) are detected as follows. That is, the area outside the range 74 is excluded from the search range of the musical score symbol 72 (n). The range 74 is a short-shaped area that is formed by a straight line parallel to the x-axis and a straight line parallel to the y-axis at the left end of the score image 70 and is long in the y-axis direction. It is also possible for the user to adjust the shooting range (distance between the score and the shooting device 10) by the shooting device 10 so that the score 74 includes score symbols 72 (1) to 72 (N). . As described above, in the first embodiment, the musical score symbol 72 (n) is searched for within the predetermined range 74 of the musical score image 70 while the musical score symbol 72 (n) is searched outside the predetermined range 74. Absent. Therefore, the processing load applied to the symbol detection unit 510 (and thus the processing load of the entire score image analysis apparatus 100) is reduced compared with the configuration in which the entire range of the score image 70 is the search target of the score symbol 72 (n). Is possible.

  The symbol string specifying unit 520 in FIG. 1 specifies an array of N score symbols 72 (1) to 72 (N) detected by the symbol detecting unit 510 (hereinafter referred to as “symbol string”). FIG. 5 is an explanatory diagram of processing of the symbol string specifying unit 520 for the musical score image 70 according to the first embodiment. As illustrated in FIG. 5, the symbol string specifying unit 520 of the first embodiment sets a symbol string graphic 78 representing a symbol string. Specifically, the symbol string specifying unit 520 sets a rectangular symbol string figure 78 that includes each vertex of the N symbol areas 76 (1) to 76 (N) and has the smallest area. To do. As described above, the symbol string is inclined with respect to the x-axis direction and the y-axis direction in the inclined state and the rotated state. Therefore, as understood from FIG. 5, the symbol string figure 78 specified by the symbol string specifying unit 520 is in the x-axis direction in accordance with the inclination of the shooting direction with respect to the normal direction of the score and the degree of rotation of the shooting range with respect to the score. And can be inclined with respect to the y-axis direction.

  The unit area setting unit 532 in FIG. 1 selects N unit areas A (1) to A (N) corresponding to each score symbol 72 (n) detected by the symbol detection unit 510, as illustrated in FIG. Set. A unit area A (n) corresponding to an arbitrary musical score symbol 72 (n) has a rectangular shape that intersects the symbol string figure 78 identified by the symbol string identifying unit 520 at the position of the musical score symbol 72 (n). It is an area. Specifically, the unit area A (n) is a rectangular area extending in a direction orthogonal to the long side of the symbol string figure 78 with the musical score symbol 72 (n) as an end point. The width of the unit area A (n) (size in the direction orthogonal to the symbol string figure 78) is set to a predetermined ratio (for example, 95%) with respect to the width of the score image 70, and the vertical length of the unit area A (n). The width (size in the direction parallel to the symbol string figure 78) is set to a predetermined ratio (for example, 120 to 130%) with respect to the vertical width of the symbol area 76 (n). As understood from FIG. 6, when the symbol string figure 78 is inclined, the unit area A (n) is similarly inclined with respect to the x-axis direction and the y-axis direction.

  The first index calculation unit 534 in FIG. 1 calculates the first index X1. The first index X1 is an index of the degree to which the shooting direction is inclined with respect to the normal direction of the score. Specifically, the first index calculation unit 534 has a staff 62 in which score symbols 72 (n) corresponding to the unit region A (n) are arranged for the unit region A (n) set by the unit region setting unit 532. The ratio R (n) included in the unit area A (n) in (n) is calculated, and according to the ratios R (1) to R (N) in the N unit areas A (1) to A (N). The first index X1 is calculated.

  FIG. 7 is a flowchart of a process S5 (hereinafter referred to as “first index calculation process”) S5 in which the first index calculation unit 534 of the first embodiment calculates the first index X1, and FIG. 8 illustrates the first index calculation process. It is explanatory drawing of S5. The first index calculation process S5 is started when the unit area setting unit 532 sets the unit area A (n).

  When the first index calculation process S5 is started, the first index calculation unit 534 sets the attention area B in the score image 70 (S5-1). The attention area B is a rectangular area composed of a straight line parallel to the x-axis and a straight line parallel to the y-axis, as illustrated in FIG. Specifically, the horizontal width of the attention area B is set to a predetermined ratio (for example, 95%) with respect to the horizontal width of the score image 70, and the vertical width of the attention area B includes all vertices of the symbol string graphic 78. Set to size.

  The first index calculation unit 534 detects the straight line 63 (hereinafter referred to as “straight line detection”) for the inside of the attention area B set in step S5-1 in FIG. 7 in the score image 70 (S5-2). . Specifically, as illustrated in FIG. 8, straight line detection is performed in the attention area B of the score image 70, and a plurality of figures including the staffs 62 (1) to 62 (N) in the score image 70 are detected. A straight line 63 is detected. For the straight line detection, a known feature detection (for example, Hough transform) process may be arbitrarily employed. Note that the outside of the attention area B is not likely to be a score, and is therefore excluded from the line detection target by the first index calculation unit 534.

  The first index calculation unit 534 classifies the plurality of straight lines 63 detected in step S5-2 of FIG. 7 into a plurality of sets 64 (n) for each staff 62 (n) (for each score symbol 72 (n)) ( S5-3). A known clustering technique (for example, k-means method) can be arbitrarily adopted for the classification of the straight line 63. Specifically, the first index calculation unit 534 classifies the plurality of straight lines 63 detected from the score image 70 into the same number of sets 64 (n) as the score symbols 72 (n) detected by the symbol detection unit 510. For example, a plurality of straight lines 63 can be classified into N sets 64 (n) by clustering in which the number N of musical score symbols 72 (n) is designated as the classification number. Further, as described above, the staff 62 (n) is a set of a plurality of straight lines parallel to each other. Therefore, the plurality of straight lines 63 detected from any one staff 62 (n) are close to each other. Distributed in position. Therefore, as illustrated in FIG. 8, the plurality of straight lines 63 detected from the staff 62 (n) in which the musical score symbol 72 (n) is arranged can be classified into the same set 64 (n). Therefore, for example, even when one unit area A (n) is inclined with respect to the x-axis direction so as to straddle a plurality of staves 62 (n), the staff 62 (n) included in the unit area A (n) The ratio R (n) can be specified with high accuracy for each staff 62 (n). Further, since the plurality of straight lines 63 detected from the plurality of staffs 62 (n) are classified into the same number of sets 64 (n) as the score symbols 72 (n), the number of classifications of the plurality of straight lines 63 is unknown. There is an advantage that a plurality of straight lines 63 can be appropriately classified.

  For each of the N unit areas A (1) to A (N), the first index calculation unit 534 includes a plurality of straight lines 63 classified into the set 64 (n) in step S5-3 in FIG. The ratio R (n) included in the region A (n) is calculated (S5-4). Specifically, the ratio R () of the number of points included in the unit area A (n) among the points (for example, end points and midpoints) on each line of the plurality of straight lines 63 classified into the set 64 (n). n) is calculated. As described above, according to the configuration in which the points on each straight line 63 are counted, for example, the ratio R (compared with the configuration in which the ratio of the length of each straight line 63 located in the unit area A (n) is calculated. There is an advantage that the calculation of n) is easy.

  In a reference state in which the shooting direction is parallel to the normal direction of the score, each staff 62 (n) extends in the x-axis direction, while the symbol string figure 78 is substantially parallel to the y-axis direction, so that each unit area A (n) extends in a direction substantially parallel to the x-axis direction (that is, a direction substantially parallel to the staff 62 (n)). Accordingly, since most of the straight lines 63 classified into the set 64 (n) are included in the unit area A (n), the ratio R (n) is a large value. On the other hand, in the inclined state in which the shooting direction is inclined with respect to the normal direction of the score, for example, as shown in FIG. 6, each staff 62 (n) extends in the x-axis direction, while the symbol string figure 78 is displayed. By inclining in the x-axis and y-axis directions, each unit region A (n) also extends in a direction inclined with respect to the x-axis and y-axis directions. Therefore, as illustrated in FIG. 8, among the straight lines 63 classified into the set 64 (n), the straight lines 63 that are not included in the unit area A (n) increase, and the ratio R (n) is a small value. . That is, the ratio R (n) can be used as an index of the degree to which the shooting direction is inclined with respect to the normal direction of the score.

  The first index calculation unit 534 calculates a first index X1 corresponding to each ratio R (n) in the N unit areas A (1) to A (N) calculated in step S5-4 in FIG. (S5-5). Specifically, the total or weighted sum of N ratios R (1) to R (N) is calculated as the first index X1. As described above, in the first embodiment, since the first index X1 is calculated according to the ratio R (n) over the N unit areas A (1) to A (N), the normal direction of the score in the shooting direction It is possible to comprehensively evaluate the degree of inclination with respect to a wide range of the score image 70. The above is a specific example of the first index calculation process S5.

  The second index calculation unit 544 in FIG. 1 calculates the second index X2. The second index X2 is an index of the inclination of the shooting direction with respect to the normal direction of the score and the degree of rotation of the shooting range with respect to the score. Specifically, the second index calculation unit 544 sets the second index X2 according to the angle of the symbol string (array of N score symbols 72 (1) to 72 (N)) specified by the symbol string specifying unit 520. Is calculated. FIG. 9 is a flowchart of a process S6 in which the second index calculation unit 544 of the first embodiment calculates the second index X2 (hereinafter referred to as “second index calculation process”). For example, the second index calculation process S6 is started when the symbol string specifying unit 520 specifies the symbol string.

  When the second index calculation process S6 is started, the second index calculation unit 544 obtains a vector C parallel to the symbol string specified by the symbol string specifying unit 520 (S6-1). Specifically, the symbol string specifying unit 520 obtains a vector C parallel to the straight line constituting the symbol string graphic 78. FIG. 10 is an enlarged view of the symbol string figure 78 for explaining the vector C. For example, a vertex C closest to the x axis among the vertices of the symbol string figure 78 is set as a starting point P, and a vector C that is parallel to the long side of the symbol string figure 78 and forms an angle θ with respect to the x axis is obtained. The vector C is a unit vector whose size (absolute value) is normalized to 1. The angle θ of the vector C can also be restated as the angle of the symbol string with respect to the x axis.

  The second index calculation unit 544 calculates the absolute value | Cy | (0 ≦ | Cy | ≦ 1) of the component Cy in the y-axis direction in the vector C as the second index X2 (S6-2). In the reference state, since the symbol string figure 78 is substantially parallel to the y-axis direction, the vector C is also substantially parallel to the y-axis. Therefore, the absolute value | Cy | of the component Cy of the vector C, that is, the second index X2 is a value close to 1 which is the maximum value. On the other hand, in the inclined state or the rotated state, the symbol string figure 78 is inclined with respect to the x-axis and y-axis directions. Therefore, the second index X2 is smaller than 1. As understood from the above description, the second index X2 is used as an index of the degree to which the shooting direction is inclined with respect to the normal direction of the score in the tilted state and the degree to which the shooting range is rotated from the reference state in the rotated state. Is possible.

  The posture index calculation unit 550 in FIG. 1 calculates the posture index Z. The posture index Z is an index indicating whether or not the posture of the user holding the photographing apparatus 10 (the photographing direction of the photographing apparatus 10) is close to the reference state. Specifically, posture index calculation unit 550 calculates posture index Z according to first index X1 calculated by first index calculation unit 534 and second index X2 calculated by second index calculation unit 544. For example, the product of the first index X1 and the second index X2 is calculated as the posture index Z. As described above, since both the first index X1 and the second index X2 are closer to the reference state (that is, closer to the state in which the x-axis direction of the shooting range is parallel to the horizontal direction of the score), the first index X1 takes a larger value. The posture index Z, which is the product of X1 and the second index X2, increases as the shooting direction approaches the reference state. That is, the posture index Z can be used as an index as to whether or not the shooting direction of the shooting apparatus 10 is close to the reference state.

  As understood from the above description, the unit area setting unit 532, the first index calculation unit 534, the second index calculation unit 544, and the posture index calculation unit 550 of FIG. 1 are indicators of the suitability of the shooting direction with respect to the score. It functions as an index calculation unit 530 that calculates the attitude index Z.

  The display control unit 560 of FIG. 1 displays the analysis result of the score image 70 on the display device 40. Specifically, as illustrated in FIG. 11, the display control unit 560 of the first embodiment includes a symbol string graphic 78 set by the symbol string specifying unit 520 and a reference graphic 80 serving as a reference for the symbol string graphic 78. The posture index Z calculated by the posture index calculation unit 550 is displayed on the display device 40. In the first embodiment, the reference graphic 80 is a rough shape of the range 74 to be searched for the musical score symbol 72 (n). Specifically, as illustrated in FIG. 11, the display control unit 560 causes the symbol string graphic 78 and the reference graphic 80 to be superimposed on the score image 70 and displayed on the display device 40. In addition, the posture index Z is displayed on the display device 40. The symbol string graphic 78, the reference graphic 80, and the posture index Z are displayed on the display device 40 in parallel with the moving image shooting of the score by the shooting device 10. Since the items displayed on the display device 40 by the display control unit 560 are arbitrary, for example, as illustrated in FIG. 11, in order to use the value (score) of the posture index Z as a guideline for the user to adjust, It is also possible to display a character string such as “Please increase the score”.

  The imaging control unit 570 in FIG. 1 controls the imaging operation of the imaging device 10. Specifically, the shooting control unit 570 of the first embodiment instructs the shooting device 10 to shoot a moving image when the user gives a moving image shooting instruction to the input device 20, and uses the still image shooting instruction. When the user gives the input device 20, the photographing device 10 is instructed to take a still image. The user can give a still image shooting instruction to the input device 20 at any time during moving image shooting. A still image score image 70 </ b> A generated by still image shooting is stored in the storage device 30 by the shooting control unit 570.

  FIG. 12 is a flowchart of processing (hereinafter referred to as “score image analysis processing”) in which the control device 50 according to the first embodiment analyzes the score image 70. The musical score image analysis process is started in response to an instruction for moving image shooting from the user. When the musical score image analysis process is started, the control device 50 sequentially acquires each of the plurality of musical score images 70 generated by the photographing device 10 through moving image photographing (S1).

  The symbol detector 510 detects a score symbol 72 (n) from the score image 70 generated by the photographing apparatus 10 (S2). The symbol string identification unit 520 identifies a symbol string (symbol string graphic 78) that is an array of a plurality of musical score symbols 72 (1) to 72 (N) detected by the symbol detection unit 510 (S3). The unit region setting unit 532 sets a unit region A (n) that intersects the symbol string specified by the symbol string specifying unit 520 at the position of the musical score symbol 72 (n) (S4). The first index calculation unit 534 calculates a ratio R (n) included in the unit area A (n) of the staff 62 (n) in which the musical score symbol 72 (n) is arranged, and N unit areas A (( A first index X1 corresponding to the ratios R (1) to R (N) in 1) to A (N) is calculated (S5). A specific example of the first index calculation process S5 is as described above with reference to FIG. As described above with reference to FIG. 9, the second index calculation unit 544 calculates the second index X2 corresponding to the angle θ of the symbol string specified by the symbol string specifying unit 520 (S6).

  The posture index calculation unit 550 calculates a posture index Z according to the first index X1 calculated by the first index calculation unit 534 and the second index X2 calculated by the second index calculation unit 544 (S7). The display control unit 560 shoots a musical score of the symbol string graphic 78 set by the symbol string specifying unit 520, the reference graphic 80 serving as a reference for the symbol string graphic 78, and the posture index Z calculated by the posture index calculation unit 550. At the same time, it is superimposed on the score image 70 and displayed on the display device 40 (S8).

  The user appropriately adjusts the direction of the photographing apparatus 10 while confirming the symbol string graphic 78 and the posture index Z displayed on the display device 40, and the state or posture index Z of the symbol string graphic 78 along the reference graphic 80 is determined. In a state where the numerical value is large, the input device 20 is operated to instruct still image shooting. The shooting control unit 570 determines whether an instruction for shooting a still image is given to the input device 20 by the user (S9). When still image shooting is instructed by the user (S9; YES), the shooting control unit 570 instructs the shooting device 10 to take a still image (S10), and the still image shot by the shooting device 10 according to the instruction. The score image 70A is stored in the storage device 30 (S11). On the other hand, when still image shooting is not instructed by the user (S9; NO), the processing of step S12 is executed without executing the processing of steps S10 and S11 as illustrated in FIG.

  When ending the moving image recording of the score (S12; YES), for example, when the user gives the input device 20 the end of moving image recording of the musical score, the musical score image analysis process of FIG. 12 ends. On the other hand, when the photographing is not finished (S12; NO), the process from step S1 to step S12 is executed for the new musical score image 70 acquired from the photographing apparatus 10. Note that the process of step S6 may be performed before the process of step S4. That is, the prior relationship between the process for calculating the first index X1 (S4 to S5) and the process for calculating the second index X2 (S6) is arbitrary.

  As understood from the above description, the processing from step S1 to step S11 is repeatedly executed until the end of moving image shooting is instructed (S12; YES). That is, the detection of the musical score symbol 72 (n) by the symbol detection unit 510 (S2), the specification of the symbol sequence by the symbol sequence specification unit 520 (S3), and the calculation of the posture index Z by the index calculation unit 530 (S4 to S7). Is executed sequentially in parallel with moving image shooting by the shooting device 10.

  In addition, an operation in which the display control unit 560 displays the symbol string graphic 78, the reference graphic 80, and the posture index Z on the display device 40 is sequentially performed in parallel with the moving image shooting. Since the user appropriately adjusts the photographing direction of the photographing apparatus 10 so that the symbol string figure 78 matches the reference figure 80, the score image 70 (for example, the position and size of the score symbol 72 (n)) also changes every moment. To do. Therefore, the symbol string figure 78 and the posture index Z change with time. That is, the display control unit 560 changes the symbol string graphic 78 and the posture index Z with time according to the symbol string specifying result by the symbol string specifying unit 520 (every time the processing of S1 to S7 is executed). . Therefore, the user can adjust the shooting direction to an appropriate direction while visually confirming the relationship among the symbol string graphic 78, the reference graphic 80, and the score image 70.

  As described above, in the first embodiment, a unit region A (n) that intersects a symbol string that is an array of a plurality of score symbols 72 (n) detected from the score image 70 is set, and the unit region A ( The first index X1 corresponding to the ratio of the staff 62 (n) included in n) is calculated. Since the ratio of the staff 62 (n) in the unit area A (n) decreases as the shooting direction is inclined with respect to the normal direction of the score, the first index X1 also decreases. Also, a second index X2 corresponding to the symbol string angle θ is calculated. Since the angle θ of the symbol string changes in accordance with the inclination angle of the shooting direction with respect to the normal direction of the score or the rotation angle of the shooting range with respect to the score (the rotation angle of the shooting device 10 in the plane parallel to the score), the angle θ Accordingly, the second index X2 also changes. Therefore, it is possible to quantitatively evaluate the suitability of the shooting direction for the score.

  In the rotated state in which the shooting range is rotated as compared with the reference state while the shooting direction is maintained parallel to the normal direction of the score, the symbol string is inclined with respect to the x-axis direction or the y-axis direction. Each staff 62 (n) is orthogonal to the symbol string. Therefore, the second index X2 is a small numerical value, while the first index X1 is a large numerical value. On the other hand, when the shooting direction is inclined with respect to the normal direction of the score, both the first index X1 and the second index X2 are small numerical values. As understood from the above description, the second index X2 reflects the inclination of the symbol string in both the tilted state and the rotated state, whereas the first index X1 includes the shooting direction in the tilted state. However, the inclination of the symbol string due to the rotation of the shooting range is not reflected. Considering the above tendency, the first index X1 is particularly suitable as an index of the degree to which the shooting direction is inclined with respect to the normal direction of the score (inclined state).

<Second Embodiment>
A second embodiment of the present invention will be described. Regarding the elements whose functions and functions are the same as those of the first embodiment in each aspect exemplified below, the detailed description of each is appropriately omitted by using the reference numerals used in the description of the first embodiment.

  FIG. 13 is a configuration diagram of the score image analyzing apparatus 100 of the second embodiment. In the first embodiment, the shooting control unit 570 instructs the shooting device 10 to take a still image in response to an operation from the user on the input device 20 (S10). As illustrated in FIG. 13, the shooting control unit 570 of the second embodiment performs still image shooting of the score image 70A on the shooting device 10 when the posture index Z calculated by the index calculation unit 530 satisfies a predetermined condition. Instruct.

  Specifically, the shooting control unit 570 instructs the shooting apparatus 10 to take a still image according to the comparison result between the posture index Z calculated by the index calculation unit 530 and a predetermined threshold value ZT. For example, when the posture index Z exceeds the threshold value ZT, the photographing apparatus 10 is instructed to take a still image. The threshold value ZT to be compared with the posture index Z is selected experimentally or statistically. In addition, the threshold value ZT can be variably set in accordance with an instruction from the user to the input device 20.

  FIG. 14 is a flowchart of a process in which the control device 50 according to the second embodiment analyzes the score image 70. From the acquisition (S1) of the score image 70 to the display of the analysis result (S8) is the same as in the first embodiment. The imaging control unit 570 determines whether or not the posture index Z calculated by the index calculation unit 530 exceeds the threshold value ZT (S9). The posture index Z becomes a larger numerical value as the shooting direction of the shooting device 10 is closer to the reference state. When the posture index Z exceeds the threshold value ZT (S9; YES), that is, when the photographing direction of the photographing apparatus 10 is appropriate, the photographing control unit 570 instructs the photographing apparatus 10 to shoot a still image (S10). In response to this, the score image 70A of the still image photographed by the photographing device 10 is stored in the storage device 30 (S11). On the other hand, when the posture index Z is less than the threshold value ZT (S9; NO), as illustrated in FIG. 14, the processing of step S12 is executed without executing the processing of steps S10 and S11. Similar to the first embodiment, the processing from step S1 to step S11 is repeated until the end of moving image shooting is instructed by the user.

  In the second embodiment, the same effect as in the first embodiment is realized. In the second embodiment, when the posture index Z satisfies a predetermined condition, the photographing apparatus 10 is instructed to shoot a still image of the score image 70A. Therefore, it is possible to acquire a still image (score image 70A) of a score shot in a shooting direction suitable for shooting a score without the user giving a shooting instruction.

<Modification>
Each aspect illustrated above can be variously modified. Specific modifications are exemplified below. Two or more modes arbitrarily selected from the following examples can be appropriately combined within a range that does not contradict each other.

(1) In each of the above-described embodiments, the total or weighted sum of the ratios R (1) to R (N) in the N unit areas A (1) to A (N) is calculated as the first index X1, The calculation method of 1 index X1 is not limited to the above illustration. For example, considering that N unit regions A (1) to A (N) are inclined at a common angle with respect to the x-axis direction, the N ratios R (1) to R (N) are mutually Expected to be close to. Accordingly, the ratio R (n) in one specific unit region A (n) among the N unit regions A (1) to A (N) can be used as the first index X1.

(2) In the above-described embodiments, the absolute value | Cy | of the component Cy in the y-axis direction of the vector C that is parallel to the long side of the symbol string figure 78 and forms the angle θ with respect to the x-axis is calculated as the second index X2. However, the method of calculating the second index X2 is not limited to the above example. For example, the absolute value of the vector C perpendicular to the symbol string in the x-axis direction (or y-axis direction) is calculated as the second index X2, or the second index is determined by a predetermined calculation for the angle θ of the symbol string with respect to the x-axis direction. A configuration for calculating X2 (thus, it is not necessary to specify the vector C) may be employed. As understood from the above description, the second index X2 is comprehensively expressed as an index corresponding to the angle θ of the symbol string, and any specific calculation method is not required.

(3) In each of the above-described embodiments, the product of the first index X1 and the second index X2 is calculated as the attitude index Z. However, the calculation method of the attitude index Z is not limited to the above examples. For example, the total or weighted sum of the first index X1 and the second index X2 can be used as the posture index Z. Further, a configuration in which the first index X1 is set as the posture index Z (calculation of the second index X2 is omitted), and a configuration in which the second index X2 is set as the posture index Z (calculation of the first index X1 is omitted). May also be employed.

(4) In the above-described embodiments, the case where the first index X1 and the second index X2 increase as the shooting direction of the shooting apparatus 10 is closer to the reference state is exemplified. The relationship between the two indices X2 can be reversed. For example, in a configuration in which the sum of N ratios R (1) to R (N) and the reciprocal of the weighted sum are calculated as the first index X1, the first index X1 increases as the shooting direction of the shooting apparatus 10 is closer to the reference state. Small number. Similarly, for the second index X2, for example, in the configuration in which the absolute value | Cx | of the component Cx in the x-axis direction of the vector C is calculated as the second index X2, the second is greater as the shooting direction of the shooting apparatus 10 is closer to the reference state. The index X2 is a small numerical value.

(5) In each of the above-described forms, the rough shape of the range 74 to be searched for the musical score symbol 72 (n) is the reference graphic 80. However, the shape and position of the reference graphic 80 are not limited to the above examples. For example, a straight line parallel to the y-axis can be used as the reference graphic 80. As understood from the above examples, the reference graphic 80 is comprehensively expressed as a graphic serving as a reference for the symbol string graphic 78 (symbol string).

(6) In each of the above-described embodiments, the rectangular symbol string figure 78 that includes each vertex of the N symbol regions 76 (1) to 76 (N) and has the smallest area is set. The shape and setting method of the symbol string graphic 78 are not limited to the above examples. For example, a straight line passing through the center of gravity of each of the symbol areas 76 (1) and 76 (N) or a rectangle along the straight line can be used as the symbol string graphic 78.

(7) In each of the above-described embodiments, the treble clef and the syllable symbol are detected as the musical score symbol 72 (n), but the type of the musical score symbol 72 (n) is not limited to the above examples. For example, it is also possible to detect either a treble clef, a diacritic, or a clef as a musical score symbol 72 (n). Further, in each of the above-described embodiments, a staff score having a staff 62 (n) as a set of five straight lines representing different pitches is exemplified as the score, but the type (notation) of the score is limited to the above examples. Not. For example, the present invention can also be applied to a musical score 62 (n) composed of six straight lines corresponding to each string of a stringed musical instrument, with a tablature in which a performance method (specifically fingering) is written as a musical score. . When a tablature is taken as an object to be photographed, it is possible to detect a character string “TAB” vertically written at the left end of each staff 62 (n) as a musical score symbol 72 (n).

(8) In each of the above-described embodiments, each of the plurality of score images 70 generated in time series by moving image shooting is a processing target. For example, the first score image 70 generated by still image shooting is the target. It is also possible to calculate the first index X1 and the second index X2. As can be understood from the above description, in the present invention, moving image shooting of a score is not essential.

(9) It is also possible to realize the score image analysis device 100 in cooperation with a server device and a terminal device that communicate with a terminal device (for example, a mobile phone or a smartphone) via a communication network such as a mobile communication network or the Internet. is there. For example, the index calculation unit 530 can be installed in the server device, and other elements (symbol detection unit 510, symbol string identification unit 520, display control unit 560, imaging control unit 570) can be installed in the terminal device. is there. As can be understood from the above description, the score image analyzing apparatus 100 is also realized by a set (system) of a plurality of apparatuses configured separately from each other, and the process performed by each apparatus is arbitrarily determined.

(10) In the second embodiment, when the posture index Z exceeds the threshold value ZT, the photographing apparatus 10 is instructed to capture a still image. However, the condition of the posture index Z to be instructed to capture a still image is not limited to the above examples. . Specifically, it is also possible to instruct the photographing apparatus 10 to shoot a still image on condition that the posture index Z stably changes over a predetermined time in a range exceeding the threshold value ZT. For example, when the distribution degree (for example, variance) of a plurality of posture indices Z calculated in time series is lower than a predetermined value, it can be determined that the posture index Z is stable. A configuration is also employed in which the temporal variation of the posture index Z is monitored, and when the posture index Z is maximized, the photographing apparatus 10 is instructed to take a still image. As understood from the above description, the imaging control unit 570 is comprehensively expressed as an element that instructs the imaging device 10 to capture a still image of the score image 70A when the posture index Z satisfies a predetermined condition. The specific content of the occurrence condition is not questioned.

(11) The musical score image analyzing apparatus 100 exemplified in each of the above-described embodiments is realized by the cooperation of the control device 50 and a program as illustrated in the above-described embodiments. A program according to a preferred embodiment of the present invention includes N staffs 62 (1) to 62 (N) arranged in parallel to each other and a musical score symbol 72 (positioned at a predetermined position with respect to each staff 62 (n). n), a symbol detection unit 510 that detects score symbols 72 (1) to 72 (N) from a score image 70 obtained by photographing a score including N), and N score symbols 72 (1) to 72 detected by the symbol detection unit 510. A symbol string identifying unit 520 that identifies a symbol string that is an array of (N), a unit region setting unit 532 that sets a unit region A (n) that intersects the symbol string at the position of the score symbol 72 (n), and a score The computer functions as the first index calculation unit 534 that calculates the first index X1 according to the ratio R (n) included in the unit area A (n) in the staff 62 (n) in which the symbol 72 (n) is arranged. Let The programs exemplified above can be provided in a form stored in a computer-readable recording medium and installed in the computer. The recording medium is, for example, a non-transitory recording medium, and an optical recording medium (optical disk) such as a CD-ROM is a good example, but a known arbitrary one such as a semiconductor recording medium or a magnetic recording medium This type of recording medium can be included. It is also possible to distribute the program to a computer in the form of distribution via a communication network.

(12) The present invention is also specified as an operation method (score image analysis method) of the score image analysis apparatus 100 according to each of the above-described embodiments. For example, in the score image analysis method according to one aspect of the present invention, N staffs 62 (1) to 62 (N) and each staff 62 (n) arranged in parallel are arranged at predetermined positions. The score symbols 72 (1) to 72 (N) are detected from the score image 70 obtained by photographing the score including the score symbols 72 (n) (S2), and the detected N score symbols 72 (1) to 72 (N). ) Is specified (S3), a unit area A (n) intersecting the symbol string is set at the position of the score symbol 72 (n) (S4), and the score symbol 72 (n) is arranged. The first index X1 corresponding to the ratio R (n) included in the unit area A (n) of the staff 62 (n) is calculated (S5).

DESCRIPTION OF SYMBOLS 100 ... Musical score image analysis apparatus, 10 ... Shooting device, 20 ... Input device, 30 ... Storage device, 40 ... Display device, 50 ... Control device, 62 ... Staff, 63 ... Straight line, 64 ... ... set, 70, 70A ... score image, 72 ... score symbol, 74 ... range, 76 ... symbol area, 78 ... symbol string figure, 80 ... reference figure, 510 ... symbol detector, 520 ... ... symbol string specifying unit, 530 ... index calculating unit, 532 ... unit area setting unit, 534 ... first index calculating unit, 544 ... second index calculating unit, 550 ... posture index calculating unit, 560 ... Display control unit, 570 ... Shooting control unit

Claims (4)

Detecting a score symbol from each score image generated in time series by shooting a moving image with a photographing device including a score including a plurality of staffs arranged in parallel and a score symbol arranged at a predetermined position with respect to each staff A symbol detector to
A symbol string identification unit that identifies a symbol string that is an array of a plurality of musical score symbols detected by the symbol detection unit;
An index calculation unit that calculates an attitude index that is an index of suitability of a shooting direction with respect to the score according to the symbol string;
A score image analysis apparatus comprising: a shooting control unit that instructs the shooting apparatus to shoot a still image of the score image when the posture index satisfies a predetermined condition. The index calculation unit
A unit region setting unit that sets a unit region that intersects the symbol string at the position of the musical score symbol;
A first index calculation unit that calculates a first index according to a ratio included in the unit area of the staff on which the musical score symbol is arranged,
The musical score image analysis apparatus according to claim 1, wherein the posture index corresponding to the first index is set. The index calculation unit
A second index calculation unit that calculates a second index according to the angle of the symbol string;
The musical score image analysis apparatus according to claim 1 or 2, wherein the posture index corresponding to the second index is set. The posture index is an index that becomes a larger numerical value as the horizontal direction of the shooting range is closer to a state parallel to the horizontal direction of the score,
The musical score image analysis apparatus according to claim 1, wherein the predetermined condition is that the posture index exceeds a predetermined threshold value.

Images