WO2019087331A1

WO2019087331A1 - Reference position detection device and reference position detection method

Info

Publication number: WO2019087331A1
Application number: PCT/JP2017/039578
Authority: WO
Inventors: 大地渡邉
Original assignee: ヤマハ株式会社
Priority date: 2017-11-01
Filing date: 2017-11-01
Publication date: 2019-05-09

Abstract

This reference position detection device is provided with an acquisition unit, an extraction unit, an identification unit, and a detection unit. The acquisition unit acquires image data representing an image obtained by imaging a keyboard and an object on the keyboard. The extraction unit extracts image data corresponding to an area which includes white keys but not black keys from the image data acquired by the acquisition unit. The identification unit identifies the object on the keyboard on the basis of the luminance value of the image data extracted by the extraction unit. The detection unit detects, for the object identified by the identification unit, a reference position which has a prescribed positional relationship on the keyboard.

Description

Reference position detection device and reference position detection method

The present invention relates to a reference position detection device and a reference position detection method for detecting a reference position on a keyboard.

According to the key area division technique used for the electronic keyboard instrument, the keyboard is divided into a plurality of key areas at key area division positions (split points), and tone control characteristics such as timbre, area or function are individually assigned to each key area. Assigned. Patent Document 1 describes a key area dividing device capable of automatically changing the key area dividing position according to the playing state of the performer. In the key area dividing device of Patent Document 1, the entire key area of the keyboard is imaged by two cameras. The images obtained by each camera are analyzed, and the type of hand of the player's right hand or left hand etc. based on the arrangement of each finger, hand size, finger length, finger thickness, skin color etc. And the position of the hand is detected. The key area division position is dynamically changed based on the detected hand type and hand position.
Patent No. 4389841

However, in the key area dividing device described in Patent Document 1, in order to detect the type of hand and the position of the hand, it is necessary to perform complicated image processing on the images obtained by the two cameras. This adds to the cost of hardware and software.

An object of the present invention is to provide a reference position detection device and a reference position detection method capable of detecting a reference position on a keyboard with simpler processing and at low cost.

According to one aspect of the present invention, a reference position detection device includes a keyboard and an acquisition unit for acquiring image data representing an image obtained by imaging an object on the keyboard, and a white key not including a black key from the acquired image data. And an identification unit for identifying an object on the keyboard based on the luminance value of the extracted image data, and a predetermined position on the keyboard with respect to the identified object. And a detection unit that detects a reference position having a relationship.

The object may be the player's hand. The hand may be both hands, and the reference position may be a position between the hands. The reference position may be a key area division position that divides the keyboard into a plurality of key areas to which individual tone control characteristics are assigned.

The reference position detection device may further include a resolution reduction unit that reduces the resolution of the extracted image data, and the identification unit may identify an object on the keyboard based on the luminance value of the image data after the resolution reduction. . The reference position detection device may further include a blurring processing unit that performs blurring processing on the extracted image data, and the identification unit may identify an object on the keyboard based on the luminance value of the image data after the blurring processing.

The reference position detection device may further include a keyboard and an imaging device for imaging an object on the keyboard, and the acquisition unit may acquire image data from the imaging device. The imaging device is a camera having a fisheye lens, and the reference position detection device further includes a correction unit that corrects the acquired image data so that the keyboard has a rectangular shape, and the extraction unit extracts a black key from the corrected image data. The image data corresponding to the area not including white and including the white key may be extracted. The acquisition unit may acquire image data indicating an image including the entire keyboard.

A reference position detection method according to another aspect of the present invention comprises: obtaining an image data representing an image obtained by imaging a keyboard and an object on the keyboard; and a white key not including a black key from the acquired image data Having a predetermined positional relationship on the keyboard with respect to the identified object, the steps of extracting image data representing an area including A, identifying the object on the keyboard based on the brightness value of the extracted image data, and Detecting a reference position.

According to the present invention, it is possible to detect the reference position on the keyboard with simpler processing and at low cost.

FIG. 1 is a block diagram showing the configuration of an electronic music apparatus including a reference position detection apparatus according to an embodiment of the present invention. FIG. 2 is a block diagram mainly showing a functional configuration of the reference position detection device of FIG. FIG. 3 is a flowchart showing a reference position detection method in the reference position detection device of FIG. FIG. 4 is a diagram showing the keyboard and the hands of the player. FIG. 5 is a schematic view showing an example of an image subjected to resolution reduction processing. FIG. 6 is a schematic view showing an example of the image subjected to the blurring process and the gray scale process. FIG. 7 is a schematic view showing an example of a binarized image. FIG. 8 is a schematic view showing an example of a histogram. FIG. 9 is a flowchart showing the area identification process of step S10 of FIG. FIG. 10 is a flowchart showing the key area division position detection process of step S11 of FIG.

Hereinafter, a reference position detection apparatus and a reference position detection method according to an embodiment of the present invention will be described in detail with reference to the drawings.

(1) Configuration of Electronic Music Device FIG. 1 is a block diagram showing the configuration of an electronic music device including a reference position detection device according to an embodiment of the present invention.

The electronic music apparatus 1 of FIG. 1 is, for example, an electronic keyboard instrument. The electronic music apparatus 1 includes an imaging device 30, a keyboard 2, a setting operation unit 3, and a display 4. The keyboard 2 has a plurality of black keys and white keys and is connected to the bus 14. The keyboard 2 may be an image of the keyboard displayed on the screen of the touch panel described later.

The imaging device 30 is disposed to capture an image of the keyboard 2 and outputs image data representing an image of the keyboard 2 to the bus 14. In the present embodiment, the imaging device 30 is a single camera having a fisheye lens. In this case, the imaging device 30 is disposed above the central portion of the keyboard 2 or obliquely rearward above the central portion, and images the entire keyboard 2. As a result, image data indicating the entire image of the keyboard 2 is output from the imaging device 30.

The setting operation unit 3 includes an operation switch that is turned on and off, an operation switch that is rotated, and an operation switch that is slid and is connected to the bus 14. The setting operation unit 3 is used to perform various instructions, various adjustments, various settings, and power on / off. The display 4 includes, for example, a liquid crystal display, and is connected to the bus 14. The display 4 displays a music title, a score, or other various information. The display 4 may be a touch panel display. In this case, a part or all of the keyboard 2 or the setting operation unit 3 may be displayed on the display 4. The user can instruct various operations by operating the display 4.

The electronic music apparatus 1 includes a sound source unit 5 and a sound system 6. The sound source unit 5 is connected to the bus 14 and generates audio data (sound signal) based on tone data etc. reproduced by the operation of the keyboard 2. Audio data is sampling data indicating a sound waveform. The sound system 6 includes a digital analog (D / A) conversion circuit, an amplifier and a speaker. The sound system 6 converts audio data supplied from the sound source unit 5 into an analog sound signal, and generates sound based on the analog sound signal.

The electronic music apparatus 1 further includes a storage device 7, a CPU (central processing unit) 8, a timer 9, a RAM (random access memory) 10, a ROM (read only memory) 11, and a communication I / F (interface) 12. . The storage device 7, the CPU 8, the RAM 10, the ROM 11 and the communication I / F 12 are connected to the bus 14. The timer 9 is connected to the CPU 8. An external device such as the external storage device 13 may be connected to the bus 14 via the communication I / F 12.

The storage device 7 includes a storage medium such as a hard disk, an optical disk, a magnetic disk or a memory card. The storage device 7 stores white key area coordinate information, note number correspondence information, and key area division position information described later. Further, the storage device 7 stores a computer program such as a reference position detection program. Furthermore, the storage device 7 stores performance data indicating a performance sound of a musical instrument and automatic accompaniment data for automatic accompaniment. In the present embodiment, the tone data includes performance data and automatic accompaniment data. The musical tone data may be audio data or MIDI (Musical Instrument Digital Interface) data.

The RAM 10 is, for example, a volatile memory and is used as a work area of the CPU 8 and temporarily stores various data. The ROM 11 is, for example, a non-volatile memory, and stores a control program. The ROM 11 may store a computer program such as a reference position detection program. The CPU 8 executes a reference position detection program stored in the storage device 7 or the ROM 11 to perform a reference position detection method described later. The timer 9 provides the CPU 8 with time information indicating time lapse. The CPU 8, the timer 9, the RAM 10 and the ROM 11 constitute a reference position detection device 100.

The reference position detection program may be provided in a form stored in a computer readable recording medium, and may be installed in the storage device 7 or the ROM 11. Also, the reference position detection program may be stored in the external storage device 13. Furthermore, when the communication I / F 12 is connected to the communication network, a reference position detection program distributed from a server connected to the communication network may be installed in the storage device 7 or the ROM 11.

(2) Functional Configuration of Reference Position Detection Device 100 FIG. 2 is a block diagram mainly showing the functional configuration of the reference position detection device 100 of FIG. As shown in FIG. 2, the reference position detection device 100 includes an acquisition unit 101, a correction unit 102, an extraction unit 103, a resolution reduction unit 104, a blurring processing unit 105, and a gray scale processing unit 106. The reference position detection device 100 further includes a binarization unit 107, a generation unit 108, an identification unit 109, a detection unit 110, a presentation unit 111, and a tone control characteristic setting unit 112. The function of each component (101 to 112) of the reference position detection device 100 is realized by the CPU 8 of FIG. 1 executing a reference position detection program stored in the storage device 7 or the ROM 11.

The imaging device 30 images the keyboard 2 and the hands of the player on the keyboard 2 and outputs image data representing an image including the entire keyboard 2 and a part of the hand. In the present embodiment, the image data is color image data representing a color image. Color image data is composed of a plurality of pixels having luminance values of a color image. The acquisition unit 101 acquires image data output from the imaging device 30. In the present embodiment, since the imaging device 30 is a camera having a fisheye lens, the image of the keyboard 2 represented by the image data is distorted such that the central portion is enlarged compared to the both ends. The correction unit 102 corrects the image data acquired by the acquisition unit 101 so that the image of the keyboard 2 has a rectangular shape.

The extraction unit 103 does not include a black key from the image data corrected by the correction unit 102 based on the white key region coordinate information acquired from the storage device 7 and includes a white key (hereinafter referred to as a white key region). Extract the image data corresponding to. The white key area coordinate information includes coordinates for specifying a white key area in the entire image of the keyboard 2.

The resolution reduction unit 104 performs resolution reduction processing to reduce the resolution of the image data extracted by the extraction unit 103. The blurring processing unit 105 performs blurring processing on the image data having the resolution reduced by the resolution reducing unit 104. The grayscale processing unit 106 converts the color image data into grayscale image data by performing grayscale processing on the image data subjected to the blurring processing. The luminance value of each pixel of gray scale image data represents a gradation value.

The binarization unit 107 performs binarization processing to convert the luminance value of the image data obtained by the gray scale processing unit 106 into binary pixel values. Thereby, binary image data is generated. Binary image data is composed of a plurality of pixels having a value of “0” or “1”. The generation unit 108 performs a histogram generation process of generating a histogram based on each pixel value of the image data obtained by the binarization unit 107. The histogram generation process will be described later.

The identification unit 109 identifies the hand on the keyboard 2 based on the histogram generated by the generation unit 108. The detection unit 110 detects the note number of the reference position having a predetermined positional relationship with the hand identified by the identification unit 109 based on the note number correspondence information acquired from the storage device 7. Details of operations of the identification unit 109 and the detection unit 110 will be described later. In the present embodiment, the predetermined positional relationship is the central position between the left hand and the right hand, and the reference position is the key area division position (split point). The note number correspondence information indicates the correspondence between a plurality of pixels constituting the image of the keyboard 2 and the note number. The note number of the key area division position detected by the detection unit 110 is stored in the storage device 7 as key area division position information. Thereby, the key area division position is set to the key having the note number indicated by the key area division position information.

The presentation unit 111 presents the key area division position detected by the detection unit 110 to the player using the guide lamp 41, the key light emission unit 42 or the display 4. The guide lamp 41 is composed of a plurality of light emitting diodes disposed behind the plurality of keys of the keyboard 2. The presentation unit 111 presents the key area division position to the player by lighting the light emitting diode disposed behind the key for which the key area division position is set. In addition, when the key emission unit 42 is provided for each key in the keyboard 2, the presentation unit 111 causes the key area division by emitting the key emission unit 42 provided for the key for which the key area division position is set. The position may be presented to the player. Furthermore, the presentation unit 111 may cause the display 4 to display the image of the keyboard 2 and display the key area division position on the image of the keyboard 2.

The musical tone control characteristic setting unit 112 sets individual musical tone control characteristics in two key areas divided left and right at the key area division position of the keyboard 2 based on the key area division position information stored in the storage device 7 Do. The individual tone control characteristics are, for example, an individual tone, an individual tone range or an individual function. For example, the tone color of the base may be set in the left key range, and the tone color of the piano may be set in the right key range. Further, a range two octaves higher than the original range may be set in the left key range, and the original range may be set in the right range. Furthermore, an automatic accompaniment function by chords may be set in the left key range, and a normal performance function may be set in the right key range.

(3) Reference Position Detection Method FIG. 3 is a flowchart showing a reference position detection method in the reference position detection apparatus 100 of FIG. FIG. 4 shows the keyboard 2 and the hands of the player. FIG. 5 is a schematic view showing an example of an image subjected to resolution reduction processing. FIG. 6 is a schematic view showing an example of the image subjected to the blurring process and the gray scale process. FIG. 7 is a schematic view showing an example of a binarized image. FIG. 8 is a schematic view showing an example of a histogram. The reference position detection method of FIG. 3 is performed by the CPU 8 of FIG. 1 executing a reference position detection program stored in the storage device 7 or the ROM 11.

As shown in FIG. 4, the keyboard 2 has an array of a plurality of black keys BK and white keys WH. A plurality of black keys BK and white keys WH are assigned note numbers that increase in order from left to right. For example, when the keyboard 2 has 88 keys, note numbers “21” to “108” are assigned in order from the leftmost key to the rightmost key.

A linear gap 21 exists between each two adjacent white keys WH. In addition, the left hand LH and the right hand RH of the player exist on the keyboard 2. A key range division position (split point) SP is set between the left hand LH and the right hand RH by a method described later. As described above, the white key area 22 is a rectangular area not including the black key BK in the keyboard 2 but including the white key WH.

First, the acquisition unit 101 of FIG. 2 acquires image data representing the entire image of the keyboard 2 from the imaging device 30 (step S1). As described above, in the present embodiment, since the imaging device 30 is a single camera having a fisheye lens, the image of the keyboard 2 represented by the image data is enlarged so that the central portion is larger than both ends. It is distorted. Therefore, the correction unit 102 corrects the image data acquired by the acquisition unit 101 so that the image of the keyboard 2 has a rectangular shape (step S2).

The extraction unit 103 acquires white key area coordinate information from the storage device 7 (step S3), and extracts image data corresponding to the white key area 22 from the image data based on the acquired white key area coordinate information (step S4). ). When the position of the white key area 22 is fixed in the entire image of the keyboard 2, white key area coordinate information indicating the position of the white key area 22 on the image is stored in advance in the storage device 7. The white key area coordinate information may be automatically calculated by image processing based on the luminance difference between the outer periphery of the keyboard 2 and the white key WH and the luminance difference between the black key BK and the white key WH in the image. In addition, white key area coordinate information may be generated by the user designating four corners or the outer periphery of the white key area 22 of the keyboard 2 on the image displayed on the display 4. Furthermore, white key area coordinate information may be generated by the user inputting the coordinates of the white key area 22 on the image. Coordinates in the horizontal direction (left and right direction) and the vertical direction (front and rear direction) are assigned to a plurality of pixels forming the image data of the white key area 22.

Next, the resolution reduction unit 104 performs resolution reduction processing on the extracted image data (step S5). In the resolution reduction process, for example, the number of pixels forming the image data is reduced by averaging the luminance values of adjacent constant pixel groups. Thereby, as shown in FIG. 5, the black straight line representing the gap 21 between the white keys WH in FIG. 4 is blurred on the image of the white key area 22. The luminance values of the left hand portion IL and the right hand portion IR corresponding to the left hand LH and the right hand RH in the image of the white key area 22 are lower than the luminance values of other portions corresponding to the white key WH. In the present embodiment, as described above, since a camera having a fisheye lens is used, the width of the gap between the white keys WH is enlarged at the central portion of the white key area 22 on the image. Thus, as shown in FIG. 5, a thick vertical line 21a representing the gap 21 between the white keys WH remains at the center of the image of the white key area 22.

The blurring processing unit 105 performs blurring processing on the image data having the resolution reduced by the resolution reducing unit 104 (step S6). In addition, the gray scale processing unit 106 performs gray scale processing on the image data on which the blurring process has been performed (step S7). Thereby, image data representing a gray scale image is obtained. In the gray scale image of FIG. 6, the vertical line 21a of FIG. 5 representing the gap 21 between the white keys WH is eliminated by being equalized to the color of the white key WH. In addition, the luminance values (gradation values) of the left hand portion IL and the right hand portion IR become lower than the luminance values (gradation values) of the other portions.

Next, the binarization unit 107 performs a binarization process of converting the luminance value of the image data obtained by the gray scale processing unit 106 into a binary pixel value (step S8). Thereby, binary image data representing a binary image is generated. Binary image data is composed of a plurality of pixels having a value of “0” or “1”. For example, in a grayscale image, the value of a pixel having a luminance value higher than a predetermined luminance threshold value is converted to "0", and the value of a pixel having a luminance value smaller than the luminance threshold value is "1". It is converted. The brightness threshold value in the binarization process is adjusted by the setting switch of the setting operation unit 3 in accordance with the brightness around the keyboard 2 or the like. As shown in FIG. 7, in the binary image, the pixel values of the left hand portion IL and the right hand portion IR are “1” (white), and the pixel values of the other portions are “0” (black). Thereby, in the image data representing a binary image, the left hand part IL and the right hand part IR are clearly distinguished from other parts.

Next, the generation unit 108 performs histogram generation processing based on binary image data (step S9). Specifically, in the binary image data, the generation unit 108 calculates the sum of pixel values for each pixel row arranged parallel to the longitudinal direction (front-back direction) of each white key WH of the keyboard 2 A histogram is generated which indicates the sum of pixel values of a pixel row (referred to as a total pixel value) as a frequency. Each section on the horizontal axis of the histogram HG in FIG. 8 is each pixel row, and the frequency on the vertical axis is a total pixel value. The histogram HG shows the distribution of total pixel values of a plurality of pixel columns. The higher the total pixel value of each pixel column, the higher the probability that a hand is present. Further, in the histogram HG, a predetermined determination threshold TH is set.

Next, the identification unit 109 performs region identification processing for identifying a hand on the keyboard 2 based on the histogram HG (step S10). Details of the area identification process will be described later. Thereafter, based on the hand area identified by the identification unit 109, the detection unit 110 performs key area division position detection processing for detecting a key area division position, which is a reference position (step S11). The keyboard 2 is divided into a left key area and a right key area around the detected key area division position. Details of the key area division position detection process will be described later.

Next, the musical tone control characteristic setting unit 112 assigns individual musical tone control characteristics to the left key area and the right key area (step S12). Also, the presentation unit 111 presents the key area division position to the player using the guide lamp 41, the key light emission unit 42 or the display 4 (step S13).

Thereafter, the detection unit 110 determines whether the end of the reference position detection process has been instructed (step S14). When the end of the reference position detection process is not instructed, the detection unit 110 returns to step S1. Thus, the processes of steps S1 to S14 are performed. In this way, when the player's hand moves on the keyboard 2, the key area division position moves dynamically along with the movement of the hand. When the end of the reference position detection process is instructed in step S14, the reference position detection process ends. Here, an instruction to end the reference position detection process is generated, for example, by the end of the music performance or the operation of the operation switch of the setting operation unit 3.

FIG. 9 is a flowchart showing the area identification process of step S10 of FIG. The identification unit 109 extracts a pixel sequence having a total pixel value equal to or greater than the determination threshold TH in the histogram HG generated by the generation unit 108 (step S101), and selects one or more adjacent pixel sequences among the extracted pixel sequences. An area composed of pixel columns is extracted (step S102). The user can operate the setting switch of the setting operation unit 3 in advance to display the histogram HG on the display 4 and adjust the determination threshold TH.

Next, the identification unit 109 determines whether the width of the extracted area is equal to or greater than a predetermined value (step S103). The predetermined value is preset based on the width of the human hand. If the width of the extracted area is equal to or greater than the predetermined value, the identification unit 109 identifies the extracted area as a hand area (step S104). The position of the hand area in the white key area 22 is represented by the coordinates in the horizontal direction (left and right direction) in the white key area 22.

In step S103, when the width of the extracted area is smaller than the predetermined value, the identification unit 109 determines that the extracted area is not a hand area (step S105). Thereby, the area due to noise is excluded from the hand area.

Thereafter, the identification unit 109 determines whether the extraction of all the regions in the image data is completed (step S106). If the extraction of all the areas has not been completed, the identification unit 109 returns to step S102 to extract another area. When the extraction of all the areas is completed in step S106, the identification unit 109 ends the area identification process, and the process proceeds to step S11 in FIG.

FIG. 10 is a flowchart showing the key area division position detection process of step S11 of FIG. First, the detection unit 110 determines whether the identification unit 109 has identified two hand regions (step S111). If two hand areas are identified, the detection unit 110 calculates the coordinates of the center position of each hand area (step S112), and the coordinates of the center position between the coordinates of the center positions of the two hand areas are keyed The coordinates of the area division position are calculated (step S113). In the example of FIG. 8, two hand regions R1 and R2 are identified. Further, the coordinates of the central position P1 of the hand region R1 and the coordinates P2 of the central region P2 of the hand region R2 are calculated, and the coordinates of the central position P0 between the coordinates of the central positions P1 and P2 of the two hand regions R1 and R2 is a key Calculated as the coordinates of the area division position.

Next, the detection unit 110 acquires note number correspondence information from the storage device 7 (step S114), and assigns a note number to the key area division position based on the note number correspondence information (step S115). After that, the detection unit 110 stores key area division position information indicating the note number of the key area division position in the storage device 7 (step S116). As a result, the key area division position is set to the keyboard 2. When the key area division position information is already stored in the storage device 7, the detection unit 110 updates the key area division position information stored in the storage device 7. Thereafter, the detection unit 110 proceeds to step S12 in FIG.

If two hand areas are not identified in step S111, the detection unit 110 proceeds to step S12 in FIG. 3 without storing the key area division position information in the storage device 7 (step S117). When the key area division position information is already stored in the storage device 7, the detection unit 110 proceeds to step S12 without updating the key area division position information of the storage device 7.

(4) Effects of the Embodiment According to the reference position detection device 100 and the reference position detection method according to the present embodiment, the brightness of the image data corresponding to the white key area 22 not including the black key BK but including the white key WH. The hand on the keyboard 2 is identified based on the value. In this case, the hand on the keyboard 2 is identified with high accuracy from other parts by a simple process based on the luminance difference between the white key WH and the hand. Further, the key area division position is detected by a simple process as a reference position having a predetermined positional relationship on the keyboard 2 with respect to the identified hand. Therefore, it becomes possible to detect the key area division position on the keyboard 2 with simpler processing and at low cost.

In the present embodiment, since the player's hand is identified as an object on the keyboard 2, it is possible to set a key area division position which dynamically changes in accordance with the movement of the hand on the keyboard 2. Further, in the present embodiment, since the key area division position between both hands is detected as the reference position, the key area division position dynamically changing according to the movement of the player's hands may be set between the both hands. it can.

In the present embodiment, since the resolution of the image data of the white key area 22 is reduced by the resolution reducing unit 104, the vertical line due to the gap 21 between the white keys WH can be removed from the image of the white key area 22. Thereby, the hand on the keyboard 2 can be more accurately identified from other parts. In addition, since the blurring process is performed on the image data by the blurring processing unit 105, the vertical line 21a due to the gap 21 between the white keys WH remaining in the image of the white key area 22 can be removed.

In the present embodiment, since a camera having a fisheye lens is used as the imaging device 30, the entire keyboard 2 can be imaged by a single camera. Further, since the image of the keyboard 2 is corrected to have a rectangular shape by the correction unit 102, it is possible to extract image data that accurately represents the rectangular white key area 22. Furthermore, in the present embodiment, since image data representing the entire image of the keyboard 2 is obtained by the imaging device 30, the hand on the keyboard 2 can be more accurately identified from other parts.

(5) Other Embodiments (a) In the above embodiment, the player's hand is identified as an object on the keyboard 2, but the object on the keyboard 2 is not limited to the hand. It may be any object of

(B) In the above embodiment, the key area division position of the center position between the player's hands is detected as the reference position having a predetermined positional relationship with the object, but the reference position is limited to the key area division position I will not. For example, the left end, the right end, or both ends of each player's hand may be detected as the reference position. In this case, information indicating the octave corresponding to the position of the hand on the keyboard on the teaching score displayed on the screen, for example, the pitch played by the player with respect to the octave to which the pitch represented in the score belongs. Information indicating whether the value is high or low may be displayed. Also, when the performer's performance is sequentially displayed on the score on the screen, the pitch played by the performer is distributed to the treble staff and the bass staff based on the position of the hand on the keyboard 2. May be displayed. In this case, even if the played pitch is the pitch that should normally be displayed on the bass staff, if it is inferred from the position of the player's right hand that the pitch is played with the right hand, then it will be played. The pitch may be displayed on the treble staff using an additive line. Conversely, even if the pitch played was normally to be displayed on the treble staff, if it was inferred from the position of the player's left hand that the pitch was played with the left hand, The indicated pitch may be displayed on the bass staff using an additive line.

(C) In the above embodiment, when the left hand and the right hand overlap, one hand area is detected. In that case, instead of calculating the key area division position based on the two hand areas, the key area division position may be calculated and updated based on one hand area. For example, the key area division position may be calculated based on the position of the left end, the right end, or the center of one detected hand area. Specifically, note numbers separated by a predetermined number of pitches from the note numbers corresponding to the positions of the end of the detected hand area or the positions of the end may be calculated as the key area division position .

(D) In the above embodiment, one key area division position is detected as the reference position, but a plurality of reference positions may be detected. For example, the keyboard 2 may be divided into three or more key areas by detecting a plurality of key area division positions as a plurality of reference positions.

(E) In the above embodiment, the imaging device 30 is a camera having a fisheye lens, but the present invention is not limited to this, and the imaging device 30 may be a camera having no fisheye lens. Also, the imaging device 30 may include a plurality of cameras. Furthermore, in the above embodiment, the coordinates of the key area division position are calculated after the image data obtained by the imaging device 30 is corrected, but the correction is not performed on the image data obtained by the imaging device 30. Coordinates of the area division position may be calculated, and the calculated coordinates may be converted into note numbers in consideration of distortion of the image due to the fisheye lens.

(F) In the above embodiment, image data representing the entire image of the keyboard 2 is obtained by the imaging device 30. However, image data representing a partial image of the keyboard 2 may be obtained by the imaging device 30. When the imaging range of the keyboard 2 by the imaging device 30 is determined in advance, it is possible to specify the note number corresponding to each part of the image. When an index or the like for specifying a note number is attached to a part of keys of the keyboard 2, it is possible to specify a note number corresponding to each part of the image. Furthermore, the user may be able to set the imaging range (key area) of the imaging device 30 by the setting operation unit 3. Alternatively, the imaging range (key area) may be automatically set by the angle of the imaging device 30.

(G) In the above embodiment, the acquisition unit 101 acquires image data representing a color image from the imaging device 30. However, the acquisition unit 101 may acquire image data representing a monochrome image from the imaging device 30. In this case, the gray scale processing unit 106 may not be provided.

(H) In the above embodiment, in the gray scale image, the value of the pixel having the luminance value higher than the luminance threshold is converted to “0”, and the value of the pixel having the luminance value smaller than the luminance threshold is “0”. The value of a pixel that is converted to 1 ′ ′ but has a luminance value higher than the luminance threshold is converted to “1”, and the value of a pixel that has a luminance value smaller than the luminance threshold is converted to “0” May be In this case, the generation unit 108 generates a histogram indicating the sum of the number of “0s” of each pixel column as a frequency.

(I) The reference position detection apparatus and the reference position detection method of the present invention are applicable not only to electronic keyboard instruments but also to electronic devices such as smart phones, tablet terminals, personal computers and the like. In this case, the keyboard may be displayed on the screen, or the keyboard may be connected to the electronic device. In the above embodiment, the reference position detection device 100 of FIG. 2 is realized by a computer such as the CPU 8 and a reference position detection program, but the reference position detection device 100 of FIG. 2 is realized by hardware such as an electronic circuit. It may be done.

Claims

An acquisition unit configured to acquire image data representing an image obtained by imaging a keyboard and an object on the keyboard;
An extraction unit for extracting image data corresponding to an area not including the black key but including the white key from the acquired image data;
An identification unit for identifying the object on the keyboard based on the luminance value of the extracted image data;
A reference position detection device comprising: a detection unit that detects a reference position having a predetermined positional relationship on the keyboard with respect to the identified object.
The reference position detection device according to claim 1, wherein the object is a player's hand.
The reference position detection device according to claim 2, wherein the hand is both hands, and the reference position is a position between the both hands.
The reference position detection device according to any one of claims 1 to 3, wherein the reference position is a key area division position where the keyboard is divided into a plurality of key areas to which individual tone control characteristics are respectively assigned.
And a resolution reducing unit that reduces the resolution of the extracted image data,
The reference position detection device according to any one of claims 1 to 4, wherein the identification unit identifies the object on the keyboard based on the luminance value of the image data after the resolution reduction.
The image processing apparatus further comprises a blurring processing unit that performs blurring processing on the extracted image data;
The reference position detection device according to any one of claims 1 to 5, wherein the identification unit identifies the object on the keyboard based on a luminance value of the image data after the blurring process.
The image processing apparatus further includes a binarization unit that performs binarization to convert luminance values of the extracted image data into binary pixel values,
The reference position detection device according to any one of claims 1 to 6, wherein the identification unit identifies the object on the keyboard based on a pixel value of image data after binarization.
In the extracted image data, the sum of the luminance values of each pixel row arranged parallel to the longitudinal direction of each white key of the keyboard is calculated, and a histogram is generated showing the sum of the luminance values of each pixel row as a frequency Further equipped with
The reference position detection device according to any one of claims 1 to 7, wherein the identification unit identifies the object on the keyboard based on the generated histogram.
The apparatus further comprises a keyboard and an imaging device for imaging an object on the keyboard,
The reference position detection device according to any one of claims 1 to 8, wherein the acquisition unit acquires image data from the imaging device.
The imaging device is a camera having a fisheye lens,
And a correction unit that corrects the acquired image data so that the keyboard has a rectangular shape,
10. The reference position detection device according to claim 9, wherein the extraction unit extracts image data corresponding to a region not including a black key but including a white key from the image data after correction.
The reference position detection device according to any one of claims 1 to 10, wherein the acquisition unit acquires image data indicating an image including the entire keyboard.
Acquiring an image data representing an image obtained by imaging a keyboard and an object on the keyboard;
Extracting image data representing an area not including the black key but including the white key from the acquired image data;
Identifying the object on the keyboard based on the luminance value of the extracted image data;
Detecting a reference position having a predetermined positional relationship on the keyboard with respect to the identified object.