KR20170115030A - Electronic device, sensing method of playing string instrument and feedback method of playing string instrument - Google Patents

Abstract

An electronic device according to various embodiments of the present invention includes an image sensor for sensing movement of a bow to a stringed instrument, a vibration sensing sensor for sensing vibration occurring in the stringed instrument, And a control module for determining the fingering position of the user. Other embodiments are also possible.

Description

TECHNICAL FIELD [0001] The present invention relates to an electronic device, a method of recognizing stringed musical performance of an electronic device, and a method of playing a stringed musical instrument in an electronic device.

The present invention relates to an electronic device that recognizes a stringed instrument performance and provides feedback.

Various electronic devices are being developed due to the development of electronic technology. For example, a device for recognizing a performance of a string player has been developed, and an attempt has been made to accurately recognize the player's performance using various kinds of sensors.

Some of the devices for recognizing the performance of a string instrument are implemented in a form mounted on the bow, which not only increases the weight of the bow but also changes the center of gravity of the bow, which can interfere with the performance.

Various embodiments of the present invention are directed to recognizing stringed musical performances using an electronic device mounted on a stringed instrument and providing various kinds of feedback to the user using the obtained performance data.

An electronic device according to various embodiments of the present invention includes an image sensor that senses the movement of a bow relative to the stringed instrument, a vibration sensor that senses vibrations occurring in the stringed instrument, And a control module for determining a fingering position of the user with respect to the stringed instrument.

An electronic device according to various embodiments of the present invention includes a display, a communication module for receiving string musical performance data of a user from an external electronic device, analyzing a user's error pattern using the performance data, And to provide feedback to the user.

A method of recognizing stringed musical performances of an electronic device according to various embodiments of the present invention includes: detecting movement of a bow to the stringed instrument; sensing vibrations occurring in the stringed instrument; And determining the fingering position of the user with respect to the stringed instrument.

A stringed playing feedback method of an electronic device according to various embodiments of the present invention includes: receiving string performance data from an external electronic device; analyzing a user's error pattern using the performance data; and feedback / RTI >

According to various embodiments of the present invention, accurate string instrument performance data can be obtained while minimizing the weight change of the bow by acquiring string instrument performance data by using an apparatus attached to the string instrument, and processing the obtained performance data in a meaningful form Thereby providing various feedbacks to the user.

1 is a view showing the configuration of a stringed musical performance system according to various embodiments of the present invention.
2 is a diagram illustrating the structure of a first electronic device according to various embodiments of the present invention.
3 is a diagram illustrating the structure of a first electronic device according to various embodiments of the present invention.
4 is a block diagram illustrating the configuration of a first electronic device in accordance with various embodiments of the present invention.
5 is a view showing a structure and viewing angle of an image sensor according to an embodiment of the present invention.
6 is a view showing a structure and viewing angle of an image sensor according to an embodiment of the present invention.
7 is a view illustrating a viewing angle of a side image sensor according to an embodiment of the present invention.
8 is a view showing an element for determining the position and posture of the bow.
9 is a diagram showing an example of an infrared image generated by an image sensor.
10 is a diagram showing an example of an infrared image generated by an image sensor.
11 is a diagram showing an example of an infrared image generated by an image sensor.
12 is a diagram illustrating a pattern of hair hair according to various embodiments of the present invention.
13 is a diagram showing an example of an infrared image generated by an image sensor.
14 is a diagram showing an example of an infrared image generated by an image sensor.
Fig. 15 is a view showing an attachment pattern of the metal attached to the bow. Fig.
16 is a view showing an attachment position of the magnet attached to the bow.
17 is a block diagram illustrating the configuration of a second electronic device in accordance with various embodiments of the present invention.
18 is a diagram illustrating a user interface according to various embodiments of the present invention.
19 is a diagram illustrating a user interface according to various embodiments of the present invention.
20 is a diagram illustrating a user interface according to various embodiments of the present invention.
21 is a diagram illustrating a user interface according to various embodiments of the present invention.
22 is a diagram illustrating a user interface according to various embodiments of the present invention.
23 is a diagram illustrating a user interface according to various embodiments of the present invention.
24 is a flowchart showing a method of recognizing stringed musical performances of an electronic device according to various embodiments of the present invention.
25 is a flow diagram illustrating a stringed playing feedback method of a second electronic device in accordance with various embodiments of the present invention.

Hereinafter, various embodiments of the present document will be described with reference to the accompanying drawings. It should be understood, however, that this invention is not intended to be limited to the particular embodiments described herein but includes various modifications, equivalents, and / or alternatives of the embodiments of this document . In connection with the description of the drawings, like reference numerals may be used for similar components.

In this document, the expressions "having," " having, "" comprising," or &Quot;, and does not exclude the presence of additional features.

In this document, the expressions "A or B," "at least one of A or / and B," or "one or more of A and / or B," etc. may include all possible combinations of the listed items . For example, "A or B," "at least one of A and B," or "at least one of A or B" includes (1) at least one A, (2) Or (3) at least one A and at least one B all together.

As used herein, the terms "first," "second," "first," or "second," and the like may denote various components, regardless of their order and / or importance, But is used to distinguish it from other components and does not limit the components. For example, the first user equipment and the second user equipment may represent different user equipment, regardless of order or importance. For example, without departing from the scope of the rights described in this document, the first component can be named as the second component, and similarly the second component can also be named as the first component.

(Or functionally or communicatively) coupled with / to "another component (eg, a second component), or a component (eg, a second component) Quot; connected to ", it is to be understood that any such element may be directly connected to the other element or may be connected through another element (e.g., a third element). On the other hand, when it is mentioned that a component (e.g., a first component) is "directly connected" or "directly connected" to another component (e.g., a second component) It can be understood that there is no other component (e.g., a third component) between other components.

As used herein, the phrase " configured to " (or set) to be "configured according to circumstances may include, for example, having the capacity to, To be designed to, "" adapted to, "" made to, "or" capable of ". The term " configured to (or set up) "may not necessarily mean" specifically designed to "in hardware. Instead, in some situations, the expression "configured to" may mean that the device can "do " with other devices or components. For example, a processor configured (or configured) to perform the phrases "A, B, and C" may be implemented by executing one or more software programs stored in a memory device or a dedicated processor (e.g., an embedded processor) , And a generic-purpose processor (e.g., a CPU or an application processor) capable of performing the corresponding operations.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of the other embodiments. The singular expressions may include plural expressions unless the context clearly dictates otherwise. Terms used herein, including technical or scientific terms, may have the same meaning as commonly understood by one of ordinary skill in the art. The general predefined terms used in this document may be interpreted in the same or similar sense as the contextual meanings of the related art and, unless expressly defined in this document, include ideally or excessively formal meanings . In some cases, even the terms defined in this document can not be construed as excluding the embodiments of this document.

Electronic devices (e.g., first electronic device 100 and second electronic device 200) according to various embodiments of the present document may be, for example, a smartphone, a tablet personal computer, Such as a mobile phone, a video phone, an e-book reader, a desktop personal computer, a laptop personal computer, a netbook computer, a workstation, A personal digital assistant (PDA), a portable multimedia player (PMP), an MP3 player, a mobile medical device, a camera, or a wearable device. According to various embodiments, the wearable device may be of the accessory type (e.g., a watch, a ring, a bracelet, a bracelet, a necklace, a pair of glasses, a contact lens or a head-mounted-device (HMD) (E. G., Electronic apparel), a body attachment type (e. G., A skin pad or tattoo), or a bioimplantable type (e.g., implantable circuit).

In some embodiments, the electronic device may be a home appliance. Home appliances include, for example, televisions, digital video disc (DVD) players, audio, refrigerators, air conditioners, vacuum cleaners, ovens, microwaves, washing machines, air cleaners, set- Such as a home automation control panel, a security control panel, a TV box such as Samsung HomeSync ™, Apple TV ™ or Google TV ™, a game console such as Xbox ™, PlayStation ™, , An electronic key, a camcorder, or an electronic frame.

In an alternative embodiment, the electronic device may be any of a variety of medical devices (e.g., various portable medical measurement devices such as a blood glucose meter, a heart rate meter, a blood pressure meter, or a body temperature meter), magnetic resonance angiography (MRA) Navigation systems, global navigation satellite systems (GNSS), event data recorders (EDRs), flight data recorders (FDRs), infotainment (infotainment) systems, ) Automotive electronic equipment (eg marine navigation systems, gyro compass, etc.), avionics, security devices, head units for vehicles, industrial or home robots, automatic teller's machines (ATMs) Point of sale, or internet of things (eg, light bulbs, various sensors, electrical or gas meters, sprinkler devices, fire alarms, thermostats, street lights, Of the emitter (toaster), exercise equipment, hot water tank, a heater, boiler, etc.) may include at least one.

According to some embodiments, the electronic device is a piece of furniture or a part of a building / structure, an electronic board, an electronic signature receiving device, a projector, Water, electricity, gas, or radio wave measuring instruments, etc.). In various embodiments, the electronic device may be a combination of one or more of the various devices described above. An electronic device according to some embodiments may be a flexible electronic device. Further, the electronic device according to the embodiment of the present document is not limited to the above-described devices, and may include a new electronic device according to technological advancement.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS An electronic apparatus according to various embodiments will now be described with reference to the accompanying drawings. In this document, the term user may refer to a person using an electronic device or a device using an electronic device (e.g., an artificial intelligence electronic device).

1 is a view showing the configuration of a stringed musical performance system according to various embodiments of the present invention.

Referring to FIG. 1, a string playing system may include a first electronic device 100, a second electronic device 200, a third electronic device 300, and a server 400. The first electronic device 100, the second electronic device 200, the server 300, and the third electronic device 300 may be networked and communicate with each other. For example, the first electronic device 100, the second electronic device, and the third electronic device 300 may be connected by a short-range wireless communication technology such as bluetooth, near field communication (NFC), zigbee, have. In another example, the server 400 may be connected to the second electronic device 200 or the third electronic device 300 via the Internet or mobile communication network.

According to various embodiments of the present invention, the first electronic device 100 may sense performance data that occurs in accordance with a user's string performance. The stringed instrument 10 may be, for example, a stringed instrument that is played using a bow 20. Various embodiments of the present invention may be applied to all string instruments played using an arc, but for the sake of convenience of description, the string instrument 10 will be described as an example of a violin. The performance data may include, for example, a pitch, a sound intensity, a rhythm, a longitudinal position of the bow, a lateral position of the bow, a relative tilt of the bow and string, The inclination of the bow to the body direction of the string, the type of string the bow is in contact with, the fingering position of the user, and the speed of the bow. According to one embodiment, the first electronic device 100 can be implemented in a structure that can be attached (or fastened) to the stringed instrument 10. According to one embodiment, the first electronic device 100 may transmit performance data to the second electronic device 200. [ For example, the first electronic device 100 may transmit performance data in the form of MIDI (music instrument digital interface) or music XML (extensible markup language).

According to one embodiment, the second electronic device 200 may be a portable electronic device such as a smart phone, a tablet PC, or the like, or a wearable electronic device such as a smart watch, smart glass, or the like. According to one embodiment, the second electronic device 200 may receive performance data of the user from the first electronic device 100. According to one embodiment, the second electronic device 200 can compare the performance data with the score data to determine the user's performance results (e.g., whether they played normally or an error occurred in performance). According to one embodiment, the second electronic device 200 may determine a user's performance result in real time and provide the user with feedback corresponding to the performance result. According to one embodiment, the second electronic device 200 may determine the performance pattern of the user (normal performance pattern and error pattern) based on the performance results. For example, the second electronic device 200 may analyze a user's performance pattern using a pattern analysis algorithm. According to one embodiment, the second electronic device 200 may determine the user's performance pattern in real time and provide the user with real-time feedback related to the error pattern.

According to one embodiment, the second electronic device 200 may transmit the performance data of the user, the performance results, and the performance pattern to the server 400. According to one embodiment, the second electronic device 200 may provide feedback corresponding to the normal playing pattern and the error pattern of the user.

According to one embodiment, the third electronic device 300 may be a wearable electronic device such as a smart watch, smart glass, or the like. According to one embodiment, the third electronic device 300 may receive and provide the user's performance results, performance patterns and feedback from the second electronic device 200 to the user.

According to one embodiment, the server 400 may store score data, a normal performance pattern, and an error pattern in the form of a database. The normal performance pattern database includes, for example, a finger number for playing each pitch from the score data, a finger position, a string number, a rhythm, the number of times each finger is used, the number of times each string is played, (Eg, a unit of measure or a multiple of a measure), and a similar performance pattern (eg, a performance measure) As shown in FIG. According to one embodiment, when a new performance pattern is analyzed, the server 400 may update the stored normal performance pattern database. The error pattern database compares, for example, performance data by a plurality of users with score data to determine a portion where a performance error has occurred, analyzes a portion where an error has occurred in a specific unit (e.g., a unit of measure) And can be generated by classifying by error pattern. According to one embodiment, when a new error pattern is analyzed, the server 400 may update the stored error pattern database.

According to one embodiment, the server 400 may receive and store at least one of the user's performance data, performance results and normal performance patterns and error patterns from the second electronic device 200. [ The server 400 may store the frequency of occurrence of user performance data, performance results, performance patterns, error patterns, and error patterns for each user. According to one embodiment, the server 400 may send at least one of a performance result, a normal performance pattern, and an error pattern based on past performance data of the user's past performance data to the second electronic device 200, (200).

2 is a diagram illustrating the structure of a first electronic device according to various embodiments of the present invention.

Figure 2 (a) shows a top view of the first electronic device 100. Referring to FIG. 2 (a), the first electronic device 100 may include a body portion 101 and a fastening portion 103. According to one embodiment, the fastening portion 103 may extend from both sides of the body portion 101. [ According to one embodiment, the body 101 may include an image sensor 111 on its upper surface.

FIG. 2 (b) shows a front view of the first electronic device 100. Referring to FIG. 2 (b), the fastening portion 103 may extend from both sides of the body 101 and may have a bent shape from the end toward the body 101. According to one embodiment, the body portion 101 may include a vibration detection sensor 113 on a lower surface thereof. The first electronic device 100 may be attached to the stringed instrument 10 such that the bottom surface of the body portion 101 faces the stringed instrument 10 so that the vibration sensing sensor 113 is attached to the stringed instrument 10 Can be contacted directly.

Fig. 2 (c) shows a perspective view with the first electronic device 100 attached to the stringed instrument 10. 2 (c), the body 101 may be attached between the finger board 11 and the bridge 13 of the stringed instrument 10. According to one embodiment, the first electronic device 100 may have a portion of the body portion 101 attached between the fingerboard 11 and the top 15 of the stringed instrument 10. According to one embodiment, the fastening portion 103 may be fastened to the c-bout 17 of the stringed instrument 10 to secure the first electronic device 100 to the stringed instrument 10.

3 is a diagram illustrating the structure of a first electronic device according to various embodiments of the present invention.

3 (a) shows a top view of the first electronic device 100. FIG. Referring to FIG. 3 (a), the first electronic device 100 may include a slit 105 in a linear shape on one side. According to one embodiment, the slit 105 may be formed from one side of the first electronic device 100 to the opposite side. According to one embodiment, the body 101 may include an image sensor 111 on its upper surface.

FIG. 3 (b) shows a side view of the first electronic device 100. Referring to FIG. 3 (b), according to one embodiment, the first electronic device 100 may include a vibration sensing sensor 113 on the underside. The first electronic device 100 can be attached to the stringed instrument 10 in a manner that the bottom surface faces the stringed instrument 10 so that the vibration sensing sensor 113 can directly contact the stringed instrument 10 .

Fig. 3 (c) shows a perspective view with the first electronic device 100 attached to the stringed instrument 10. Referring to FIG. 3C, the first electronic device 100 includes a first electronic device 100 having a bridge of stringed instruments 10 inserted into the slit 105 of the first electronic device 100, .

4 is a block diagram illustrating the configuration of a first electronic device in accordance with various embodiments of the present invention.

4, the first electronic device 100 includes a sensor module 110, a communication module 120, an audio module 130, a power management module 140, a battery 150, and a control module 160 can do.

According to one embodiment, the sensor module 110 may sense the movement of a bow to a stringed instrument and the vibrations occurring in the stringed instrument. According to various embodiments, the sensor module 110 includes an image sensor 111, a vibration sensing sensor 113, a metal sensing sensor 115, a magnetic field sensor 117, an inertia measurement unit 118, A sensor 119 may be included.

According to one embodiment, the image sensor 111 may sense the movement of a bow relative to the stringed instrument 10. According to one embodiment, the image sensor 111 may be located on the top surface of the first electronic device 100 to sense an infrared image of the bow located between the fingerboard 11 of the stringed instrument 10 and the bridge 13. [ have. According to one embodiment, the image sensor 111 may generate an infrared image by emitting an infrared signal and receiving an infrared signal reflected by the bow 20 (or hair).

According to one embodiment, the image sensor 111 may be implemented as an array image sensor (or two-dimensional image sensor). For example, the image sensor 111 can sense a two-dimensional area between the base plate 11 of the stringed instrument 10 and the bridge 13. [

5 is a view showing a structure and viewing angle of an image sensor according to an embodiment of the present invention.

5 (a) shows a side section of the first electronic device with the first electronic device 100 attached to the stringed instrument. Referring to FIG. 5A, the image sensor 111 is positioned between the fingerboard 19 and the bridge 13 of the string to generate a two-dimensional infrared image in the upward direction of the image sensor. According to one embodiment, the image sensor 111 may include a transmit module 31, a receive module 32, and an infrared filter 33. The transmitting module 31 can transmit an infrared signal. For example, the transmitting module 31 may include a light emitting diode (LED) module for generating an infrared signal. According to one embodiment, the transmitting module 31 may be located on the outer surface of the image sensor 111. [ The receiving module 32 may receive an infrared signal reflected from the bow 20 (or haircut) of the infrared signal transmitted from the transmitting module 31. [ According to one embodiment, the receiving module 32 may be located at the center of the image sensor 111. According to one embodiment, the receiving module 32 may include a photodiode capable of detecting an infrared signal, and the photodiode may be arranged two-dimensionally within the receiving module 32. According to one embodiment, the infrared filter 33 may be located above the receiving module 32 and may pass only infrared signals among the signals incident on the image sensor 111 and filter the remaining signals (e.g., visible light) . Accordingly, the receiving module 32 can receive only the infrared signal.

5 (b) and 5 (c) show viewing angles of the image sensor 111 viewed from the side and upper surface of the first electronic device 100, respectively. Referring to FIGS. 5 (b) and 5 (c), the image sensor 111 may have a viewing angle that spreads upward. Accordingly, the image sensor senses the area including the string between the fingerboard 11 and the bridge 13, and can detect the movement of the bow which is in contact with the string between the fingerboard 11 and the bridge 13. [

According to one embodiment, the image sensor 111 may be implemented with a line image sensor. For example, the line image sensor may sense a line in the current direction of the stringed instrument 10. According to one embodiment, the line image sensor is capable of sensing a plurality (e.g., two) of lines in the current direction of the stringed instrument 10.

6 is a view showing a structure and viewing angle of an image sensor according to an embodiment of the present invention.

6 (a) shows a top view of the first electronic device with the first electronic device 100 attached to the stringed instrument. Referring to FIG. 6A, the image sensor 111 may be positioned between the fingerboard 19 and the bridge 13 of the string to generate a one-dimensional infrared image in the upper direction of the image sensor. According to one embodiment, the image sensor 111 may include a transmitting module 34 and a receiving module 35. [ The transmitting module 34 can transmit an infrared signal. For example, the transmitting module 34 may include a light emitting diode (LED) module for generating an infrared signal. According to one embodiment, the transmitting module 31 may be located at the center of the image sensor 111 in the form of a line in the direction of the current 19. The receiving module 35 can receive the infrared signal reflected from the bow among the infrared signals transmitted from the transmitting module 34. [ According to one embodiment, the receiving module 35 may be located on the left and right sides of the transmitting module 34. According to one embodiment, the receiving module 35 may comprise a photodiode capable of detecting an infrared signal, and the photodiode may be arranged one-dimensionally in the receiving module 35 in the direction of the current 19 . According to one embodiment, the receiving module 35 may comprise a low pass filter. The receiving module 35 may filter the remaining signals (e.g., visible light) except the infrared signal from the analog signal detected by the photodiode using a low-pass filter.

6B is a view showing the viewing angle of the image sensor 111 viewed from the front of the first electronic device 100. As shown in FIG. Referring to FIG. 6 (b), the image sensor 111 may have a viewing angle of a shape spreading upward. Accordingly, the image sensor 111 senses two lines in the present direction, and can sense the movement of the bow 20 in contact with the strings between the base plate 11 and the bridge 13. [

According to one embodiment, the sensor module 110 may include an image sensor 111 that senses an infrared image in a lateral direction of the first electronic device 110.

7 is a view illustrating a viewing angle of a side image sensor according to an embodiment of the present invention.

Referring to FIG. 7, the lateral image sensor 111 may include a receiving module 36. [0030] The receiving module 36 can receive an infrared signal emitted from a hairball (frog) 23 of the bow 20. According to one embodiment, the bow 20 may include a transmitting module 23 for transmitting an infrared signal. According to one embodiment, the transmitting module 41 may be located in the hair clipper 23. According to one embodiment, the transmitting module 41 may include at least one light emitting diode (LED) 43 for generating an infrared signal. For example, the transmitting module 41 may include two LEDs 43 arranged vertically in the hair clipper 23. According to one embodiment, the receiving module 36 may include a photodiode capable of detecting an infrared signal, and the photodiode may be arranged two-dimensionally within the receiving module 36. [ Although not shown in FIG. 7, the image sensor 111 in the lateral direction may include an infrared filter. The infrared filter can pass only the infrared signal among the signals incident on the image sensor 111 and filter the remaining signals (e.g., visible light). Accordingly, the receiving module 36 can receive only the infrared signal.

According to one embodiment, the lateral image sensor 111 may include a transmission module for transmitting an infrared signal. When the side image sensor 111 includes a transmitting module, the LED 43 located in the hair pin holder 23 may be implemented as a reflector reflecting the infrared signal. Accordingly, the receiving module 36 can receive the infrared signal reflected by the reflection plate located on the hair clipper 23 among the infrared signals transmitted from the transmitter module. There is an advantage that the weight distribution of the bow and the overall weight change of the bow are hardly generated when the reflector is attached to the hair clipper 23. However, as compared with the case where the LED 43 is attached, The amount of signal may increase and the power consumption of the first electronic device 100 may increase.

The vibration detection sensor 113 can sense the vibration (or sound) generated in the string instrument 10. [ For example, the vibration detection sensor 113 may include a piezo sensor. The vibration sensor senses vibration generated in the stringed instrument 10 and converts the vibration into an electrical signal.

The metal sensing sensor 115 may sense the metal located around the first electronic device 100. According to one embodiment, the metal sensing sensor 115 may sense movement of a metal (e.g., aluminum) attached to the rib. For example, the coil included in the metal detection sensor 115 may cause an impedance change due to the movement of the metal attached to the metal strip, and the metal detection sensor 115 may sense the impedance change of the coil. According to one embodiment, the metal sensing sensor 115 may sense a plurality of (e.g., two) areas separated by a specified distance. According to one embodiment, the sensor module 110 may include a plurality of metal sensing sensors 115. The plurality of metal detection sensors 115 may be spaced apart from each other by a predetermined distance to sense a different area. According to one embodiment, the metal sensing sensor 115 may include a plurality of coils capable of sensing a different area, respectively, at a specified distance apart.

The magnetic field sensor 117 may sense a change in the magnetic field around the first electronic device 100. According to one embodiment, the magnetic field sensor 117 may sense a magnetic field change due to the movement of a magnet attached to the barb.

The inertial sensor 118 may sense movement of the stringed instrument 10. According to one embodiment, the inertial sensor 118 may include an acceleration sensor and a gyro sensor. The acceleration sensor can sense the acceleration of the stringed instrument 10. For example, the acceleration sensor may sense the acceleration of the stringed instrument 10 and output an acceleration value of the stringed instrument 10 in three axis directions (e.g., x, y and z axis). The gyro sensor is capable of sensing the rotational angular velocity of the string instrument 10. For example, the gyro sensor may sense the angular velocity of the stringed instrument 10 and output the angular velocity of the stringed instrument 10 in three directions (e.g., x, y, and z).

Proximity sensor 119 may determine if an object is approaching within a certain distance. The proximity sensor 119 may sense the area between the base plate 11 and the bridge 13 of the stringed instrument 10 to determine whether the bow 20 has contacted the strings.

The communication module 120 may communicate with the second electronic device 200. For example, the communication module 120 may communicate with the second electronic device 200 using a short-range wireless communication technology such as Bluetooth, NFC or ZigBee. According to one embodiment, the communication module 120 may transmit performance data to the second electronic device 200. According to one embodiment, the communication module 120 may receive information about the movement of the bow 20 from an electronic device attached to the bow when the bow 20 is attached with an inertial sensor.

The audio module 130 may, for example, generate an audio signal. According to one embodiment, the audio module 130 may generate an audio signal using the vibration of the stringed instrument 10 sensed by the vibration sensing sensor 113. The audio module 130 may output an audio signal through an audio interface capable of connecting a speaker or earphone (or headphone). According to one embodiment, the audio module 130 may provide sound effects (e.g., sound field, distortion, etc.) to the audio signal through reverberation, delay, and equalizer operations.

The power management module 140 may manage the power of the first electronic device 100. For example, the power management module 140 may utilize the battery 150 to power or block components of the first electronic device 100. According to one embodiment, the power management module 140 may include a power management integrated circuit (PMIC). According to one embodiment, the power management module 140 may measure the remaining amount of the battery 150, the voltage during charging, the current, or the temperature. According to one embodiment, the battery 150 may include, for example, a rechargeable battery and / or a solar battery. According to one embodiment, the power management module 140 may block power supplied to the sensor module 110 when the amount of light measured by the image sensor 111 is less than a specified reference.

According to one embodiment, the control module 160 may analyze the movement of the bow sensed by the sensor module 110 and the vibrations of the strings to generate performance data. The performance data may include, for example, a pitch, a sound intensity, a rhythm, a longitudinal position of the bow, a lateral position of the bow, a relative tilt of the bow and string, The inclination of the bow to the body direction of the string, the type of string the bow is in contact with, the fingering position of the user, and the speed of the bow.

8 is a view showing an element for determining the position and posture of the bow.

8, when a user plays a stringed instrument, the longitudinal position of the bow relative to the string, the lateral position of the bow, the relative angle (tilt) of the bow and string, The position and posture of the bow can be determined by the skewness of the bow and the inclination of the bow to the body direction of the string.

According to one embodiment, the control module 160 uses the sensed value of the image sensor 111 to determine the longitudinal position of the bow, the skewness of the bow toward the fingerboard, And the speed of the bow can be determined. According to one embodiment, the control module 160 may binarize the infrared image of the image sensor 111 and use the binarized image to determine the elements described above.

9 is a diagram showing an example of an infrared image generated by an image sensor.

According to one embodiment, the control module 160 may determine the longitudinal position using the infrared image of the array image sensor. 9 (a) and 9 (b), the area reflected by the bow may appear as a bright color in the infrared image, and the area not reflected by the bow may appear as a dark color in the infrared image. If the infrared image is projected in the horizontal direction (or in the vertical direction of the string), a graph 51 or 52 showing the distribution of bright color pixels as shown on the right side can be obtained. The x-axis of the graphs 51 and 52 represents the position in the fingerboard direction, and the y-axis represents the cumulative value of bright pixels at a specific position. According to one embodiment, the control module 160 may determine a point where the cumulative value of the bright color pixels is the maximum value in the graphs 51 and 52 as the position of the finger's fingerboard direction. In another example, the control module 160 may determine a point corresponding to the average value of the bright pixels as the position of the finger in the fingerboard direction.

According to one embodiment, the control module 160 may indicate the position of the bow in the fingerboard direction relative to the midpoint between the fingerboard and the bridge. According to one embodiment, the position of the bow in the direction of the fuselage may have a positive value when the bow is close to the fingerboard, and a value of - when the bow is close to the bridge. As shown in FIG. 9 (a), when the bow is shifted in the direction of the back plate, the control module 160 can determine the position of the bow in the direction of the back plate as +10 mm. 9 (b), the control module 160 can determine the position of the bow in the direction of the fingerboard to be 0 when the bow is positioned at the middle of the fingerboard and the bridge.

10 is a diagram showing an example of an infrared image generated by an image sensor.

According to one embodiment, the control module 160 may use the infrared image of the array image sensor to determine the skewness of the bow toward the fingerboard. According to one embodiment, the control module 160 can determine the angle between the vertical direction of the string and the bow as an angle to the direction of the fingerboard. According to one embodiment, the control module 160 may determine the center position 55,57 of the bow in the vertical direction (x-axis) of the string. According to one embodiment, the control module 160 may determine the tilt angle of the center position of the bow as an angle in the direction of the finger's fingerboard. When the central position 55 of the bow in the vertical direction of the string is the same as shown in FIG. 10 (a), the control module 160 can determine the angle of the bow in the direction of the fingerboard to 0 °. 10 (b), when the center position 57 of the bow increases from the fourth string to the first string, the control module 160 can determine the angle of the bow to the fingerboard as +10 .

11 is a diagram showing an example of an infrared image generated by an image sensor.

According to one embodiment, the control module 160 may use an infrared image of the array image sensor to determine an inclination of the bow to the string body direction. Referring to Figures 11 (a) to 11 (d), the area reflected by the bow may appear as a bright color in the infrared image, and the area not reflected by the bow may appear as a dark color in the infrared image. According to one embodiment, the thickness of the bow that appears in the infrared image in the vertical direction of the string can vary. For example, even if the bows have the same thickness, a portion closer to the image sensor 111 may be thicker depending on the perspective, and the thickness may be thinner as the distance from the image sensor 111 increases. When the infrared image is projected in the vertical direction (or in the horizontal direction of the string), it is possible to obtain a graph 61, 63, 65, 67 showing the distribution of bright color pixels as in the lower side of the infrared image. The x-axis of the graphs 61, 63, 65, and 67 represents the position of the string in the horizontal direction, and the y-axis represents the cumulative value of bright color pixels at a specific position. According to one embodiment, the control module 160 may determine the slope of the cumulative value of the bright color pixels using the graphs 61, 63, 65, and 67. The control module 160 may determine an inclination of the bow in the string body direction based on the slope value.

According to one embodiment, the control module 160 may determine the lateral position of the bow using the infrared image of the array image sensor. For example, the control module 160 can determine the bow position of the bow and the speed (direction and speed) of the bow using the pattern of hairclips contained in the infrared image.

12 is a diagram illustrating a pattern of hair hair according to various embodiments of the present invention.

The pattern of the hair bulb 21 may be formed by staining a part of the hair bulb 21 with a color (for example, black) that is in contrast to the basic color (e.g. white) of the hair bulb 21. Referring to FIG. 12 (a), the hair 21 included in the bow can be separated into two lines 71 and 72 in the bow direction. The two lines 71 and 72 may each have a different pattern. For example, the first line 71 may be bisected and dyed black on one side. The second line 72 may be quadrisected and patterned so that black and white are repeated. Referring to Fig. 14 (b), the hair 21 included in the bow can be separated into three lines 73, 74 and 75 in the bow direction. The three lines 73, 74 and 75 may each have a different pattern. For example, the first line 73 may be bisected and dyed black on one side. The second line 74 may be quadrisected and patterned so that black and white are repeated. The third line 75 may be divided into four equal parts, so that black and white are repeated.

According to one embodiment, the control module 160 may analyze the pattern of the plurality of lines contained in the hair bulb 21 to recognize the lateral position of the bow. According to one embodiment, as the number of lines included in the hair bulb 21 increases, the accuracy of the lateral position of the bow can be improved. According to one embodiment, the control module 160 may analyze pattern changes of a plurality of lines contained in the hair bulb 21 to determine whether the bow 20 is moving in the head direction or in the hair bulb direction have. According to one embodiment, the control module 160 can determine the speed at which the bow moves by using the pattern change rate of a plurality of lines included in the hair bulb 21.

13 is a diagram showing an example of an infrared image generated by an image sensor.

According to one embodiment, the control module 160 may use the infrared image of the line image sensor to determine the skewness of the string bow of the bow toward the fingerboard and the inclination towards the bow of the bow . The infrared image generated from the line image sensor may include, for example, two lines in the horizontal direction of the strings as shown in Figs. 13 (a) to 13 (d).

Referring to Figures 13 (a) and 13 (b), the area reflected by the bow may appear bright in the infrared image, and the area not reflected by the bow may appear dark in the infrared image. The control module 160 may determine the center position of the bright color pixel for each of the two lines. The control module 160 can determine the angle of the bow in the direction of the fingerboard using the distance between the two line sensors and the distance between the two center positions. For example, the control module 160 determines the angle of the bow in the direction of the fingerboard to + 10 degrees with respect to the image shown in Fig. 13 (a) Can be determined to be -10 deg.

13 (c) and 13 (d), the light amount of the infrared signal received by the image sensor of the bow may be changed according to the distance between the image sensor 111 and the bow. That is, when the bow 20 is close to the image sensor 111, it can be displayed brightly on the infrared image, and it can be displayed darker on the farther side from the image sensor 111. For example, when the head portion is closer to the image sensor 111 than the frog portion of the bow, the bow included in the left image sensor is brightly displayed as shown in FIG. 13 (c) The arrows shown on the display may be dimmed. In another example, when the frog portion of the bow is closer to the image sensor 111 than the head portion, the bow included in the right image sensor is brightly displayed as shown in FIG. 13 (d) The bow displayed on the sensor may be darkened. According to one embodiment, the control module 160 may determine the inclination toward the body of the bow using the brightness (or brightness difference) of the pixels included in the line image sensor.

14 is a diagram showing an example of an infrared image generated by an image sensor.

According to one embodiment, the control module 160 uses the infrared image of the side image sensor to determine the skewness of the string bow of the bow in the direction of the fingerboard, the inclination of the bow toward the body direction, The lateral position can be determined. The infrared image generated from the lateral image sensor may include a plurality of (e.g., two) points in the two-dimensional image, for example, as shown in Figs. 14 (a) to 14 (f). The plurality of points may be an infrared signal received from an infrared transmitter module attached to the hair clipper 21 or an infrared signal reflected by a reflector attached to the hair clipper 21 as described with reference to FIG.

According to one embodiment, the control module 160 may determine the bow position of the bow using the distance between the plurality of points included in the infrared image or the size of the plurality of points. For example, the control module 160 can determine that the bow 20 is close to the stringed instrument 10 when the distance between the two points is large or the size of the point is large as shown in FIG. 14 (a) As shown in (b) of Fig. 14, if the distance between the two points is close or the size of the point is small, it can be determined that the bow is relatively far away.

According to one embodiment, the control module 160 may determine the skewness of the bow in the direction of the fingerboard using the lateral position of the plurality of points included in the infrared image. For example, the control module 160 may determine the angle of the bow to the fingerboard using the lateral position of the infrared point in a state where the longitudinal position of the bow and the lateral position of the bow are determined skewness. For example, the control module 160 determines the angle of the bow in the direction of the fingerboard with respect to the image shown in (c) of FIG. 14 as +10 degrees, Can be determined to be -10 deg.

According to one embodiment, the control module 160 may determine the inclination in the body direction of the bow using the longitudinal position of the plurality of points included in the infrared image. For example, the control module 160 determines the angle of the bow in the direction of the body of the bow using the longitudinal position of the infrared point in a state in which the bow 20 is in contact with the string and the lateral position of the bow is determined, it is possible to judge the inclination. Whether or not the bow 20 is in contact with the string can be judged by using the proximity sensor 119. For example, the control module 160 determines that the angle of the bow in the direction of the fingerboard is + 10 degrees with respect to the image shown in (e) of Fig. 14, Can be determined to be -10 deg.

According to one embodiment, the control module 160 may determine the relative tilt of the bow and string using the slopes of the plurality of points included in the infrared image. For example, when the inclination of a plurality of points is infinite as shown in Figs. 14A to 14E (i.e., when all the hair is in contact with the strings), the control module 160 determines the relative angle Quot; 0 ". The control module 160 determines that the bow 20 is tilted to the right when the inclination of the plurality of points is negative as shown in FIG. 14 (g), and determines that the relative angle of the bow and the bow is +20 . The control module 160 determines that the bow 20 is inclined to the left when the inclination of the plurality of points is positive as shown in FIG. 14 (g), and determines that the relative angle between the bow and the bow is -20 have.

According to one embodiment, the control module 160 may use the sensed value of the metal sensing sensor 115 to determine the speed (direction and speed) of the bow.

Fig. 15 is a view showing an attachment pattern of the metal attached to the bow. Fig.

Referring to FIG. 15, the bow 20 may include a metal 71 having a specific pattern. According to one embodiment, the metal 71 may be attached to a stick 25. According to one embodiment, the metal 71 may be attached to have a periodic pattern. For example, the metal 71 may be attached so that the region where the metal material 71 is attached along the band 25 and the region where the metal material 71 is not attached are periodically repeated. According to one embodiment, the length of the area to which the metal material 71 is attached and the area where the metal material 71 is not attached is shorter than the length of the area where the metal detection sensor 115 can sense (e.g., between two) May be set differently from the interval. According to one embodiment, the control module 160 determines whether the bow 20 is moving in the head direction or in the bulb holder direction using the detection of a plurality of regions of the metal sensing sensor 115 Can be determined. According to one embodiment, the control module 160 may use the metal sensing period of the metal sensing sensor 115 to determine the speed at which the bow moves.

According to one embodiment, the control module 160 may use the sensed value of the magnetic field sensor 117 to determine the lateral position of the bow and the speed (direction and speed) of the bow.

16 is a view showing an attachment position of the magnet attached to the bow.

Referring to FIG. 16, the bow 20 may include at least one magnet 73. According to one embodiment, the magnet 73 may be attached to a stick 25. According to one embodiment, the position at which the magnet 73 is attached may be determined according to the number of the magnets 73. [ For example, if there are three magnets 73 attached to the barb 25, each of the three magnets may be affixed to a position that equally divides the entire arc length. According to one embodiment, at least one magnet 73 attached to the barb 25 may be arranged so that the direction of the magnetic field is different. For example, when there are three magnets 73 attached to the ribs 25, the N poles of the three magnets may be arranged to face the x axis, the y axis, and the z axis, respectively. The magnetic field formed by the three magnets may be measured differently depending on the position of the barb 25 and the control module 160 analyzes the magnetic field sensed by the magnetic field sensor 117 to determine the lateral position of the bow position. According to one embodiment, the control module 160 may use a change in the magnetic field sensed by the magnetic field sensor 117 to determine whether the bow 20 is moving in the head direction or in the hair clip direction . According to one embodiment, the control module 160 may use the rate of change of the magnetic field sensed by the magnetic field sensor 117 to determine the speed at which the bow moves.

According to one embodiment, the control module 160 controls the movement of the string 10 sensed by the inertia measurement unit 118 and the movement of the bow sensed by the inertial sensor attached to the bow 20 The longitudinal position of the bow, the lateral position of the bow, the relative tilt of the bow and string, the skewness of the bow toward the fingerboard, and the bow direction of the string And the speed of the bow can be determined.

According to one embodiment, the control module 160 may use an inclination of the bow in the direction of the string body to determine the bow that the bow is in contact with. For example, the control module 160 determines that the bow is in contact with the first string when the angle of the bow to the stringed instrument body falls within the first range, and when the angle falls within the second range, If it falls within the third range, it is judged that it is in contact with the third string. If it is within the fourth range, it can be judged that it is in contact with the fourth string.

According to one embodiment, the control module 160 may analyze the vibration sensed by the vibration sensing sensor 113 to determine the pitch, the sound intensity, and the rhythm. For example, the pitch can be judged by the frequency of the vibration, the intensity of the sound can be judged by the amplitude of the vibration, and the rhythm can be judged by the timing at which the vibration is sensed.

According to one embodiment, the control module 160 may improve the reliability of pitch determination using information about the strings that the bow contacts. For example, the vibration sensed by the vibration sensor 113 may have a plurality of partial sounds as compound sounds. The vibration may include a harmonic having a fundamental frequency and an integer multiple of the fundamental frequency. When the vibration sensed by the vibration sensor 113 is converted from the time domain to the frequency domain, the strength (or level) of the frequency corresponding to the fundamental sound may be strongest. Accordingly, the frequency having the strongest intensity can be determined as the pitch of the vibration. However, if the harmonic intensity is higher than the fundamental sound, an octave error may be generated to determine that the harmonic is the fundamental sound. According to one embodiment, the control module 160 may determine whether the pitch determined by the vibration is a pitch that can occur in a bow-touched string. That is, the control module 160 can determine the pitch using a frequency component that may occur in a string where the bow contacts a plurality of frequency components included in the vibration. Accordingly, it is possible to prevent an octave error that may occur in the pitch determination process.

According to one embodiment, the control module 160 may apply a window function when converting the vibration sensed by the vibration sensor 113 to time domain to frequency domain to sense the pitch. For example, it is possible to filter only a vibration signal for a certain period of time necessary for determining the pitch of the vibration sensed by the vibration sensor 113, and convert it into a frequency domain. According to one embodiment, the control module 160 may set the time axis size of the window function differently depending on the type of string the bow contacts. According to one embodiment, the control module 160 sets the time axis size of the window function to a smaller value as the string corresponding to the treble (for example, string 1) You can set the time axis size of the window function to be larger. Accordingly, the time taken to determine the pitch and the data throughput can be reduced.

According to one embodiment, the control module 160 may determine the fingering position of the user based on the strings the pitch and bow touch. Due to the characteristics of stringed instruments, the same pitch can occur in different strings depending on the fingering position of the user. Therefore, when determining the fingering position of the user with only the pitch, it may not be possible to determine the accurate fingering position. According to one embodiment, the control module 160 may determine the fingertip position of the string where the bow is in contact, out of a plurality of fingertip positions corresponding to the pitch, as the fingertip position of the user. For example, the pitch determined by the control module 160 can be generated in the first string and the second string, and in the case where the strings in which the bow is in contact are string 1, the control module 160 generates the pitch Can be determined as the fingering position of the user. That is, the control module 160 can determine the position at which the pitch can be generated in the string where the bow contacts, as the fingering position of the user. Accordingly, even when there are a plurality of fingering positions having the same pitch, the fingering position of the user can be accurately determined.

17 is a block diagram illustrating the configuration of a second electronic device in accordance with various embodiments of the present invention.

17, the second electronic device 200 may include a communication module 210, an input module 220, a memory 230, a control module 240, a display 250 and an audio module 260 have.

The communication module 210 may communicate with the first electronic device 200, the third electronic device 300, and the server 400. The communication module 210 may communicate with the first electronic device 200 and the third electronic device 300 using a short range wireless communication technology such as, for example, Bluetooth, NFC or ZigBee. The communication module 210 can communicate with the server 400 through, for example, the Internet network or the mobile communication network.

According to one embodiment, the communication module 210 may receive performance data of the user from the first electronic device 200. The performance data may include, for example, a pitch, a sound intensity, a rhythm, a longitudinal position of the bow, a lateral position of the bow, a relative tilt of the bow and string, The inclination of the bow to the body direction of the string, the type of string the bow is in contact with, the fingering position of the user, and the speed of the bow.

According to one embodiment, the communication module 210 can transmit the occurrence data of the user, the performance result and the occurrence frequency of the normal performance pattern, the error pattern, and the error pattern to the server 400.

The input module 220 can receive user operations. According to one embodiment, the input module 220 includes a touch sensor panel that senses a touch operation of a user, a pen sensor panel that senses a pen operation of the user, a gesture sensor (or a motion recognition sensor) And a voice recognition sensor for recognizing the voice of the user.

According to one embodiment, the memory 230 may store user performance data received from the communication module 210. According to one embodiment, the memory 230 may store the performance result of the user determined by the performance result determination module 261. According to one embodiment, the memory 230 may store a pattern analysis algorithm. According to one embodiment, the memory 230 may store the performance pattern of the user determined by the pattern analysis algorithm. According to one embodiment, the memory 230 may store the score data.

The control module 240 may control the overall operation of the second electronic device 200. For example, the control module 240 may drive an operating system or application program (e.g., a string lesson application) so that the control module 240 may control a number of hardware or software components connected to it, Operation can be performed.

According to one embodiment, the control module 240 may include a performance result determination module 241 and a pattern analysis module 243.

According to one embodiment, the performance result determination module 241 can determine the performance technique that the user played using the performance data. For example, the performance result determination module 241 can determine which bow performance technique is used by using performance data related to the movement of the bow. The performance result judging module 241 can judge that the staccato method is used when the bow moves by more than 1/2 of the bow length within 1500 ms (millisecond). The performance result determination module 241 can determine that the slur or tie technique is used when two or more notes are played without changing the direction of the bow.

According to one embodiment, the performance result determination module 241 can compare the performance data with the score data to determine the performance result of the user in real time. For example, the performance result determination module 241 can determine whether the user has played to match the score data or a performance error has occurred. According to one embodiment, the performance result determination module 241 can determine the performance result of the user by dividing the performance into a pitch (or finger playing), a rhythm, and a bow performance. For example, the pitch can be determined by whether the pitch of the score data matches the pitch of the performance data (or within a specified error range). The rhythm can be judged as to whether or not the timings of sound generation of the score data and performance data coincide (or within a specified error range). It can be determined whether or not the performance data matches the movement or performance technique of the bow of the performance data (or within the specified error range).

According to one embodiment, the performance result determination module 241 may provide feedback on performance results. According to one embodiment, the performance result determination module 241 can provide error information and error correction information in real time when a performance error occurs. According to one embodiment, the performance result determination module 241 may provide feedback in the form of images or texts via the display 150, or may provide feedback in the form of speech in the audio module 260. According to one embodiment, the performance result determination module 241 can provide feedback by synthesizing the performance results of the user when the performance of the user is completed. For example, the performance result determination module 241 may provide feedback on performance results for each of the determination factors (e.g., pitch, rhythm, and bow performance). As another example, the performance result determination module 241 may provide feedback on a comprehensive performance result that includes the respective determination factors.

According to one embodiment, the pattern analysis module 243 may analyze the performance pattern of the user using the performance data of the user. The performance pattern of the user may include, for example, a normal performance pattern that the user is playing skillfully and an error performance pattern that is frequently mistaken by the user. For example, the pattern analysis module 243 may determine which fingers are frequently incorrect, which strings are frequently played with fingers, and which strings are often wrongly played. According to one embodiment, the pattern analysis module 243 may analyze the performance pattern of the user using the pattern analysis algorithm stored in the memory 230. [ According to one embodiment, the pattern analysis algorithm can learn a performance pattern using a normal performance pattern database and an error pattern database.

According to one embodiment, the pattern analysis module 243 may provide feedback related to the performance pattern of the user. According to one embodiment, the pattern analysis module 243 may provide feedback in the form of images or texts via the display 150, or may provide feedback in the form of speech in the audio module 260. According to one embodiment, the pattern analysis module 243 can analyze the error pattern by analyzing the performance result of the user determined by the performance result determination module 241 in real time. The pattern analysis module 243 can provide calibration information in real time on the analyzed error pattern in real time.

According to one embodiment, the pattern analysis module 243 may analyze a performance pattern by analyzing the entire performance result of the user when the performance of the user is completed. According to one embodiment, the pattern analysis module 243 may provide feedback related to the analyzed performance pattern (e.g., lecture content related to error patterns or error patterns) when the user's performance is complete. According to one embodiment, the pattern analysis module 243 may count the number of occurrences of the performance pattern and provide feedback based on the number of occurrences of the performance pattern. That is, the pattern analysis module 243 can provide feedback related to the performance pattern by considering the performance patterns analyzed in the past. Table 1 below shows examples of calibration information for error patterns and error patterns that the pattern analysis module 243 can analyze.

Error pattern Calibration information The overall tone increases after finger finger 4 You can go along the bridge to your thumb while finger fingers 4 times. Stretching of the hand is required. Watch the video and try stretching. Each time you turn on the lower part of the bow, the bow tilts toward you
- Wrist problems The bow tilts toward the body gradually.
Bend your wrists down a little and start boating. Each time you turn on the lower part of the bow, the bow tilts toward you
- When there is a force on the shoulder (upper arm) The bow tilts toward the body gradually.
As you lower your bow, you release your shoulders and stretch your arms while at the same time pulling your elbow outward to the halfway point of the bow. Each time you turn on the lower part of the bow, the bow tilts toward you
- Phlegm problems The bow tilts toward the body gradually. If you catch the bow incorrectly, the bow will move along the bow at the bottom of the bow. Be aware of the phasing method, remove your fingers and do Boeing Each time you turn on the lower part of the bow, the bow tilts toward you
- Problems of movement of arms (Habakkum) The bow tilts toward the body gradually.
From the point where the bow is halfway down, do not let your elbow go out any more, and you can pull the bow by moving the lower part of your arm. Bow tilts outward whenever you turn on the lower part of the bow
- Wrist / Habakkup / phage / supine The bow tilts toward the body gradually. Body posture is not straight Try playing with your waist open.
If you play G-string, sometimes you can play it down
If you lean back, it becomes difficult for A, E boing.

According to one embodiment, the control module 240 can request and receive the required data from the server 400 through the communication module 210 when the data required for the specific operation is not stored in the memory 230. [ For example, the control module 240 may request and receive the past performance data of the user, the performance result, the performance pattern, the content related to the performance pattern, and the like to the server 400.

According to one embodiment, the control module 240 may use the performance data to determine whether tuning of the strings is necessary. For example, if the frequency obtained from a sound required for an open-ended performance during a string instrument is compared with the theoretical frequency of the open-circuit, if the difference exceeds a certain value (for example, 5 Hz), it can be judged that tuning of the string is necessary. According to one embodiment, the control module 240 may notify the user via the display 250 or the audio module 260 when it is determined that tuning of the string is necessary. According to one embodiment, the control module 240 may display a user interface that allows the display 250 to select whether to enter the tuning mode when it is determined that tuning of the strings is necessary. According to one embodiment, when the user selects the tuning mode entry, the control module 240 may enter a tuning mode to display a user interface on the display 250 that may guide the tuning of the string.

According to one embodiment, the display 250 may display the user interface provided in the string lesson application. The user interface may include, for example, user performance results, error information, error correction information, recommended content, and lesson content. According to one embodiment, the user interface can provide the user's performance results in real time based on the performance data of the user. For example, the user interface can provide the user's fingertip position and movement of the bow in real time. According to one embodiment, the user interface may provide error information and error correction information according to the user's performance result in real time. According to one embodiment, the user interface may provide the recommended content and the lesson content based on the performance pattern and the error pattern of the user when the performance of the user is completed.

18 is a diagram illustrating a user interface according to various embodiments of the present invention.

Referring to FIG. 18, the display 250 may display a user interface including a user's real-time performance result, error information, and error correction information. According to one embodiment, the user interface may include a region (or a score region) 81 for displaying a score and a region (or a fingering region) 82 for visualizing and displaying the fingering position.

The score area 81 can display score data. According to one embodiment, the score area 81 may include an indicator 81A indicating the current performance position. The indicator 83 can be moved, for example, according to the passage of time. According to one embodiment, in the score area 81, error information and error correction information can be displayed when an error occurs in the performance of the user. For example, the pitch correction object 81B may be displayed in the score area 81 when the pitch is played high or low. For another example, if the up / down direction of the bow is wrong, the up / down symbol 81C may be displayed differently from the conventional one. For example, the size, color, brightness of the up / down symbol 81C may be changed or a highlight or flicker effect may be given to the up / down symbol. As another example, if the position or angle of the bow is wrong, the bow calibration object 81D may be displayed. As another example, if the speed of the bow is wrong, an object 81E may be displayed which induces correcting the speed of the bow.

According to one embodiment, the fingernail region 82 may be displayed with a fingerboard image 82A of a stringed instrument. According to one embodiment, the fingerboard image 82A may be displayed with an object 82B that indicates the fingering position that is to be currently played. An object 82C indicating the actual fingering position according to the performance data of the user may be displayed on the fingerboard image 82B. According to one embodiment, the object 82C representing the actual fingertip position can be displayed only when an error has occurred.

19 is a diagram illustrating a user interface according to various embodiments of the present invention.

Referring to FIG. 19, the display 250 may display a user interface including a user's real-time performance result, error information, and error correction information. According to one embodiment, the user interface may include a region (or a score region) 81 for displaying a score, and an area 83 or 84 for visually displaying the movement of the bow .

The score area 81 can display score data. Since the music score area has been described with reference to FIG. 18, a detailed description thereof will be omitted.

According to one embodiment, the bow playing areas 83 and 84 include an area (or skewness area) 83 indicating the angle of the finger's fingerboard direction and an area (or inclination area) indicating the bow string body angle, (Not shown). According to one embodiment, the skewness area 83 may display an image 83A and a bow image 83B of the waist c-bout portion of the string. According to one embodiment, the bow image 83B can be changed in angle and position according to the actual bow performance of the user. For example, the position and angle of the bow image 83B can be determined by the fingerboard direction angle of the bow included in the performance data of the user and the fingerboard direction position of the bow. According to one embodiment, the skewness area 83 may display a range 83C in which the bow can move. According to one embodiment, when the bow image 83B is out of the range 83C in which the bow can move, the color, brightness, or highlight or flicker effect of the range 83C in which the bow can be moved can be given . According to one embodiment, the inclination region 84 may display the bridge image 84A and the bow image 84B of the strings. According to one embodiment, the bow image 84B can be changed in angle and position according to the actual bow performance of the user. For example, the position and angle of the bow image 83B may be determined by the string body inclination of the bow included in the user's performance data and the lateral positon of the bow.

20 is a diagram illustrating a user interface according to various embodiments of the present invention.

Referring to FIG. 20, the display 250 may display a user interface including a user's performance result, error information, error correction information, and content related to a user's performance pattern when the user's performance ends. According to one embodiment, the user interface may include an icon representing a playing element 85A, pitch (or finger playing) 85B, and playing result by rhythm 85C. According to one embodiment The user interface may include an icon 85D representing the overall performance result. According to one embodiment, the user interface may include error correction information 86 for the user's performance results, for example, The error correction information 86 may be provided in text form. According to one embodiment, the user interface may include content 87 associated with the user ' s error pattern. For example, The user interface may include content 88 associated with the user ' s normal performance pattern. ≪ RTI ID = 0.0 > . For example, a recommendation song, which can be played by the user without difficulty, including a normal play pattern of the user, may be provided in a link form. The user interface shown in Fig. 21 can be displayed on the display 250. [0156] Fig.

21 is a diagram illustrating a user interface according to various embodiments of the present invention.

Referring to FIG. 21, a display 250 may display a user interface related to lecture contents. The user interface may include a region (or a score region) 91 for displaying a score, a region (or a fingering region) 92 for visualizing and displaying the finger position, and an area 93 for playing the moving image lecture . According to one embodiment, the content displayed in the score area 91 and the fingering area 92 may be changed corresponding to the contents of the lecture contents. For example, the score displayed in the score area 91 may be changed according to the contents of the lecture contents. For example, the fingertip area 92 may be changed to an area indicating the angle of the bow according to the contents of the lecture contents.

The audio module 260 may generate and output an audio signal. For example, the audio module 260 may include an audio interface or an internal speaker capable of connecting a speaker or earphone (or headphone). According to one embodiment, the audio module 260 may generate an audio signal using performance data.

22 is a diagram illustrating a user interface according to various embodiments of the present invention.

22 shows a user interface that provides real-time calibration information when the second electronic device 200 is implemented as a smart watch. For example, a user may play a string instrument using a paper score, and the second electronic device 200 may analyze the performance of the user and the performance pattern in real time to provide correction information.

Referring to FIG. 22 (a), the display 250 may provide a user interface for selecting a song (or score) to be played by the user. For example, the display 250 may display a list of music data stored in the memory 230. Referring to FIG. 22 (b), the display 250 may display a user interface for selecting the type of calibration information. For example, the type of the calibration information can select at least one of a bow performance, a pitch (or a finger performance), and a rhythm, which are elements for determining a user's performance result. According to one embodiment, when the user selects one of the determination factors, a user interface that allows selection of the details of the selected determination factor may be displayed. Referring to FIGS. 22 (c) and 22 (d), the display 250 may display calibration information according to a user's performance result and a real-time analysis result of a performance pattern.

For example, when a user plays a stringed instrument, the first electronic device 100 may determine the pitch (or frequency) of the sound generated by the string performance. The first electronic device 100 may transmit the determined pitch (or frequency) to the second electronic device 200 and the second electronic device 200 may perform the operation corresponding to the pitch.

23 is a diagram illustrating a user interface according to various embodiments of the present invention.

According to one embodiment, a user may enter a user command into the second electronic device 200 through playing a stringed instrument. Referring to FIG. 23, the second electronic device 200 may display a user interface corresponding to the start screen of the stringed instrument lesson application. According to one embodiment, the user interface may include a plurality of menus and code information 95 corresponding to each menu. When the user plays a note corresponding to a particular code, the first electronic device 100 can determine the pitch (or frequency) of the notes generated by the string playing. The first electronic device 100 may transmit the determined pitch (or frequency) to the second electronic device 200 and the second electronic device 200 may perform the operation corresponding to the pitch. For example, in FIG. 23, a stringed music lesson application may be started when the user plays a G-code.

According to one embodiment, a user may enter a user command into the second electronic device 200 through movement of the string. For example, when a user moves a stringed instrument, the first electronic device 100 attached to the stringed instrument may sense movement of the stringed instrument using the inertial sensor. The first electronic device 100 may transmit motion information of the string to the second electronic device 200 and the second electronic device 200 may perform an operation corresponding to the motion of the stringed instrument.

24 is a flowchart showing a method of recognizing stringed musical performances of an electronic device according to various embodiments of the present invention. The flowchart shown in FIG. 24 may be configured with operations that are processed in the first electronic device 100 shown in FIG. Accordingly, the contents described with respect to the first electronic device 100 with reference to Figs. 4 to 16 can be applied to the flowchart shown in Fig. 24, even if omitted from the following description.

Referring to FIG. 24, the first electronic device 100, in 2410 operation, can sense movement of the bow. According to one embodiment, the first electronic device 100 can sense the movement of the bow using an image sensor. According to one embodiment, the first electronic device 100 may sense movement of the bow using a metal sensing sensor or a magnetic field sensor.

According to one embodiment, the first electronic device 100, in 2420 operation, is capable of sensing vibrations occurring in a stringed instrument. According to one embodiment, the first electronic device 100 can sense vibrations occurring in a stringed instrument using a vibration sensing sensor.

According to one embodiment, in the 2430 operation, the first electronic device 100 may analyze the movement of the bow and the vibrations of the string to produce performance data. The performance data may include, for example, a pitch, a sound intensity, a rhythm, a longitudinal position of the bow, a lateral position of the bow, a relative tilt of the bow and string, The inclination of the bow to the body direction of the string, the type of string the bow is in contact with, the fingering position of the user, and the speed of the bow.

According to one embodiment, the first electronic device 100 uses the sensed value of the image sensor to determine the longitudinal position of the bow, the skewness of the bow toward the fingerboard, The inclination and the speed of the bow can be determined. According to one embodiment, the first electronic device 100 may binarize the infrared image of the image sensor and determine the elements described above using the binarized image. According to one embodiment, the first electronic device 100 can determine the speed (direction and speed) of the bow using the sensed value of the metal sensing sensor. According to one embodiment, the first electronic device 100 can determine the lateral position of the bow and the speed (direction and speed) of the bow using the sensed value of the magnetic field sensor. According to one embodiment, the control module 160 may use an inclination of the bow in the direction of the string body to determine the bow that the bow is in contact with.

According to one embodiment, the first electronic device 100 may analyze the vibration sensed by the vibration sensing sensor to determine the pitch, the sound intensity, and the rhythm. According to one embodiment, the first electronic device 100 can determine the pitch using a frequency component that may occur in a string where the bow contacts a plurality of frequency components included in the vibration. According to one embodiment, the first electronic device 100 may apply a window function when converting the vibration sensed by the vibration sensor 113 to the frequency domain from time domain to sense pitch. According to one embodiment, the first electronic device 100 may set the time axis size of the window function differently depending on the type of string the bow is in contact with.

According to one embodiment, the first electronic device 100 may determine the fingering position of the user based on the pitch at which the pitch and bow touch. According to one embodiment, the first electronic device 100 may determine the fingertip position of the string where the bow is in contact among the plurality of fingertip positions corresponding to the pitch, as the fingertip position of the user.

According to one embodiment, the first electronic device 100 may transmit performance data to the second electronic device 200, at 2440 operation.

25 is a flow diagram illustrating a stringed playing feedback method of a second electronic device in accordance with various embodiments of the present invention. The flowchart shown in Fig. 25 can be configured with operations that are processed in the second electronic device 200 shown in Fig. Therefore, the contents described with respect to the second electronic device 200 with reference to Figs. 17 to 23 can be applied to the flowchart shown in Fig. 25, even if omitted from the following description.

Referring to Fig. 25, the second electronic device 200 may receive string performance data from the first electronic device 100, at 2510 operation. The performance data may include, for example, a pitch, a sound intensity, a rhythm, a longitudinal position of the bow, a lateral position of the bow, a relative tilt of the bow and string, The inclination of the bow to the body direction of the string, the type of string the bow is in contact with, the fingering position of the user, and the speed of the bow.

According to one embodiment, in the 2520 operation, the second electronic device 200 may determine the user's performance using the performance data. According to one embodiment, the second electronic device 200 may use the performance data to determine which performance technique the user has used. According to one embodiment, the second electronic device 200 can compare the performance data with the score data to determine the performance result of the user in real time. For example, the second electronic device 200 may determine whether the user has played to match the score data or a performance error has occurred. According to one embodiment, the second electronic device 200 may be divided into a pitch (or finger playing), rhythm, and bow playing to determine a user's playing result.

According to one embodiment, the second electronic device 200 can provide error information and error correction information in real time when a performance error occurs. According to one embodiment, the second electronic device 200 may provide feedback in the form of an image or text via a display, or may provide feedback in the form of a voice through an audio module. According to one embodiment, the second electronic device 200 may provide feedback by synthesizing the user's performance results when the user's performance is complete.

According to one embodiment, in the 2530 operation, the second electronic device 200 may analyze the performance pattern of the user using the performance results of the user. The performance pattern of the user may include, for example, a normal performance pattern that the user is playing skillfully and an error performance pattern that is frequently mistaken by the user. According to one embodiment, the second electronic device 200 may analyze a user's performance pattern using a pattern analysis algorithm stored in memory. According to one embodiment, the pattern analysis algorithm can learn a performance pattern using a normal performance pattern database and an error pattern database.

According to one embodiment, the second electronic device 200 may analyze the user's performance results in real time to analyze the error pattern. According to one embodiment, the second electronic device 200 may analyze a performance pattern by analyzing the entire performance result of the user when the performance of the user is completed.

According to one embodiment, the second electronic device 200 may provide feedback on the performance pattern of the user in the 2540 operation. According to one embodiment, the second electronic device 200 may provide feedback in the form of an image or text via a display, or may provide feedback in the form of a voice through an audio module. According to one embodiment, the second electronic device 200 can provide calibration information in real time for the analyzed error pattern in real time. According to one embodiment, the second electronic device 200 may provide feedback related to the analyzed performance pattern (e.g., lecture content related to error patterns or error patterns associated with the error pattern) when the user's performance is complete. According to one embodiment, the second electronic device 200 may count the number of occurrences of the performance pattern and provide feedback based on the number of occurrences of the performance pattern.

As used in this document, the term "module" may refer to a unit comprising, for example, one or a combination of two or more of hardware, software or firmware. A "module" may be interchangeably used with terms such as, for example, unit, logic, logical block, component, or circuit. A "module" may be a minimum unit or a portion of an integrally constructed component. A "module" may be a minimum unit or a portion thereof that performs one or more functions. "Modules" may be implemented either mechanically or electronically. For example, a "module" may be an application-specific integrated circuit (ASIC) chip, field-programmable gate arrays (FPGAs) or programmable-logic devices And may include at least one.

At least a portion of a device (e.g., modules or functions thereof) or a method (e.g., operations) according to various embodiments may include, for example, computer-readable storage media in the form of program modules, As shown in FIG. When the instruction is executed by a processor (e.g., control module 160 and control module 240), the one or more processors may perform a function corresponding to the instruction. The computer-readable storage medium may be, for example, a memory 230.

The computer readable recording medium may be a hard disk, a floppy disk, a magnetic media (e.g., a magnetic tape), an optical media (e.g., a compact disc read only memory (CD-ROM) digital versatile discs, magneto-optical media such as floptical disks, hardware devices such as read only memory (ROM), random access memory (RAM) Etc. The program instructions may also include machine language code such as those produced by a compiler, as well as high-level language code that may be executed by a computer using an interpreter, etc. The above- May be configured to operate as one or more software modules to perform the operations of the embodiment, and vice versa.

Modules or program modules according to various embodiments may include at least one or more of the elements described above, some of which may be omitted, or may further include additional other elements. Operations performed by modules, program modules, or other components in accordance with various embodiments may be performed in a sequential, parallel, iterative, or heuristic manner. Also, some operations may be performed in a different order, omitted, or other operations may be added. And the embodiments disclosed in this document are presented for the purpose of explanation and understanding of the disclosed technology and do not limit the scope of the technology described in this document. Accordingly, the scope of this document should be interpreted to include all modifications based on the technical idea of this document or various other embodiments.

100: first electronic device 110: sensor module
120: communication module 130: audio module
140: power management module 150: battery
160: control module 200: second electronic device
210: communication module 220: input module
230: memory 240: control module
250: Display 260: Audio module
300: third electronic device 400: server

Claims (15)

1. An electronic device mountable on a body part in a stringed instrument comprising at least one string and a body part in which the strings are arranged,
A communication unit for communicating with the user terminal device;
A vibration sensor mounted on a part of the stringed instrument for sensing vibration of the stringed instrument associated with performance;
An image sensor for detecting movement of the bow; and
And a control unit for controlling the communication unit to transmit information about vibration of the stringed instrument sensed through the vibration sensor and information on the movement of the bow sensed through the image sensor to the user terminal device. The method according to claim 1,
The electronic device includes:
And a groove for inserting the stringed instrument. 3. The method of claim 2,
Wherein the groove has a shape corresponding to a bridge of a stringed instrument. The method according to claim 1,
And an inertial sensor for sensing movement of the electronic device corresponding to movement of the body portion of the stringed instrument. The method according to claim 1,
Wherein the vibration sensor is disposed to face the underside of the electronic device in contact with the stringed instrument,
Wherein the image sensor is disposed to face the top surface of the electronic device. The method according to claim 1,
Wherein,
And controls the communication unit to transmit information on the fingertip position to the user terminal device based on the movement of the bow detected through the image sensor and the vibration sensed through the vibration sensor. The method according to claim 1,
Wherein,
And acquires information on the movement of the bow based on infrared rays emitted from an instrument attached to the bow detected through the image sensor. A computer-readable recording medium containing a program for executing a control method of a user terminal,
In the control method,
The vibration of the stringed instrument detected by the vibration sensor included in the electronic device mounted on the body part of the stringed instrument in which the strings of the stringed musical instrument played by the user using the current bow are arranged, Acquiring performance data obtained from the electronic device based on the movement of the player; And
And providing information on the current performance position and information on the movement of the bow on the music score corresponding to the music currently being played, based on the obtained performance data. 9. The method of claim 8,
And providing information on a fingering position corresponding to the currently playing music together with information on a fingering position of the user determined based on the obtained performance data. 9. The method of claim 8,
Wherein the performance data includes motion information of the bow,
In the control method,
And providing an image corresponding to the bow on the image corresponding to the stringed instrument based on the motion information of the bow. 9. The method of claim 8,
And providing information on a fingering position of the user determined based on the obtained performance data, together with an image corresponding to the stringed instrument. 9. The method of claim 8,
And providing cumulative performance results from a performance start point to a performance end point based on performance data obtained from the electronic device. 13. The method of claim 12,
And providing a recommended musical performance based on the accumulated performance results. 9. The method of claim 8,
And providing error information if it is determined that a performance error relating to the currently playing music has occurred, based on the obtained performance data. 9. The method of claim 8,
And providing tuning guide information if it is determined that tuning of the strings of the strings is necessary based on the obtained performance data.

Images