JP7440727B2 - Rhythm comprehension support system - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act (1) On January 24, 2020, Eiichi Shoji, the right holder at the time of the act, held the 2019 Department of Mechanical and Systems Engineering R Course Graduation Research Presentation (Reiwa He published the support system of the present invention in the proceedings of the paper published on January 28, 2015. (2) Eiichi Shoji, the right holder at the time of the act, posted the address of the website YouTube on February 8, 2020: https://youtu. He published the support system of the present invention on be/Eq3oB_o0bek. (3) Eiichi Shoji, the right holder at the time of the act, posted the address of the website YouTube on February 9, 2020: https://youtu. The support system of the present invention was published on be/TtqIDiwaiVY. (4) Eiichi Shoji, the right holder at the time of the act, posted on February 10, 2020 the address of the website YouTube: https://youtu. He published the support system of the present invention on be/zsvs8i7WNh0. (5) Eiichi Shoji, the right holder at the time of the act, posted on February 19, 2020 the address of the website YouTube: https://youtu. He published the support system of the present invention on be/h29veEQ_rgA. (6) Eiichi Shoji, the right holder at the time of the act, posted on March 5, 2020 the address of the website YouTube: https://youtu. The support system of the present invention was disclosed to be/-SbvZpsJQz4. (7) Eiichi Shoji, the right holder at the time of the act, posted the address of the website YouTube on March 8, 2020: https://youtu. The support system of the present invention was disclosed to be/rTJUH2ee35o. (8) Eiichi Shoji, the right holder at the time of the act, posted the address of the website YouTube on March 9, 2020: https://youtu. The support system of the present invention was disclosed to be/ZO38N3uiApU. (9) Eiichi Shoji, the right holder at the time of the act, posted on March 10, 2020 the address of the website YouTube: https://youtu. The support system of the present invention was disclosed to be/Yo8SO1KkvL0. (10) Eiichi Shoji, the right holder at the time of the act, posted on March 10, 2020 the address of the website YouTube: https://youtu. The support system of the present invention was published on be/2rVaNQadg-4.

Application of Article 30, Paragraph 2 of the Patent Act (11) Eiichi Shoji, the right holder at the time of the act, uploaded the website YouTube address: https://youtu.com on March 10, 2020. The support system of the present invention was disclosed to be/pzfk-gysWJo. (12) Eiichi Shoji, the right holder at the time of the act, posted on March 15, 2020 the address of the website YouTube: https://youtu. The support system of the present invention was published on be/eDiaxqHVbdw. (13) Eiichi Shoji, the right holder at the time of the act, posted on March 15, 2020 the address of the website YouTube: https://youtu. The support system of the present invention was disclosed to be/v-PMsjAv-sA. (14) Eiichi Shoji, the right holder at the time of the act, posted on March 19, 2020 the address of the website YouTube: https://youtu. The support system of the present invention was disclosed to be/3FnGPHfLrNE. (15) Eiichi Shoji, the right holder at the time of the act, posted on March 20, 2020 the address of the website YouTube: https://youtu. The support system of the present invention was published on be/75K-4QyA80M.

The present invention relates to a rhythm comprehension support system for grasping rhythm.

2. Description of the Related Art In order for deaf people to dance to music, there is a person known as a count man who performs movements to guide the rhythm of the music (for example, see Non-Patent Document 1).

The counting man shown in Non-Patent Document 1 performs an action to match the rhythm of a music being played or a gesture indicating the next action to a hearing-impaired person. By dancing while watching the count man's gestures, people with hearing disabilities can dance to the rhythm of the music even if they cannot hear the music well.

However, the gestures performed by the count man require special knowledge, and not everyone can perform them. For this reason, it is difficult for a hearing-impaired person to easily dance and enjoy the above-mentioned dance by preparing a counting man.

Japan Broadcasting Corporation, [searched on April 15, 2020], Internet <URL: http://www6.nhk.or.jp/baribara/lineup/single.html?i=282>

An object of the present invention is to provide a rhythm understanding support system that allows, for example, a hearing-impaired person to easily dance to music or enjoy an ensemble.

Such objects are achieved by the present invention described in (1) to (7) below.
(1) A rhythm comprehension support system that supports a user in grasping rhythm,
an input section into which music data is input;
a simulation video generation unit that generates a simulation video in which a count person performs an auxiliary movement to guide the rhythm of a song in which the music data is played, based on the music data input to the input unit;
A rhythm comprehension support system comprising: a display control unit that displays the simulation video generated by the simulation video generation unit.

(2) The rhythm comprehension support system according to (1) above, wherein the auxiliary motion is a motion that indicates a rhythm temporally preceding the rhythm of the music.

(3) In the simulation video, the simulation video generation unit is configured to determine the count person's costume, the count person's facial expression, the count person's type, the count person's size, the count person's voice quality, and the background. The rhythm comprehension support system according to (1) or (2) above, wherein at least one of the rhythms is changed in accordance with the music data.

(4) The simulation video described in any one of (1) to (3) above is a video in which a main person other than the count person dances to the rhythm of the song data together with the count person. rhythm comprehension support system.

(5) The rhythm comprehension support system according to (4) above, wherein the simulation video is a video in which the count person and the main person are lined up.

(6) The rhythm comprehension according to any one of (1) to (5) above, wherein the auxiliary movement is a movement in which the count person rocks at least a part of his body from side to side in accordance with the rhythm of the music data. support system.

(7) The rhythm comprehension support system according to any one of (1) to (6) above, which is a system for supporting hearing-impaired people to dance or play in ensemble to music. Grasping support system.

According to the present invention, it is possible to provide a rhythm comprehension support system that allows, for example, a hearing-impaired person to easily dance to music or enjoy an ensemble.

FIG. 1 is a schematic configuration diagram of a rhythm comprehension support system (first embodiment) of the present invention. FIG. 2 is a block diagram of the rhythm comprehension support system shown in FIG. 1. FIG. 3 is a diagram showing an example of a simulation video. FIG. 4 is a diagram showing an example of a simulation video. FIG. 5 is a diagram showing an example of a simulation video. FIG. 6 is a diagram showing an example of a simulation video. FIG. 7 is a diagram showing an example of a simulation video. FIG. 8 is a schematic configuration diagram of the rhythm comprehension support system (second embodiment) of the present invention.

DESCRIPTION OF THE PREFERRED EMBODIMENTS The rhythm understanding support system of the present invention will be described in detail below based on preferred embodiments shown in the accompanying drawings.

<First embodiment>
FIG. 1 is a schematic configuration diagram of a rhythm comprehension support system (first embodiment) of the present invention. FIG. 2 is a block diagram of the rhythm comprehension support system shown in FIG. 1. 3 to 7 are diagrams showing examples of simulation videos.

The rhythm comprehension support system 1 shown in FIG. 1 includes a healthy person (hereinafter referred to as "user 100") and a hearing-impaired person (regardless of whether it is mild or severe), such as a hearing-impaired person due to age-related hearing loss. This system supports a hearing-impaired person (hereinafter referred to as "user 200") to dance (including gymnastics such as radio gymnastics) or perform in ensemble to music. In other words, the rhythm comprehension support system 1 is a system for supporting the user 200 in dancing or playing in ensemble. Note that, as an example, a case where the user 200 dances will be described below. Furthermore, the present invention can be used by people who are not deaf but are not good at dancing, or who have difficulty grasping rhythm when playing musical instruments, to help them grasp rhythm.

In addition, the main person A and the count person B may not only use body movements, but also the sounds of musical instruments, singing voice synthesis technology, utterances using voice synthesis technology, vocal singing, commands, explanations, etc. It may also be accompanied by human utterances, vocal singing, commands, explanations, etc. Further, musical instruments such as wood blocks, triangles, tambourines, etc. may be played by sound effects such as chimes or by a musical instrument playing robot in accordance with the body movements.

In addition, even if you are deaf or hard of hearing, you may be able to feel the rhythm with your body due to vibrations in the air or the floor, so it is extremely meaningful to play sounds together with this system, and the system functions. It is important to be able to do this, and it is necessary to be able to do so.

As shown in FIGS. 1 and 2, the rhythm comprehension support system 1 includes a playback unit 2 that plays music, a display device 3 that displays videos in accordance with the music, and a control device 4 that controls their operations. A setting input device 5 is provided. Further, the control device 4 generates a video that guides the user 200 to a rhythm that matches the music played by the playback unit 2, and causes the display device 3 to display the video so that the user 200 can understand the rhythm of the music. Thereby, even the user 200 can enjoy dancing to the music.

Note that the term "rhythm" in this specification includes not only changes in tempo but also changes in dynamics (gradually becoming stronger, gradually becoming weaker, etc.).
This will be explained in detail below.

[Playback section]
As shown in FIGS. 1 and 2, the reproduction section 2 includes a speaker 21. As shown in FIG.

The speaker 21 is electrically connected to the control device 4, converts the electrical signal of music data transmitted from the control device 4 into a physical signal, and reproduces the music by emitting sound. This reproduction may be accompanied by singing voice synthesis technology, speech by voice synthesis technology, vocal singing, commands, explanations, etc., or may be accompanied by actual human speech, vocal singing, commands, explanations, etc.

Communication between the speaker 21 and the control device 4 may be wireless or wired. Further, the speaker 21 may be provided with an adjustment section that adjusts the volume.

Further, in the illustrated configuration, the reproduction section 2 has one speaker 21, but the present invention is not limited to this, and the reproduction section 2 may have a plurality of speakers 21.

Furthermore, instead of the speaker 21, the playback unit 2 may use a device worn by the user, such as earphones or headphones. In the case of this configuration, a jack to which earphones or headphones are connected may be provided in a casing (not shown) in which the control device 4 is housed or in the speaker 21.

[Display device]
As shown in FIGS. 1 and 2, the display device 3 includes a projection device 31 and a screen 32.

The projection device 31 is, for example, a light emitting device having a light source. This method includes, but is not particularly limited to, a liquid crystal method, a DLP method, an LCOS method, and the like.

As shown in FIG. 1, the screen 32 has a projection surface on which the image light emitted from the projection device 31 is projected. In the illustrated configuration, the screen 32 is in the form of a sheet that can be folded or rolled to be stored or expanded.

The screen 32 is not limited to a reflective type as shown in the figure, but may be any type that interacts with light (reflection, refraction, diffraction, luminescence, fluorescence, phosphorescence, etc.) and allows the projected image to be recognized. , may be of a transmission type.

Note that the present invention is not limited to the above, and instead of the screen 32, for example, a mist screen may be used, or a configuration may be used in which the image is projected onto a wall surface such as a wall of a building. Alternatively, the screen 32 may be omitted and the image light may be formed in the air. Furthermore, the image may be projected onto an object on a transmissive screen using a projection mapping technique. Alternatively, projection mapping may be used to map and project onto the surface of an actual object, vapor, mist, droplets, powder, or liquid surface.

Further, the display device 3 is not limited to the above configuration, and may be a display device such as a liquid crystal or organic EL, or a wearable device such as a head mounted display. Further, the display device 3 may be a projector used for movies or the like.

[Control device]
As shown in FIGS. 1 and 2, the control device 4 includes a CPU (Central Processing Unit) 41, an I/O interface (receiving section) 42, and a memory (storage section) 43.

The CPU 41 executes a plurality of modules and various programs stored in the memory 43. Modules may include integrated components in hardware circuits including logic gates, semiconductor devices, integrated circuits, or any other discrete components. A module may also be part of a software program. Execution of a module as a software program may include a set of logical operation instructions to be executed by CPU 41 or any other hardware entity. Additionally, a module may be embodied as a set of instructions or programs using an interface.

The I/O interface 42 is for communicating with other communication devices (playback section 2, display device 3, etc.). Examples of the I/O interface 42 include, but are not limited to, audio, analog, digital, monaural, RCA, stereo, IEEE-1394, serial bus, universal serial bus (USB), infrared, PS/2, BNC, coaxial component, Composite, DVI (Digital Visual Interface), HDMI (High-Definition Multimedia Interface) (registered trademark), RF antenna, S-Video, VGA, Bluetooth (registered trademark), Cellular (for example, CDMA (Code-Di vision (Multiple Access), HSPA+ (High-Speed Packet Access), GSM (Global System for Mobile communications) (registered trademark), LTE (Long-Term Evolution), WiMax, etc.), Wi-Fi standard, wireless L Adopting communication protocols/methods such as AN It's okay.

This I/O interface 42 has a function of importing music data from the outside. That is, the I/O interface 42 functions as an input section into which music data is input from the outside.

Song data is not particularly limited, but includes, for example, audio data in audio file formats such as WAVE (RIFF waveform Audio Format) and mp3 (MPEG Audio Layer-3), and MIDI data (MIDI standard (Musical Instrument Digital Intel)). rface) (registered trademark) music data). The music data may include not only the sounds played by musical instruments, but also vocal synthesis technology, utterances using voice synthesis technology, vocal singing, commands, explanations, etc., and includes actual human utterances, vocal singing, commands, explanations, etc. May be accompanied by In the following, it will be explained as MIDI data. By using MIDI data, as will be described later, when generating a simulation video, it is possible to easily grasp the rhythm of a song, making it easier to generate a simulation video. Moreover, MIDI data can be synchronized with data of a simulation video, which will be described later, by, for example, a simple method of embedding a marker signal in music data. In addition, the simulation video includes not only musical instrument performances, but also singing voice synthesis technology, utterances using voice synthesis technology, vocal singing, commands, explanations, utterances by actual humans, vocal singing, commands, explanations, etc. that can be easily synchronized. can do. Further, when processing and importing music data into the memory 43, it is possible to suppress an increase in the data capacity.

The memory 43 is composed of, for example, an EEPROM (Electrically Erasable Programmable Read-Only Memory), which is a type of nonvolatile semiconductor memory. The modules stored in the memory 43 include a playback module 431, a display control module 432 (display control section), a video generation module 433 (simulation video generation section), and a communication module 434.

The playback module 431 is configured so that when executed by the CPU 41, the playback unit 2 plays the music data taken in by the I/O interface 42 by controlling the energization conditions for the playback unit 2.

When executed by the CPU 41, the display control module 432 (display control unit) controls the operation of the light source device of the display device 3 by controlling the energization conditions for the display device 3, thereby controlling the operation of the light source device of the display device 3. Play (display) the simulation video.

When executed by the CPU 41, the video generation module 433 generates a simulation video based on the information of the music data. The video generation module 433 may have a configuration in which a person issues an instruction using the setting input device 5 and executes the instruction. Alternatively, a configuration may be adopted in which a simulation video generated in advance for each type of music data is stored in the memory 43, and a simulation video corresponding to the music data is selected based on the input music data. Further, the configuration may be such that instructions are executed in accordance with sounds such as music by beat tracking.

The simulation video generated by the video generation module 433 is displayed on the display device 3 by the operation of the display control module 432. Further, when the simulation video is displayed, the music and the simulation video are synchronized by the operation of the playback module 431, and the music is played from the playback unit 2. Further, if necessary, singing voice synthesis technology, voice synthesis technology, utterances by actual humans, vocal singing, commands, explanations, etc. are synchronously reproduced. The simulation video will be detailed later.

Note that the format of the simulation video is not particularly limited, and examples thereof include AVI, MP4, MOV, MPEG2-TS, MPEG2-PS, MKV, WMV, FLV, ASF, VOB, WebM, OGM, and the like. Further, the file format is not particularly limited, and examples thereof include avi, mp4, m4A, mov, qt, m2ts, ts, mpeg, mpg, mkv, flv, f4v, asf, wmv, webm, ogm, and the like. Further, the video codec is not particularly limited, and examples include VP6, H. 263, H. Examples include H.264, Xvid, Divx, MPEG-4, MPEG-2, MPEG-1, WMV9, VP9, and VP8. Furthermore, the audio codec is not particularly limited, and examples include AAC, MP3, LPCM, Voribis, AC-3, FLAC, WMA, ADPCM, MP2, and the like.

The communication module 434 is configured to communicate with a signal output interface or the like of another communication device when executed by the CPU 41.

In addition to the above, the memory 43 also stores control programs, etc., which will be described later. Such a control device 4 may be built in the display device 3 or the setting input device 5, or may be built in a housing different from the display device 3 or the setting input device 5.

[Setting input device]
The setting input device 5 has a function of communicating with the control device 4 and inputting various settings to the control device 4. Various settings performed by the setting input device 5 will be described in detail later. The setting input device 5 is not particularly limited, and examples thereof include a smartphone, a tablet terminal, a personal computer, and the like.

[Lighting equipment, performance equipment, communication equipment]
Further, as shown in FIG. 1, the rhythm comprehension support system 1 includes a lighting device 11, a presentation device 12, and a communication device 13.

Examples of the lighting device 11 include, but are not limited to, a mirror ball, a color ball, a spotlight, a horizontal light, a flat light, a moving head, a brute light, a smoke machine, a bubble machine, a laser projector, a laser beam, a laser light, a strobe, It is possible to use a black light, an LED tape light, a DMX light, a swing light, a spinner, etc., a highly directional laser light, a diffused light illumination, or the like that provides visual effects. Furthermore, the lighting device 11 allows the intensity, color, pattern, timing, etc. of the light to be changed in accordance with the melody. The operation of the lighting device 11 is controlled by the control device 4.

The effect device 12 has the function of producing visual effects such as smoke, fireworks, flames, confetti, tape, crackers, and other visual effects such as smoke, fireworks, flames, crackers, and olfactory effects such as fragrance. In addition, the strength, amount, timing, etc. can be changed to match the melody. The operation of the production device 12 is controlled by the control device 4.

The communication device 13 is composed of a wearable device such as a bracelet or a clip, and has a function of generating vibration and light. The communication device 13 can communicate with the control device 4 using the communication protocol/method exemplified as the I/O interface 42. This allows more people to communicate with a performance system at another venue (i.e., install a copy of the above-mentioned rhythm comprehension support system 1 at another venue) and remotely perform synchronous performances at multiple venues at the same time. You can enjoy it at the same time. Furthermore, when communicating with another venue, the number of participants does not matter, such as one person, a small number of people, or a large number of people.

The configuration of the rhythm comprehension support system 1 has been described above.
Here, as described above, the simulation video generated by the video generation module 433 is a video in which the main person A and the count person B perform different actions, as shown in FIGS. 1 and 3 to 6. It is.

In the simulation video, main person A dances to the rhythm of a song. For example, main person A performs a main movement of dancing by moving at least one of the legs, waist, arms, neck, etc. Further, if necessary, singing, commanding, or explanation may be performed by vocalization or utterance using singing voice synthesis technology or voice synthesis technology. Furthermore, if necessary, an actual person may sing, give commands, or provide explanations. The user can enjoy watching the main person A displayed on the display device 3 and dancing the same dance as the main person A while listening to the music being played.

In the simulation video, Count Person B performs auxiliary movements to guide the user to the rhythm of the music. If necessary, a song may be sung, a command may be given, or an explanation may be made by vocalization or utterance using a singing voice synthesis technique or a voice synthesis technique. Also, if necessary, a human may sing a song, give commands, or give an explanation. The auxiliary movement performed by count person B is one that allows even the hearing-impaired user 200 to grasp the rhythm of the music being played by watching the movement. The auxiliary movement may be an action that is performed temporally ahead or behind the music that is being played, or may be an action that is performed with the same rhythm as the rhythm of the music that is being played. The operation may be a combination of the following. Further, the time-delayed actions by count person B can convey in real time how the rhythm and strength change over time during the performance.

When the auxiliary action is an action that is performed temporally prior to the music being played, examples of the auxiliary action include actions as shown in FIG. In FIG. 3, main person A displayed on the left side of the screen 32 dances to the rhythm. On the other hand, the count person B on the right side of the screen 32 differs from the dance movement performed by the main person A, and his movement temporally precedes the rhythm of the music being played. In addition, in advance in time, Count Person B moves at least part of his body in a large manner to clearly convey the nuances of speed and strength, such as the beginning of a measure, a quarter note, a change in melody, etc. Perform the action that expresses it. That is, Count Person B performs an action that conveys the nuances of the music through gestures, like a conductor waving a tact.

Furthermore, the lighting device 11 can assist the presentation. In other words, by changing the intensity, color, pattern, timing, etc. of the light to match the melody, the nuances of the song can be conveyed more clearly.

In addition, the performance can be assisted by the performance device 12. That is, by changing the intensity, amount, timing, etc. of the performance by the performance device 12 in accordance with the melody, the nuances of the music can be conveyed more clearly.

Further, by changing the strength of vibration, timing of vibration, pattern of vibration, etc. in accordance with the melody while the user 200 is wearing the communication device 13, the nuances of the music can be conveyed more clearly. The interlocking of such a device and count person B helps visually impaired people to understand count person B's movements.

In this way, the auxiliary motion performed by count person B is a motion that temporally precedes the rhythm of the music and indicates the rhythm. Thereby, the user 200 can grasp the rhythm in advance while looking at the count person B, even though the user 200 cannot hear the music well. Therefore, the user 200 can dance and enjoy the dance together with the user 100.

On the other hand, when the auxiliary motion is a motion performed in the same rhythm as the rhythm of the music being played, the auxiliary motion may include, for example, the motions shown in FIGS. 4 and 5. In FIG. 4, count person B is in a state where his body is tilted to the right side in FIG. 4, and in FIG. 5, count person B is in a state in which his body is tilted to the left side in FIG. By repeating these actions in accordance with the rhythm of the song, the user 200 can clearly grasp the rhythm of the song being played while looking at Count Person B, even though the user 200 cannot hear the song well. can. If necessary, healthy people can also enjoy the program by using singing voice synthesis technology or voice synthesis technology to sing, give commands, and provide explanations.

Furthermore, in the configurations shown in FIGS. 4 and 5, count person B plays a role like a metronome. Existing metronomes generally have a configuration in which, for example, a first note is sounded for each quarter note, and a second note different from the first note is sounded for each bar. When using such an existing metronome, the user 200 can grasp the rhythm, but it is difficult for the user 200 to grasp the flow of the song, such as the melody, dynamics, scene changes, etc. Also, it is difficult to instantly tell what beat a note is by just looking at the sound. On the other hand, in the present invention, as described above, Count Person B visually sways his body from side to side according to the rhythm, shows a large rhythm for each measure, and while moving his body back and forth from side to side, his arms, The rhythm of a quarter note, or the rhythm within a measure or across multiple measures, can be shown in detail by movements of the neck, hips, etc., and facial expressions. Furthermore, rhythms such as odd time signatures are also expressed, and the user 200 can grasp this rhythm. This makes it possible to predict the movement of the character by visually following it.

For example, if the melody in the next measure is a gentle melody, the user 200 can predict that the next development will be a gentle melody by making the movement indicating the quarter note rhythm a modest movement. I can do it. Further, for example, if the melody in the next measure is a violent melody, the user 200 can predict that the next development will be a violent melody by making the motion indicating the quarter note rhythm a violent motion. can. In this way, according to the rhythm comprehension support system 1, the count person B can not only play the role of a metronome but also express the rhythm in a richly expressive manner. Further, if necessary, singing, commanding, or explanation may be performed by vocalization or utterance using singing voice synthesis technology or voice synthesis technology. Furthermore, if necessary, an actual person may sing, give commands, or provide explanations.

In addition, Count Person B's costume may be changed for each song being played. The costume of count person B may be changed depending on the scene of the melody, B melody, etc. Patterns for changing costumes include, for example, changing colors, types, drawn characters, and the like.

In addition, Count Person B's singing voice, commands, and explanatory voice may be changed for each song being played. Count Person B's singing voice, commands, and explanatory voice may be changed depending on the scene, such as the chorus, A melody, B melody, etc. Examples of voice change patterns include changing voice quality, pitch, strength, intonation, and changing to the voice of another character.

Further, the count person B's facial expression may be changed for each song being played, or the count person B's facial expression may be changed in the middle of the song being played according to the melody. Examples of facial expressions include a smiling face and a sad face.

Further, the type of count person B may be changed for each piece of music being played, or the type of count person B may be changed in the middle of the piece of music being played depending on the melody. In other words, the character may be changed to a different character. It may become transparent and disappear if desired for visual effect.

In addition, the size of Count Person B (body shape, size of body parts) may be changed for each song being played, and in the middle of the song being played, Count Person B's size may be changed according to the melody. You may change the size as shown.

Further, as shown in FIG. 6, the background may be changed for each song being played, or the background may be changed in the middle of the song being played according to the melody. Further, the background may be a real-life background or may be computer graphics.

In this way, the video generation module 433 as a simulation video generation unit generates the costume of count person B, the facial expression of count person B, the type of count person B, the size of count person B, and the voice quality of count person B in the simulation video. , and the background may be changed in accordance with the music data. As a result, the music being played becomes more expressive, and it becomes easier for the user 200 to remember what kind of dance to perform.

Such a change may be made by a person operating the setting input device 5, or may be made by a preset program.

Further, by operating the setting input device 5, the tempo of the music may be changed. In this case, the playback speed of the simulation video is also changed in accordance with the change in tempo.

Further, as described above, the simulation video is a video in which the main person A, who is different from the count person B, dances (main motion) along with the count person B in accordance with the rhythm of the music data. Thereby, user 100 and user 200 can enjoy dancing.

Further, the simulation video is a video in which count person B and main person A are lined up. Thereby, the user 200 can dance while watching both the main person A and the count person B. Therefore, the user 200 can enjoy dancing more.

Note that the simulation video is not limited to such a configuration, and the simulation video may be a video in which only count person B is displayed, as shown in FIG. The video may be distributed to other venues using communication devices. Furthermore, it may be distributed so that it can be displayed on a mobile phone, smartphone, tablet, computer terminal, etc. In FIG. 7, a simulation video is displayed on a tablet terminal. In the case of such a configuration, the user 200 can enjoy and practice dancing alone at home or the like. Furthermore, users 200 at other venues can watch the video of Count Person B on their smartphones, mobile terminals, etc. and dance together according to the commands. This allows able-bodied people and people with disabilities to dance together at the same time in a location of their choice, creating a sense of unity and having fun.

Further, in the configuration shown in FIG. 7, the auxiliary motion performed by count person B is a motion in which count person B rocks at least a portion of his body left and right in accordance with the rhythm of music data. This allows you to use the character's left and right movements as a metronome. By keeping the tempo constant or changing it, you can use it to play instruments or practice singing while watching it.

In addition, in the configuration shown in Figure 7, there is a function that allows the speed of left and right body movements to be varied, and singing, commands, and explanations can be made using singing voice synthesis technology and voice synthesis technology to produce sounds and utterances according to left and right movements. It may have a function.

In addition, by muting the sound, you can use it not only when playing instruments or practicing singing, but also during the actual concert, to adjust the performance timing and performance intention (starting point, tempo, crescendo, etc.) according to the movements of the count person on the screen. It can be used to check things to keep in mind when playing. Particularly when silence is required at a concert where silence is a concern, by using changes in the character's color, size, costume, etc., as described above, it can be used as a visually effective metronome. Can be used.

In addition, in addition to main person A and count person B, information such as text explaining the movement, lyrics, composer, lyricist, arranger, etc. may be displayed in the simulation video, and information such as singing voice synthesis technology and voice synthesis technology may also be displayed. These may be explained using utterances or utterances. The voice quality may be changed for each song.

In this way, the rhythm comprehension support system 1 supports the user 200 in grasping the rhythm. In addition, the rhythm comprehension support system 1 includes an I/O interface 42 (input section) into which music data is input, and a count person B who plays music data based on the music data input to the I/O interface 42. A video generation module 433 (simulation video generation unit) that generates a simulation video that performs auxiliary movements to guide the rhythm of the song, and a display control module 432 (display control unit) that displays the simulation video generated by the video generation module 433. and. According to such a configuration, the user 200 can enjoy dancing to the rhythm of the music. Furthermore, in the simulation video, Count Person B uses his body to guide the user to the rhythm, so it is easy for the user 200 to understand and guide the user to the rhythm. In particular, there is no need to actually prepare a count person, and the count person can be displayed as a simulation video. Therefore, a hearing-impaired person, ie, the user 200, can easily enjoy dancing to music.

In addition, the main person A and the count person B may be accompanied by not only body movements, but also the sounds of musical instruments, singing voice synthesis technology, utterances using voice synthesis technology, vocal singing, commands, explanations, etc. It may also be accompanied by human utterances, vocal singing, commands, explanations, etc. Furthermore, sound effects such as chimes may be played in accordance with the body movements, or musical instruments such as wood blocks, triangles, tambourines, etc. may be played by a musical instrument playing robot.

Moreover, the main person A and the count person B may sing a song or shout while dancing. For animation, you can also move the character's mouth. By looking at the movements of the mouth, the user 200 can easily understand which part of the song is currently being played, and what action to perform next.

Further, in this embodiment, the main person A and the count person B are virtual characters, but the present invention is not limited to this, and may be, for example, real-life humans. If necessary, the body may be moved by singing, commanding, or giving explanations using singing voice synthesis technology or voice synthesis technology. Further, if necessary, an actual person may sing a song, give a command, or provide an explanation by voice or utterance.

Furthermore, in this embodiment, the main person A and the count person B are humanoid characters, but the present invention is not limited to this. It may be a character created by computer animation (registered trademark), computer graphics (CG), 3DCG, 3DCG animation, or the like. Specifically, virtual YouTubers, VTubers, virtual livers, virtual idols, virtual talents, virtual singers, virtual autograph singers, etc. may be mentioned. In addition, these characters are able to create synthesized voices and sounds such as songs, commands, explanations, sound effects, etc. using singing voice synthesis technology, voice synthesis technology, voice synthesis technology, VOCALOID (registered trademark), VOCALOID (registered trademark), singing voice synthesizer, etc. may be issued. It is also possible to create the effect of speaking using the voice of an actual human. In addition to voices, sound effects, beeps, and the like may be emitted to notify the viewer of a signal or intention.

Further, such a rhythm grasping support system 1 can be used in amusement parks, concert venues, stages, karaoke, physical education classes, dance classes, music classes, various schools, dancing at home, practicing playing musical instruments, etc. can. Therefore, both able-bodied people and hearing-impaired people can enjoy dance and music in a wide variety of situations.

<Second embodiment>
FIG. 8 is a schematic configuration diagram of the rhythm comprehension support system (second embodiment) of the present invention.

Hereinafter, a second embodiment of the present invention will be described with reference to this figure, but the explanation will focus on the differences from the first embodiment described above, and the explanation of similar matters will be omitted.

As shown in FIG. 8, the rhythm comprehension support system 1 of this embodiment includes, in addition to the playback unit 2, display device 3, control device 4, and setting input device 5 described in the first embodiment, described in the first embodiment. The apparatus further includes a musical instrument 6, a performance device 7 for playing the musical instrument 6, and a motion detection section 8.

A performance device 7 shown in FIG. 8 is a performance device that plays the musical instrument 6.
The musical instrument 6 is not particularly limited, and includes, for example, percussion instruments, string instruments, keyboard instruments, wind instruments, etc., which will be described later. , ride cymbal, crash cymbal).

Note that in this specification, musical instruments include those that themselves emit sound, those that amplify the sounds that they themselves produce and output them from speakers, etc., and those that do not themselves produce sounds but receive electrical signals. This includes devices that output sound from speakers, etc.

The performance device 7 includes a striking section provided for each part of the drum (snare drum, bass drum, hi-hat, ride cymbal, crash cymbal), and a driving section that drives the striking section. The striking part and the driving part can be configured as described in Japanese Patent Application Laid-open No. 2019-144360.

Further, the operation of the performance device 7 is controlled by controlling the energization conditions by the control device 4.

According to such a performance device 7, for example, it is possible to create a sense of realism such as overtones of a musical instrument that cannot be reproduced by a speaker or the like.

Furthermore, by synchronizing the performance speed with the music playback speed and the simulation video described in the above embodiment, it is possible to add depth to the music played by the musical instrument 6 by adding the performance of the musical instrument 6 to the music played from the playback section 2. .

As described above, in this embodiment, the songs are songs that are obtained by playing music data created in advance (music that is played from the playback unit 2), and songs that are obtained by playing musical instruments in real time. This allows the music to have depth. It should be noted that the present invention is not limited to this configuration, and it is sufficient that music played by reproducing music data created in advance and music obtained by playing musical instruments in real time are played.

In the illustrated configuration, the musical instruments 6 are drums and congas, but the musical instruments 6 are not particularly limited, and examples thereof include other percussion instruments, string instruments, keyboard instruments, keyboard percussion instruments, wind instruments, pneumatic instruments, and the like.

Other percussion instruments include electronic drums, Japanese drums, cajon, djembe, tambourine, triangle, castanet, wood block, cowbell, clapboard, claves, mokugyo, bongo, timpani, concert chime, wind chime, cabasa, sleigh bell, Examples include Tin Palace and Agogo.

Stringed instruments include acoustic guitars, electric guitars, acoustic basses, electric basses, harps, mandolins, violins, violas, cellos, double basses, and the like.

Keyboard instruments include pianos, honky-tonk pianos, electric pianos, keyboards, harpsichords, celestas, organs, electronic organs, pipe organs, reed organs, electric tones, keyboard harmonicas, accordions, synthesizers, and the like.

Keyboard percussion instruments include the xylophone, xylophone, xirolimba, marimba, glockenspiel, vibraphone, antique cymbals, glass harp, and armonica.

Wind instruments include woodwind instruments such as the flute, recorder, ocarina, shakuhachi, clarinet, oboe, saxophone, bassoon, sho, and bagpipes, and brass instruments such as the trumpet, trombone, horn, euphonium, tuba, whistle, and samba whistle. etc.

Examples of air sound instruments include harmonicas, bagpipes, whistles, tuning flutes, air sound toys, and the like.

[Motion detection section]
The motion detection unit 8 shown in FIG. 8 is attached to a part of the body of the user 200 and has a function of detecting the motion of the user 200. In this embodiment, the motion detection section 8 is configured with an angular velocity sensor that detects angular velocity. The angular velocity sensor includes a casing (not shown), a sensor element (not shown) built into the casing, and a communication section (not shown) that communicates with the control device 4 .

The housing can be worn on the wrist of the user 200 using a wristband, for example. Note that the device is not limited to this configuration, and may be attached to any desired location such as a finger, head, abdomen, foot, or the like.

The sensor element may be of any type including capacitance type, strain gauge type, piezoelectric type, eddy current type, electromagnetic induction type, and optical type. Information on the angular velocity detected by the sensor element is converted into an electrical signal and transmitted to the control device 4 by the communication section. Communication between the communication unit and the control device 4 may be either wireless or wired.

When the user 200 dances while watching the simulation video, the motion detection unit 8 attached to the hand moves. Information on the angular velocity detected by the motion detection section 8 during this movement is transmitted to the control device 4. Then, the control device 4 can detect the peaks of the angular velocity and estimate the tempo at which the user 200 is dancing based on the interval between each peak. Therefore, for example, the operations of the playback unit 2 and the display device 3 can be controlled so that the tempo of the song or the playback speed of the simulation video match the tempo at which the user 200 is dancing. As a result, the user 200 can enjoy dancing without worrying about whether the tempo of the dance he or she is dancing matches the tempo of the song.

In addition, in this embodiment, the case where the motion detection unit 8 is an angular velocity sensor has been described, but the present invention is not limited to this, and may include an acceleration sensor, an angular acceleration sensor, a magnetic sensor, a pressure-sensitive sensor, a displacement sensor, etc. May be used. These housings and sensors may be configured to be attached to the body of the user 200 or an object such as clothing or shoes to detect the user's movements, although they are not particularly limited. In this case, the user 200 can convey his or her intention to the control device 4 based on the speed of the swing, the width of the swing, and the like.

Furthermore, the motion detection unit 8 may be a line of sight input device. The user 200 can convey his/her intention to the control device 4 through his/her line of sight using eye movements, blinking, and the like.

Although the rhythm grasping support system of the present invention has been described above with reference to the illustrated embodiment, the present invention is not limited to this, and each part constituting the rhythm grasping support system may be any arbitrary unit that can perform the same function. It can be replaced with the configuration of Moreover, arbitrary components or steps may be added.

Further, in the embodiment, the storage unit (memory) is built in the control device, but the present invention is not limited to this, and the storage unit (memory) is stored in an external device (for example, a USB flash memory). It may be. In this case, by connecting an external device to the control device, control as described in the previous embodiment can be performed.

1 Rhythm comprehension support system 2 Playback section 3 Display device 4 Control device 5 Setting input device 6 Musical instrument 7 Performance device 8 Movement detection section 11 Lighting device 12 Performance device 13 Communication device
21 Speaker 31 Projection device 32 Screen 41 CPU
42 I/O interface 43 Memory 431 Playback module 432 Display control module 433 Video generation module 434 Communication module 100 User 200 User A Main person B Count person

Claims (7)

A rhythm comprehension support system that assists a user in grasping rhythm,
an input section into which music data is input;
a simulation video generation unit that generates a simulation video in which a count person performs an auxiliary movement to guide the rhythm of a song in which the music data is played, based on the music data input to the input unit;
A rhythm comprehension support system comprising: a display control unit that displays the simulation video generated by the simulation video generation unit. 2. The rhythm comprehension support system according to claim 1, wherein the auxiliary motion is a motion that indicates a rhythm temporally preceding the rhythm of the music. The simulation video generation unit is configured to generate at least one of the count person's costume, the count person's facial expression, the count person's type, the count person's size, the count person's voice quality, and the background in the simulation video. The rhythm understanding support system according to claim 1 or 2, wherein one of the rhythms is changed according to the music data. 2. The rhythm comprehension support system according to claim 1, wherein the simulation video is a video in which a main person other than the count person dances to the rhythm of the music data along with the count person. 5. The rhythm comprehension support system according to claim 4, wherein the simulation video is a video in which the count person and the main person are lined up. 6. The rhythm comprehension support system according to claim 1, wherein the auxiliary motion is a motion in which the count person rocks at least a portion of his or her body from side to side in accordance with the rhythm of the music data. The rhythm grasping support system according to any one of claims 1 to 6, wherein the rhythm grasping support system is a system for supporting hearing-impaired persons to dance or play in ensemble to music.

Images