JP5098996B2 - Head mounted display - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a head mounted display capable of specifying and indicating who causes an interval difference or a faster/slower tempo even when a plurality of persons make voices simultaneously. <P>SOLUTION: When a power source is turned on and a voice is picked up by a microphone (S13: YES), the input voice begins to be stored (S15) and is compared with a stored musical sound (S17). When it is determined that the acquired voice and the stored musical sound have a difference in one of sound level, tempo, and interval (S21: YES), difference contents are displayed on the head mounted display together with the difference amount (S23) to make the wearing person visually recognize them. A microphone capable of specifying an arrival direction of a voice is used as a microphone to determine the direction where the voice determined to have the difference is output, so that the corresponding person can be specified and displayed. <P>COPYRIGHT: (C)2010,JPO&amp;INPIT

Description

  The present invention relates to a head mounted display. More specifically, the present invention relates to a head mounted display capable of performing display for identifying a person who has emitted a sound when the sound is emitted at a pitch or tempo different from that of a predetermined music piece.

Traditionally, the singing voice uttered by the singer and the performance sound played by the performer are compared with the music sound to extract the pitch and tempo deviation, and by notifying the singer and the performer, A system that supports performance practice has been proposed. For example, as a singing sound practice support system for a singer, it is possible to use a karaoke device capable of comparing the performance sound and the voice of the singer and displaying the difference in pitch when there is a gap between the two. (For example, see Patent Document 1).
Japanese Patent Laid-Open No. 4-13176

  However, the above-described apparatus has a problem in that it is impossible to specify who caused the pitch difference or the tempo deviation when sounds are simultaneously emitted from a plurality of persons.

  The present invention has been made in order to solve the above-mentioned problems, and even if a sound is emitted from a plurality of persons at the same time, the cause of occurrence of a pitch difference or a tempo shift is attributed to any person. It is an object of the present invention to provide a head-mounted display that can identify and notify whether or not.

In order to solve the above-described problems, the head-mounted display according to the first aspect of the present invention includes a sound acquisition unit that acquires sound, and a direction specification that specifies an arrival direction of the sound acquired by the sound acquisition unit. First storage control means for associating and storing the sound acquired by the sound acquisition means and the arrival direction specified by the direction specifying means in the first storage means; It is determined whether the acquired sound information, which is the sound information stored in the storage means, and the music sound information, which is the sound information of the predetermined music and is stored in the second storage means, match. It includes a sound matching determination means for, when it is determined that both are different in the sound matching judgment means, and a display control means for displaying on the display means the arrival direction that corresponds to the acquired sound information with different The sound match determination means The acquired sound amplitude extracting means for extracting the acquired sound amplitude as an amplitude component from the acquired sound information stored in the first storage means, the acquired sound amplitude extracted by the acquired sound amplitude extracting means, and the first Whether or not the acquired sound information and the music sound information match by determining whether or not the music sound amplitude that is the amplitude component of the music sound information stored in the second storage means matches. Amplitude matching judgment means for judging .

According to a second aspect of the present invention, there is provided a head mounted display comprising: a sound acquisition unit that acquires sound; a direction specifying unit that specifies a direction of arrival of the sound acquired by the sound acquisition unit; and a sound acquisition unit. The sound stored in the first storage means and the first storage control means for storing in the first storage means in association with the sound acquired by the direction specifying means and the arrival direction specified by the direction specifying means The sound matching judgment means for judging whether or not the acquired sound information which is the information of the above and the music sound information which is the information of the sound of the predetermined music and stored in the second storage means, Display control means for displaying on the display means the arrival direction corresponding to the acquired sound information determined to be different when the sound match determination means determines that both are different, the sound match determination means, Stored in the first storage means From the acquisition sound information, stored and acquired sound amplitude extraction means for extracting the obtained sound amplitude is an amplitude component, the acquisition sound amplitude extracted by the acquisition sound amplitude extraction means, said second storage means Amplitude matching determination means for determining whether or not the acquired sound information and the music sound information match by determining whether or not the timing matches with the music sound amplitude that is an amplitude component of the music sound information And.
According to a third aspect of the present invention, there is provided a head mounted display comprising: a sound acquisition unit that acquires sound; a direction specifying unit that specifies a direction of arrival of the sound acquired by the sound acquisition unit; and a sound acquisition unit. The sound stored in the first storage means and the first storage control means for storing in the first storage means in association with the sound acquired by the direction specifying means and the arrival direction specified by the direction specifying means The sound matching judgment means for judging whether or not the acquired sound information which is the information of the above and the music sound information which is the information of the sound of the predetermined music and stored in the second storage means, Display control means for displaying on the display means the arrival direction corresponding to the acquired sound information determined to be different when the sound match determination means determines that both are different, the sound match determination means, Stored in the first storage means The acquired sound frequency extracting means for extracting the acquired sound frequency as a frequency component from the acquired sound information, the acquired sound frequency extracted by the acquired sound frequency extracting means, and the second storage means are stored. Frequency matching determination means for determining whether or not the acquired sound information and the music sound information match by determining whether or not the music sound frequency that is a frequency component of the music sound information matches. I have.

In addition to the configuration of the invention according to claim 1 or 2 , the head-mounted display according to claim 4 has a first degree of difference when it is judged that the amplitude coincidence judging means is different. A first difference calculating unit configured to calculate a difference amount; and when the first difference calculating unit calculates the first difference amount, the display control unit outputs the calculated first difference amount to the first difference amount. It is characterized by being displayed on a display means.

In addition to the configuration of the invention of claim 3 , the head mounted display of the invention according to claim 5 is a second difference amount which is a degree of difference when it is judged that the frequency coincidence judging means is different. A second difference calculating means for calculating the second difference amount when the second difference amount is calculated by the second difference calculating means. It is characterized by being displayed.

According to a sixth aspect of the present invention, in addition to the configuration of the fourth aspect of the invention, the first difference amount is calculated when the first difference amount is calculated by the first difference calculation means. A third storage control unit that stores a difference amount in a third storage unit together with the arrival direction associated with the acquired sound information that is the basis of the acquired sound amplitude determined to be different; and the display control unit includes: When the first difference amount is stored in the third storage unit, the first difference amount is displayed on the display unit.

According to a seventh aspect of the present invention, in addition to the configuration of the fifth aspect of the invention, when the second difference amount is calculated by the second difference calculating means, A fourth storage control unit that stores a difference amount in a fourth storage unit together with the arrival direction associated with the acquired sound information that is the basis of the acquired sound frequency determined to be different; and the display control unit includes: When the second difference amount is stored in the fourth storage unit, the second difference amount is displayed on the display unit.

In addition to the configuration of the invention according to any one of claims 1 to 7 , the head-mounted display according to the invention according to claim 8 includes the acquired sound information, the music sound information, and Are determined to be different, the different location information, which is information for specifying the performance location of the predetermined music corresponding to the different location, is associated with the acquired sound information determined to be different. A fifth storage control means for storing the fifth storage means together with the direction, wherein the display control means is specified by the difference information when the difference information is stored in the fifth storage means; that the play position of the predetermined musical composition, before the person playing the predetermined piece of music is played, and characterized by displaying the direction of arrival associated with the different part information on the display means That.

The head mounted display of the invention according to claim 9, in addition to the configuration of the invention according to any one of claims 1 to 8, wherein the acquiring sound information in the sound matching determining means and said music sound information A first accumulating unit for accumulating a difference count, which is the number of times determined to be different, for each of the arrival directions corresponding to the acquired sound information determined to be different; and the display control unit includes: When the number of differences is accumulated, the number of differences is displayed on the display means for each direction of arrival.

In the head-mounted display according to the first aspect, when the acquired sound and the music sound are different, the arrival direction of the sound that is determined to be different can be specified and displayed on the display means. This makes it possible for a conductor or the like wearing a head-mounted display to identify a performer or a singer who is out of the pitch or tempo of the sound being emitted. This makes it possible to bring the performance and singing closer to the ideal state. In addition, the amplitude component of the sound can be extracted and compared with the music sound. As a result, the conductor or the like wearing the head mounted display can specify the difference in sound level between the sound emitted from the performer or singer and the sound of the music. Therefore, it is possible to point out that the sound level should be noted, and it is possible to give a more specific correction instruction.

In the head mounted display of the invention according to claim 2 , when the acquired sound and the music sound are different, it is possible to specify the arrival direction of the sound that is determined to be different and display it on the display means. This makes it possible for a conductor or the like wearing a head-mounted display to identify a performer or a singer who is out of the pitch or tempo of the sound being emitted. This makes it possible to bring the performance and singing closer to the ideal state. Further, since the difference in timing between the acquisition amplitude and music amplitudes are compared, it is possible to specify a tempo deviation of the sound emitted from the performer and singer. Therefore, it is possible to indicate that the tempo should be noted, and it is possible to give a more specific correction instruction.
In the head mounted display according to the third aspect of the present invention, when the acquired sound and the music sound are different, it is possible to specify the direction of arrival of the sound that is determined to be different and display it on the display means. This makes it possible for a conductor or the like wearing a head-mounted display to identify a performer or a singer who is out of the pitch or tempo of the sound being emitted. This makes it possible to bring the performance and singing closer to the ideal state. In addition, the frequency component of the sound can be extracted and compared with the music sound. As a result, the conductor or the like wearing the head mounted display can specify the difference in pitch between the sound emitted from the performer or singer and the sound of the music. Therefore, it is possible to point out that the pitch should be noted, and it is possible to give a more specific correction instruction.

Further, in the head mounted display of the invention according to claim 4, in addition to the effect of the invention of claim 1 or 2 , when the sound emitted from the performer or singer and the music sound are different, It becomes possible to grasp the amount of difference quantitatively. Therefore, it is possible to give a more quantitative correction instruction.

In addition, in the head mounted display of the invention according to claim 5 , in addition to the effect of the invention of claim 3, when the sound emitted from the performer or singer and the music sound are different, the difference between the two Can be grasped quantitatively. Therefore, it is possible to give a more quantitative correction instruction.

In addition, in the head mounted display of the invention according to claim 6 , in addition to the effect of the invention of claim 4, it is possible to perform display for notifying the amount of difference in advance. As a result, a conductor or the like wearing a head-mounted display can promptly perform a precise performance or utterance in advance quantitatively to the performer or singer.

In addition, in the head mounted display of the invention according to claim 7 , in addition to the effect of the invention of claim 5 , it is possible to perform display for notifying the amount of difference in advance. As a result, a conductor or the like wearing a head-mounted display can promptly perform a precise performance or utterance in advance quantitatively to the performer or singer.

Moreover, in the head mounted display of the invention according to claim 8 , in addition to the effect of the invention according to any one of claims 1 to 7 , when a difference between the acquired sound and the music sound occurs, it corresponds to a different part The performance location of the predetermined music to be recorded is stored together with the direction of arrival. And when the same music is emitted again, it becomes possible to perform the display for notifying the corresponding performance location in advance along with the arrival direction. As a result, a conductor or the like wearing a head-mounted display can prompt the player or singer to perform accurately or speak in advance.

In addition, in the head mounted display of the invention according to claim 9 , in addition to the effect of the invention according to any one of claims 1 to 8 , the cumulative number of times when it is determined that the acquired sound information and the music sound information are different. Are displayed on the display means for each direction of arrival. As a result, the conductor or the like wearing the head mounted display can recognize from which direction the error is likely to occur in the incoming sound.

  Hereinafter, embodiments of the head mounted display of the present invention will be described with reference to the drawings. These drawings are used for explaining the technical features that can be adopted by the present invention, and the configuration of the apparatus described, the flowcharts of various processes, etc., unless otherwise specified. It is not intended to be limited to that, but merely an illustrative example.

  First, with reference to FIG. 1, the outline | summary of the choral practice system 1 which is an example of the system by which the head mounted display 2 is used is demonstrated. FIG. 1 is a schematic diagram showing an outline of a choral practice system 1 in which a head mounted display 2 is used.

  As shown in FIG. 1, the head mounted display 2 of the present invention is attached to a conductor 3 who conducts a choir 6, for example. The head mounted display 2 detects the occurrence of a level shift, a tempo shift, and a pitch shift of the voice uttered by the choir 6 and specifies the direction in which the person who uttered these voices is present. Information on the detection result and the specified direction is displayed on the head mounted display 2 and is visually recognized by the conductor 3. As a result, the conductor 3 can identify the person who uttered the sound whose level, tempo, and pitch are shifted, and therefore can identify the person concerned and prompt correction.

  In the choral practice system 1, a microphone 12 for inputting voice is provided (microphone 121 to microphone 125). Audio information detected by the microphone 12 is transmitted to the head mounted display 2 by radio. In the microphones 121 to 125, different frequency channels are set in advance for each microphone in order to prevent interference between radio signals. Thereby, it is possible to distinguish and receive each radio signal on the radio signal receiving side (head mounted display 2, which will be described later). As the microphone 12, a conventionally known microphone having directivity is used. The microphones 12 are arranged so that the direction with high reception intensity (directing direction) is directed to the choir 6 and the microphones 121 to 125 have different directivity directions. In the example shown in FIG. 1, the microphone 121 is arranged in a state where the directing direction is directed to the leftmost direction when the choir 6 is viewed from the conductor 3 side. The microphone 122 is in the middle left direction (between the leftmost direction and the center direction), the microphone 123 is in the center direction, the microphone 124 is in the middle right direction (between the rightmost direction and the center direction), and the microphone 125 is in the rightmost direction. They are arranged with their directing directions facing each other. As a result, the microphone 12 can acquire the voice uttered by each member of the choir 6 by specifying the direction of arrival.

  The head mounted display 2 receives audio information (hereinafter referred to as “audio information”) wirelessly transmitted from the microphone 12. The voice information stored in the flash memory 49 of the head-mounted display 2 (see FIG. 3, described later) in advance, and the voice information of the same music as the music chorused by the choir 6 (hereinafter “ "Musical sound information") and the received audio information are compared. When it is determined that the two are different as a result of the comparison, it is determined that the sound level, tempo, and pitch of a specific person in the choir 6 are shifted. A notification that the sound level, tempo, and pitch are shifted is displayed on the head mounted display 2 together with information on the arrival direction of the corresponding sound. As a result, the conductor 3 can specify the person who uttered the sound whose pitch or tempo is shifted.

  In the present embodiment, a plurality of directional microphones 12 are used, but the present invention is not limited to this configuration. Therefore, for example, a single microphone array that can specify the direction of arrival of sound may be used. The microphone 12 and the head mounted display 2 communicate audio information using radio waves, but the present invention is not limited to this. Accordingly, the microphone 12 and the head mounted display 2 may be connected via a communication cable, and audio information may be communicated via the communication cable.

  A schematic configuration of the head mounted display 2 will be described with reference to FIG. FIG. 2 is a diagram showing an external configuration of the head mounted display 2 that is mounted on the conductor 3.

  As shown in FIG. 2, the head-mounted display 2 emits laser light modulated and scanned in accordance with an image signal (hereinafter referred to as “image light 4”), and is emitted to the retina of at least one eye of the conductor 3. Project the image directly. Thereby, the conductor 3 is made to visually recognize an image. As shown in FIG. 2, the head mounted display 2 emits image light 4 according to an image signal, guides the image light 4 emitted from the emission device 100, and directs the image light 4. It includes at least a prism 150 that reflects toward the eye of the person 3 and a support member 200 that supports the emission device 100 and the prism 150.

  The emission device 100 causes the image light 4 to be emitted to the prism 150. The prism 150 is in a fixed position with respect to the emitting device 100, guides the image light 4 emitted from the emitting device 100, and reflects the image light 4 toward the conductor 3 's eyes. The prism 150 includes a beam splitter (not shown), transmits the external light 5 from the outside, and guides it to the eyes of the conductor 3. The prism 150 causes the image light 4 incident from the side of the conductor 3 to enter the eyes of the conductor 3 and causes the external light 5 from the outside to enter the eyes of the conductor 3. Thereby, not only an actual field of view but also an image based on the image light 4 emitted from the emission device 100 can be visually recognized.

  The head mounted display 2 includes a camera 7. The camera 7 is an element that can image the visual field direction of the conductor 3. The image information obtained as a result of the imaging is used as a basis for an image to be visually recognized by the conductor 3 according to the situation. For example, the arrangement state of the members of the choir 6 is grasped based on the image information obtained as a result of being photographed by the camera 7. And the arrangement | positioning state of the grasped member and the direction of arrival of a sound are compared, and the person who generated the sound level shift, the tempo shift, and the pitch shift is specified. In this way, information for identifying a specific person among the members of the choir 6 can be displayed on the head mounted display 2 and can be visually recognized by the conductor 3.

  In the present embodiment, the prism 150 allows the external light 5 and the image light 4 to be viewed simultaneously. However, the present invention is not limited to this configuration, and for example, a half mirror can be used instead of the prism 150. As a result, the image light 4 from the emitting device 100 can be reflected by the half mirror and incident on the conductor 3, and the external light 5 can be transmitted through the half mirror and incident on the conductor 3. Become.

  In this embodiment, a see-through head mounted display using retinal scanning is described as an example of the head mounted display. However, the present invention is not limited to this example. For example, a head using a reflective LCD It may be a mount display.

  As the camera 7 of the present embodiment, a conventionally known element can be used. For example, a CMOS image sensor or a CCD image sensor can be used.

  The electrical configuration of the head mounted display 2 will be described with reference to FIG. FIG. 3 is a schematic diagram showing an electrical configuration of the head mounted display 2.

  As shown in FIG. 3, the head mounted display 2 includes a display unit 40 for allowing the conductor 3 to visually recognize an image, an input unit 41 for performing various operations and data input, and a proximity for receiving audio information from the microphone 12. A distance wireless communication unit 43, a flash memory 49 that stores information related to audio and video, a control unit 46 that controls the entire head mounted display 2, a camera 7, an acceleration sensor 14, and a power supply unit 47 are provided.

  The display unit 40 receives a control signal (video signal) formed by the control unit 46 and is necessary for causing the conductor 3 to visually recognize the video, and directly projects it onto the retina of the conductor 3. A video signal processing unit 70 is provided for conversion into a signal. Further, a laser group 72 (blue output laser (B laser) 721, green output laser (G laser) 722, red output laser (R laser) 723) for outputting laser light, and laser light output from the laser A laser driver group 71 that performs control is provided. The video signal processing unit 70 is electrically connected to the laser driver group 71 so that a desired laser beam can be output at a desired timing under the control of the video signal processing unit 70. The laser driver group 71 is electrically connected to the B laser 721, the G laser 722, and the R laser 723, respectively. Further, the video signal processing unit 70 and the control unit 46 are electrically connected via a bus so that the video signal processing unit 70 can receive a video signal from the control unit 46.

  The display unit 40 also includes a galvano mirror 812 that is a vertical scanning mirror that performs scanning by reflecting the laser beam output from the laser in the vertical direction, a vertical scanning control circuit 811 that performs drive control of the galvano mirror 812, and an output from the laser. A galvano mirror 792 that is a horizontal scanning mirror that performs scanning by reflecting the laser beam that is reflected in the horizontal direction and a horizontal scanning control circuit 791 that controls driving of the galvano mirror 792 are provided. The video signal processing unit 70, the vertical scanning control circuit 811, and the horizontal scanning control circuit 791 are electrically connected so that the laser light can be reflected in a desired direction by the control of the video signal processing unit 70. It is connected to the. The vertical scanning control circuit 811 is electrically connected to the galvanometer mirror 812. The horizontal scanning control circuit 791 is electrically connected to the galvanometer mirror 792.

  The input unit 41 includes an operation button group 50 including various function keys and the like, and an input control circuit 51 that detects that a key of the operation button group 50 is operated and notifies the control unit 46 of the operation. The operation button group 50 is electrically connected to the input control circuit 51 so that the information input to the keys of the operation button group 50 can be recognized by the control unit 46. The input control circuit 51 is electrically connected to the control unit 46.

  The short-range wireless communication unit 43 controls the short-range wireless communication module 57 that receives radio waves and the short-range wireless communication control circuit 57 that controls the short-range wireless communication module 57 and receives audio information transmitted from the microphone 12. 58. The short-range wireless communication modules 57 are provided by the number of microphones 12 (121 to 125) to be installed. Each short-range wireless communication module 57 is associated with each microphone 12 (121 to 125), and the reception frequency is set so as to be able to receive an audio signal transmitted from the associated microphone 12. Is set. With such a configuration, the audio information received by the short-range wireless communication module 57 can be associated with one of the microphones 12. The received audio signal is converted into a predetermined format, and is associated with information (microphone identification information) for identifying the transmission source microphone 12 and stored in the flash memory 49 (details will be described later). ). The control unit 46 and the short-range wireless communication control circuit 58 are electrically connected via a bus so that the control unit 46 can acquire an audio signal. The short-range wireless communication module 57 and the short-range wireless communication control circuit 58 are electrically connected.

Note that the communication method of the short-range wireless communication module 57 is not particularly limited, and a conventionally known wireless communication method can be used. For example, a wireless communication system based on Bluetooth (registered trademark), UWB (Ultra Wide Band) standard, wireless LAN (IEEE802.11b, etc.) standard, WirelessUSB standard, or the like can be used. In addition, a wireless communication system based on IrDA (Infrared Data Association) standard using infrared rays can be used.

  The camera 7 includes an image sensor that can image the visual field direction of the conductor 3. And the camera 7 and the control part 46 are electrically connected so that the image information of the imaged result can be recognized in the control part 46. FIG.

  The acceleration sensor 14 is used to specify the direction in which the visual field of the conductor 3 wearing the head mounted display 2 is facing. In the head mounted display 2, the movement angle from the reference position determined at the time of initialization is obtained by the acceleration sensor 14, and the visual field direction of the conductor 3 is specified. The identified visual field direction is compared with the direction in which the microphones 121 to 125 are installed. Thereby, matching processing between the directivity direction of the microphones 121 to 125 and the visual field direction is performed (details will be described later).

  The power supply unit 47 supplies a battery 59 serving as a power source for driving the head mounted display 2 and power of the battery 59 to the head mounted display 2 and also supplies power supplied from a charging adapter (not shown). A charge control circuit 60 that supplies the battery 59 to charge the battery 59 is provided.

  In the flash memory 49, information obtained as a result of converting received audio information into a predetermined audio format (hereinafter referred to as "audio data information") and information obtained by converting music sound information into a predetermined audio format (hereinafter referred to as "audio data information"). "Musical sound data information"), and image data (hereinafter referred to as "video information") such as images (graphics) and text to be visually recognized by the conductor 3. The file format is not particularly limited, and conventionally known file formats (WAV, WMA, MP3, etc.) can be used. The flash memory 49 is electrically connected to the control unit 46 via the bus so that the information stored in each storage area can be referred to by the control unit 46.

  The control unit 46 has a function of controlling the entire head mounted display 2. For example, a process of receiving voice information from the microphone 12 via the short-range wireless communication unit 43, a process of displaying desired information on the display unit 40, and a predetermined operation according to the operation of the input unit 41 by the conductor 3 are performed. Processing and the like are executed by the control unit 46. The control unit 46 includes a CPU 61, a ROM 62 that stores various programs, a RAM 48 that temporarily stores various data, and the like. In the control unit 46, the CPU 61 reads out various programs stored in the ROM 62, and executes the processes described above. The RAM 48 provides storage areas for various flags and data required when the CPU 61 executes each process.

  The outline of the process of forming the image light 4 on the display unit 40 will be described in detail with reference to FIG. FIG. 4 is a schematic diagram showing an electrical and optical configuration of the display unit 40. As shown in FIG. 4, the display unit 40 includes a light source unit 65, a collimating optical system 77, a horizontal scanning system 79, a first relay optical system 80, a vertical scanning system 81, and a second relay optical system 82. The light source unit 65 includes a video signal processing unit 70, a laser driver group 71, a laser group 72, a collimating optical system 73, a dichroic mirror group 74, and a coupling optical system 75. The horizontal scanning system 79 includes a horizontal scanning control circuit 791 and a galvanometer mirror 792. The vertical scanning system 81 includes a vertical scanning control circuit 811 and a galvanometer mirror 812.

  The configuration of the light source unit 65 will be described in detail. The video signal processing unit 70 is electrically connected to the control unit 46 as described above. Then, the video signal formed by the control unit 46 is input. A luminance signal 66 (B luminance signal 661, G luminance signal 662, R luminance signal 663), vertical synchronizing signal 67, and horizontal synchronizing signal 68 are electrically connected to the video signal processing unit 70, respectively. In the video signal processing unit 70, each signal (luminance signal, vertical synchronization signal, horizontal synchronization signal) that is an element for projecting the input video information onto the retina is generated. Each of the generated signals is output for each pixel with respect to the luminance signal 66, the vertical synchronization signal 67, and the horizontal synchronization signal 68.

  The luminance signal 66 (B luminance signal 661, G luminance signal 662, R luminance signal 663) is electrically connected to the laser driver group 71 (B laser driver 711, G laser driver 712, R laser driver 713), respectively. . The horizontal synchronization signal 68 is connected to the horizontal scanning control circuit 791 of the horizontal scanning system 79. The vertical synchronization signal 67 is connected to the vertical scanning control circuit 811 of the vertical scanning system 81.

  The laser driver group 71 is electrically connected to the laser group 72 (B laser 721, G laser 722, R laser 723), respectively. The laser driver group 71 applies laser light that has been intensity-modulated based on each luminance signal of each signal transmitted via the luminance signal 66 (B luminance signal 661, G luminance signal 662, R luminance signal 663). In order to emit light from 72, the laser group 72 is driven.

  Further, a collimating optical system 73 (731 to 733) capable of collimating three colors (blue, green, and red) of laser light emitted from the laser group 72 under the control of the laser driver group 71 into parallel light, and a collimator A dichroic mirror group 74 (741 to 743) capable of combining the laser beams collimated by the optical system 73 and a coupling optical system 75 for guiding the combined laser light to the optical fiber 76 are provided. It is done.

  As the laser group 72 (B laser 721, G laser 722, R laser 723), a semiconductor laser such as a laser diode or a solid-state laser may be used.

  The configuration of the display unit 40 excluding the light source unit 65 will be described in detail. A collimating optical system 77 for guiding the laser light guided from the light source unit 65 to the optical fiber 76 to the galvanometer mirror 792 of the horizontal scanning system 79 is provided. The galvanometer mirror 792 scans the laser beam in the horizontal direction. The laser light incident on the deflection surface 793 of the galvanometer mirror 792 is scanned in the horizontal direction in synchronization with the horizontal synchronization signal 68 received from the video signal processing unit 70 under the control of the horizontal scanning control circuit 791. The horizontal scanning system 79 of this embodiment is an optical system for horizontally scanning a laser beam in the horizontal direction for each scanning line of an image to be displayed.

  Further, a first relay optical system 80 for guiding the horizontally scanned laser light to the vertical scanning system 81 and a galvanometer mirror 812 for scanning the laser light guided by the first relay optical system 80 in the vertical direction are provided. It is done. The laser light incident on the deflection surface 813 of the galvanometer mirror 812 is scanned in the vertical direction in synchronization with the vertical synchronization signal 67 received from the video signal processing unit 70 under the control of the vertical scanning control circuit 811. The vertical scanning system 81 of this embodiment is an optical system that vertically scans a laser beam vertically from the first scanning line to the last scanning line for each frame of an image to be displayed.

  A second relay optical system 82 for guiding the vertically scanned laser light (image light 4) to the prism 150 is also provided. The image light 4 guided by the second relay optical system 82 enters the prism 150. As described above, the prism 150 is arranged between the second relay optical system 82 and the pupil 90 of the conductor 3. The prism 150 guides the image light 4 to the user's pupil 90 by totally reflecting it.

  In the display unit 40 described above, the horizontal scanning system 79 is designed to scan the laser beam at a higher speed, that is, at a higher frequency than the vertical scanning system 81. The first relay optical system 80 is provided so that the galvanometer mirror 792 of the horizontal scanning system 79 and the galvanometer mirror 812 of the vertical scanning system 81 are conjugate. The second relay optical system 82 is provided so that the galvanometer mirror 812 and the pupil 90 of the user are conjugate.

  A process from when the head-mounted display 2 according to the embodiment of the present invention receives an image signal from the outside to when the image is projected onto the retina of the user will be described with reference to FIG.

  In the head mounted display 2 of the present embodiment, the video signal processing unit 70 provided in the light source unit 65 receives a video signal from the control unit 46. Next, the video signal processing unit 70 outputs a luminance signal 66 including a B luminance signal 661, a G luminance signal 662, and an R luminance signal 663 for outputting laser beams of red, green, and blue, a horizontal synchronization signal 68, A vertical synchronizing signal 67 is output.

  The B laser driver 711, the G laser driver 712, and the R laser driver 713 are respectively applied to the B laser 721, the G laser 722, and the R laser 723 based on the input B luminance signal 661, G luminance signal 662, and R luminance signal 663. Output each drive signal.

  Based on the drive signal described above, the B laser 721, the G laser 722, and the R laser 723 each generate intensity-modulated laser light. The generated laser light is output to the collimating optical system 73. Each of the laser beams is collimated into parallel light by the collimating optical system 73, and further, is incident on the dichroic mirror group 74 to be combined into one laser beam. The combined laser beam is guided to enter the optical fiber 76 by the coupling optical system 75.

  The laser light transmitted through the optical fiber 76 is guided from the optical fiber 76 to the collimating optical system 77. Then, the light is emitted to the horizontal scanning system 79.

  The laser light incident on the horizontal scanning system 79 is scanned in the horizontal direction in synchronization with the horizontal synchronization signal 68 on the deflection surface 793 of the galvanometer mirror 792. The galvanometer mirror 792 is reciprocally oscillated in synchronization with the horizontal synchronizing signal 68 so that the deflecting surface 793 reflects incident light in the horizontal direction. The galvano mirror 792 scans the laser beam in the horizontal direction. The horizontally scanned laser light is emitted to the vertical scanning system 81 via the first relay optical system 80.

  In the first relay optical system 80, the deflection surface 793 of the galvanometer mirror 792 and the deflection surface 813 of the galvanometer mirror 812 are adjusted to have a conjugate relationship, and the surface tilt of the galvanometer mirror 792 is corrected.

  The laser light incident on the vertical scanning system 81 is scanned in the vertical direction in synchronization with the vertical synchronization signal 67 on the deflection surface 813 of the galvanometer mirror 812. The galvanometer mirror 812 is reciprocally oscillated so that the deflection surface 813 reflects incident light in the vertical direction in synchronization with the vertical synchronization signal 67 in the same manner as the galvanometer mirror 792 is synchronized with the horizontal synchronization signal 68. . Laser light is scanned in the vertical direction by the galvanometer mirror 812.

  The laser light (image light 4) scanned two-dimensionally in the vertical and horizontal directions by the horizontal scanning system 79 and the vertical scanning system 81 has a conjugate relationship between the deflection surface 813 of the galvano mirror 812 and the pupil 90 of the user. The light is incident on the user's pupil 90 by the second relay optical system 82 and the prism 150, and projected onto the retina.

  Through the process described above, the conductor 3 can recognize the image by the image light 4 that is two-dimensionally scanned and projected onto the retina. The galvanometer mirror 792 of the horizontal scanning system 79 and the galvanometer mirror 812 of the vertical scanning system 81 have the same names, but their reflecting surfaces are swung (rotated) so as to scan light. It goes without saying that any drive system such as a resonance type, non-resonance type, piezoelectric drive, electromagnetic drive, electrostatic drive, or the like may be used.

  The storage area of the flash memory 49 and the stored information will be described with reference to FIGS. FIG. 5 is a schematic diagram showing a storage area of the flash memory 49. FIG. 6 is a schematic diagram showing audio data related information stored in the audio related storage area 491. FIG. 7 is a schematic diagram showing difference information stored in the difference information storage area 493.

  The storage area of the flash memory 49 will be described with reference to FIG. As shown in FIG. 5, the flash memory 49 is provided with an audio-related storage area 491, a music sound storage area 492, a difference information storage area 493, an audio storage area 494, a video storage area 495, and other areas. . The sound related storage area 491 stores information related to the sound data information (hereinafter referred to as “sound data related information”). The music sound storage area 492 stores music sound data information. The difference information storage area 493 stores information (hereinafter referred to as “difference information”) as a result of comparison between the audio data information and the music sound data information. Audio data information is stored in the audio storage area 494. Video information is stored in the video storage area 495. Details of the stored information will be described below.

  The audio data related information stored in the audio related storage area 491 will be described with reference to FIG. In the example illustrated in FIG. 6, information items “file name” and “microphone identification information” are provided as audio data related information. As the “file name”, the file name of the audio data information is stored. As the “microphone identification information”, the identification information of the microphone 12 associated with the short-range wireless communication module 57 that has received the voice that is the basis of the voice data information is stored.

  In the head mounted display 2, when audio information is received from the microphone 12, the acquired audio information is converted into a predetermined audio data format to generate audio data information. The created audio data information is stored in the audio storage area 494. The information of the file name of the audio data information is associated with the identification information of the microphone 12 that transmitted the basic audio information, and is stored in the audio related storage area 491 as the audio data related information. Since the microphone 12 has directivity, the voice arrival direction can be specified by storing the voice data information and the identification information of the microphone 12 in association with each other.

  In the example shown in FIG. 6, the file name of the audio data information is “AA1.mp3”, and the microphone identification information is “121”. Therefore, audio information that is the basis of “AA1.mp3” is emitted from the leftmost direction when the choir 6 is viewed from the conductor 3 side. The file name of the audio data information is “AA2.mp3”, and the microphone identification information is “122”. Therefore, voice information that is the basis of “AA2.mp3” is emitted from the left middle direction when the choir 6 is viewed from the conductor 3 side.

  The music sound data information stored in the music sound storage area 492 will be described. As the music sound data information, a plurality of music sound data information of music to be sung by the choir 6 is stored in advance. These pieces of information are used as comparison targets when voice information is received as ideal choral voice data information. If it is determined that there is a difference between the two, it is determined that a deviation (sound level, tempo, pitch, etc.) has occurred in the received audio information (details will be described later).

  The difference information stored in the difference information storage area 493 will be described with reference to FIG. As for the difference information, when the audio data information and the music sound data information are compared and it is determined that the two sounds are different, information regarding the difference contents is stored. The difference information is associated with the music pieces of the music data information compared and stored for each music piece. In FIG. 7, only the difference information corresponding to the only music is shown.

  As shown in FIG. 7, information items of “microphone identification information”, “difference portion”, “difference content”, “difference amount”, and “total number of times” are provided as difference information. As “microphone identification information”, identification information of the microphone 12 is stored. As the “difference part”, the elapsed time from the start point of the music to the part determined to be different is stored as information on the part determined to be different.

  As “difference content”, the difference content when it is determined that there is a difference is stored. In the example shown in FIG. 7, “level” indicating that the amplitude levels of both voices are different and “level deviation” has occurred, and “tempo” indicating that both voices have different timings and “tempo deviation” has occurred. ”And“ Pitch ”indicating that the frequencies of both voices are different and“ pitch deviation ”is occurring, is stored as“ difference content ”.

  As the “difference amount”, a quantitative value of the degree of difference when it is determined that there is a difference is stored. In the example shown in FIG. 7, when the difference content is “level deviation”, an amplitude difference (unit: dB) is stored. When the difference is “tempo shift”, the timing shift time (unit: s) is stored. When the difference is “pitch deviation”, the frequency difference (unit: Hz) is stored. As the “total number of times”, the number of differences is stored for each difference content.

  In the example shown in FIG. 7, as a result of comparing the audio data information of the audio received from the microphone 121 with the music sound data information, the tempo is delayed by 0.5 s at the timing after 10 s from the start of the music, and the cumulative number is 2 It is. The tempo advances by 1 s at the timing after 24 s from the start of the music, and the cumulative number is 1. Further, as a result of comparing the audio data information of the audio received from the microphone 122 with the music sound data information, the pitch is shifted to the high frequency side of 0.3 Hz at the timing after 21 s from the start of the music, and the cumulative number is 3. is there. The level is 3 dB lower at the timing after 36 s from the start of the music, and the cumulative number is 4.

  In the audio storage area 494, audio data information corresponding to the file name of the audio data related information stored in the audio related storage area 491 is stored.

  Examples of the video information stored in the video storage area 495 include member notification video information for identifying a predetermined member of the choir 6 and difference amount video information for quantitatively visualizing the difference amount. Remembered. The video information is read from the CPU 61 of the control unit 46 when processing for causing the conductor 3 to visually recognize the video is executed.

  Details of processing executed by the CPU 61 of the head mounted display 2 will be described with reference to FIGS. FIG. 8 is a flowchart of the difference notification process executed by the CPU 61. FIG. 9 is a subroutine flowchart of the difference specifying process. The difference notification process is activated and executed by the CPU 61 when the power of the head mounted display 2 is turned on.

  A flag used in the difference notification process will be described. The flag 1 is used to specify whether or not a level shift has occurred. 1 is stored when a level shift occurs. The flag 2 is used to specify whether or not a tempo shift has occurred. 1 is stored when a tempo shift occurs. The flag 3 is used to specify whether or not a pitch shift has occurred. 1 is stored when a pitch shift occurs. These flags are stored in the RAM 48, and 0 is stored and initialized when the power is turned on. Hereinafter, it is referred to as “turned off” when 0 is stored in the above flag, and “turned on” when 1 is stored.

  As shown in FIG. 8, in the CPU 61 of the head mounted display 2, first, information from the acceleration sensor 14 is referred to, and the visual field direction of the conductor 3 wearing the head mounted display 2 is specified (S11). Thereby, matching with the directivity direction of the microphones 121-125 and the visual field direction of the conductor 3 is made. Specifically, it is specified which of the directional directions of the microphones 121 to 125 matches the central direction of the visual field of the conductor 3. Thereby, the correlation between the display area of the head mounted display 2 and the directivity direction of the microphone 12 is established.

  Next, the choir 6 is photographed by the camera 7. Image information of the photographed result is subjected to image processing, and the arrangement state of the members of the choir 6 is specified. And the arrangement | positioning state of the member of the specified choir 6 and the directivity direction of the microphone 12 mentioned above are related, and the member of the choir 6 arrange | positioned in the directivity direction of the microphone 12 is specified, respectively. .

  Next, it is determined whether audio information has been received from the microphone 12 via the short-range wireless communication module 57 and the short-range wireless communication control circuit 58 (S13). When no audio information is received from the microphone 12 (S13: NO), the process returns to S11 without performing any special process, and the process of specifying the visual field direction of the conductor 3 wearing the head-mounted display 2 is executed continuously. Is done.

  When voice information is received via the short-range wireless communication module 57 and the short-range wireless communication control circuit 58 (S13: YES), conversion processing of the received voice information into voice data information is then started. The converted audio data information is stored in the audio storage area 494 of the flash memory 49. Further, the identification information of the microphone 12 associated with the short-range wireless communication module 57 that has received the voice information that is the basis of the voice data information is stored in the voice-related storage area 491 together with the file name of the converted voice data information. It is stored in the stored audio data related information (S15).

  Next, a process is performed in which the music that is the basis of the audio information whose reception is started in S13 is specified for which music is selected and the corresponding music sound data information is selected. Specifically, first, the voice data information started to be stored in S15 is voice-recognized. In speech recognition, phrases of lyrics included in speech data information are extracted and decomposed into phonemes. Next, the pitch and phoneme length of the speech waveform for each decomposed phoneme are compared with the pitch and phoneme length based on the music sound data information stored in the music sound storage area 492. Then, the music sound data information when the two match is selected as the music sound data information to be compared (S17).

  The above selection process is an example, and the music sound data information may be selected by another method. For example, music sound data information to be compared may be selected based on music information input in advance via the input unit 41.

  Simultaneously with the selection process described above, it is specified which performance part in the music sound data information corresponds to the sound of the audio information being received. When the corresponding sound is specified, the elapsed time from the start of performance of the music sound data information to the timing at which the sound is played is stored in the counter of the RAM 48. Thereafter, the counter is updated over time. In the difference specifying process (see FIG. 9), which will be described later, the information at the time when the elapsed time specified by the counter in the music sound data information has elapsed is compared with the audio data information of the audio information immediately after reception.

  Next, the audio information portion immediately after reception in the audio data information started to be stored in S15 is compared with the selected music sound data information, resulting in a level shift, tempo shift, and pitch shift between the two. It is determined whether or not there is (S19).

  The difference identification process (S19, see FIG. 8) will be described with reference to FIG. In the difference specifying process, the voice information portion immediately after reception is extracted from the voice storage area 494 for each microphone identification information in the voice data information that has been stored. Then, it is compared with the music sound data information selected in S17 (see FIG. 8), and the presence / absence of a deviation (level deviation, tempo deviation, pitch interval) between them is determined. If the amplitude level is different between the two, it is determined that a “level shift” has occurred, if the timing is shifted, it is determined that a “tempo shift” has occurred, and if the frequency is different, It is determined that a “pitch deviation” has occurred. Based on the counter stored and updated in the RAM 48, the voices of the portions where the performance parts match between the voice data information and the music sound data information are compared and executed.

  As shown in FIG. 9, first, an audio amplitude component is extracted based on audio data information. The extracted amplitude component is compared with the amplitude component of the voice extracted based on the music sound data information, and it is determined whether or not both amplitude levels match (S41). If it is determined that both amplitude levels match (S41: NO), it is determined that there is no level shift between the two, and the process proceeds to S49.

  If it is determined that the two amplitude levels are different (S41: YES), the flag 1 is turned on to indicate that a level shift has occurred (S43). Next, a difference between the amplitude level based on the audio data information and the amplitude level based on the music sound data information is calculated, and is set as a difference amount of the level deviation (S45). Next, the elapsed time from the performance start time is specified based on the value of the counter (the specified information is referred to as “difference part information”). The difference contents (level deviation) specified in S41, the difference amount calculated in S45, and the difference part information are included in the RAM 48 together with the identification information of the microphone 12 to which the voice information of the target voice is transmitted. Is temporarily stored (S47). Then, the process proceeds to S49.

  In S49, an audio amplitude component is extracted based on the audio data information. The extracted amplitude component is compared with the amplitude component of the voice extracted based on the music sound data information, and it is determined whether or not the peak positions of both amplitudes match (S49). If it is determined that the peak positions of both amplitudes match (S49: NO), it is determined that there is no tempo shift between the two, and the process proceeds to S57.

  If it is determined that the peak positions of both amplitudes are different (S49: YES), the flag 2 is turned on to indicate that a tempo shift has occurred (S51). Next, the difference between the peak position of the amplitude based on the audio data and the peak position of the amplitude based on the music sound data information is calculated and used as the difference amount of the tempo shift (S53). Next, the difference location information is specified based on the value of the counter. Then, the difference content (tempo shift) specified in S49, the difference amount calculated in S53, and the difference part information are included in the RAM 48 together with the identification information of the microphone 12 to which the voice information of the target voice is transmitted. Is temporarily stored (S55). Then, the process proceeds to S57.

  In S57, the frequency component of the voice is extracted based on the voice data information. The extracted frequency component is compared with the frequency component of the sound extracted based on the music sound data information, and it is determined whether or not both frequencies match (S57). For example, among the frequency components of the audio based on the audio data information, the maximum frequency component is compared with the audio frequency component extracted based on the music sound data information, and it is determined that both frequencies match. In this case (S57: NO), it is determined that there is no pitch difference between the two, and the process proceeds to S67.

  If it is determined that the two frequencies are different (S57: YES), the flag 3 is turned on to indicate that a pitch shift has occurred (S59). Next, a difference between the frequency based on the audio data information and the frequency based on the music sound data information is calculated, and is set as a difference amount of the frequency deviation (S61). Next, the difference location information is specified based on the value of the counter. Then, the difference content (pitch deviation) specified in S57, the difference amount calculated in S61, and the difference part information together with the identification information of the microphone 12 to which the voice information of the target voice is transmitted are stored in the RAM 48. Is temporarily stored (S63). Then, the process proceeds to S67.

  In S67, it is determined whether or not the above-described processing has been executed for all audio data information stored in the audio storage area 494 (S67). If there is audio data information for which comparison processing has not been performed (S67: NO), audio data information for audio for which the above-described processing has not been performed among the audio received by the microphones 121 to 125 remains. It is judged that In this case, the process returns to S41, and the above-described processing is repeatedly performed for the remaining audio data information. On the other hand, if the above comparison process has been executed for all audio data information (S67: YES), the difference specifying process is terminated and the process returns to the difference notification process.

  As shown in FIG. 8, after the difference specifying process, it is determined whether any of a level shift, a tempo shift, and a pitch shift has occurred (S21). If it is determined that at least one of the deviations has occurred, at least one of the flags 1 to 3 is set to ON. Therefore, these flags are referred to. If any of the flags is ON (S21: YES), processing for displaying that a deviation has occurred is executed (S23 to S27). On the other hand, if none of the flags are turned on (S21: NO), there is no need to perform a video display process for the conductor 3, so the process proceeds to S29 without performing any special process.

  In S23, the type of deviation that has occurred is determined based on the flag information. If flag 1 is ON, it is determined that a level shift has occurred. When flag 2 is ON, it is determined that a tempo shift has occurred. When the flag 3 is ON, it is determined that a pitch shift has occurred.

  The microphone identification information, the difference contents, and the difference amount stored in the RAM 48 are read out. Next, video information for displaying such information and causing the conductor 3 to visually recognize it is read from the video storage area 495 of the flash memory 49. Based on the read video information, a video signal is formed by the control unit 46 and transmitted to the video signal processing unit 70 of the display unit 40. In the display unit 40, the image light 4 to be emitted to the retina for the conductor 3 to visually recognize is formed. The formed image light 4 is emitted to the retina of the conductor 3. Thereby, the conductor 3 is made to visually recognize the difference information (direction of the corresponding member, difference content, difference amount) (S23).

  With reference to FIG. 10, an example of an image visually recognized by the conductor 3 through the head mounted display 2 will be described. FIG. 10 is a schematic diagram illustrating an example of an image visually recognized by the conductor 3 via the head mounted display 2.

  As shown in FIG. 10, in order to notify the visual field of the conductor 3 of the display 8 for identifying the members of the choir 6 that directly pass through the prism 150, some members, the difference contents and the difference amount. Are displayed. The display 8 is displayed by associating the microphone identification information stored in the RAM 48 with the position of the member recognized through the image processing based on the image captured by the camera 7. In the example shown in FIG. 10, the display when the sound data information of the sound received by the microphone 122 (directing direction: left middle direction, see FIG. 1) is compared with the music sound data information and is determined to be different is illustrated. ing. Further, the display 9 displays the difference content and the difference amount stored in the RAM 48. By causing the conductor 3 to visually recognize such an image, the conductor 3 can identify the members of the chorus 6 who made the wrong utterance, and can recognize the content and amount of the difference. .

  Next, as shown in FIG. 8, the elapsed time from the start of display is monitored in order for the conductor 3 to visually recognize the displayed video for a certain period (S25: NO). When the elapsed time has passed a predetermined time (for example, 1 s) (S25: YES), the display is stopped (S27). Then, the process proceeds to S29.

  In S29, it is determined whether or not the voice signal is continuously received from the microphone 12 (S29). When the audio signal is continuously received (S29: NO), the process returns to S19 and the above-described process is repeatedly executed. When the transmission of the audio signal from the microphone 12 is stopped and not received (S29: YES), the audio signal storing process started in S15 is stopped (S31).

  Next, information stored in the RAM 48 (microphone identification information, difference content, difference amount, and difference portion information) is stored as difference information (see FIG. 7) in the difference information storage area 493 (S33). First, the difference information corresponding to the music piece to be compared is selected. Next, it is determined whether or not the same difference contents and different portions are already stored as difference information in the difference information storage area 493 as the same microphone identification information. When the same difference contents and difference portions are not stored as difference information, information (microphone identification information, difference contents, difference amount, difference portion information) stored in the RAM 48 is newly stored as difference information. . When the same difference content and difference part are already stored in the difference information storage area 493 as difference information, the accumulated number of stored information is updated by adding one. Then, the difference notification process is terminated.

  As described above, the head-mounted display 2 identifies the direction in which members of the choir 6 are causing a level shift, a tempo shift, and a pitch shift. It can be displayed together with the conductor 3 so that the conductor 3 can visually recognize it. As a result, the conductor 3 can prompt the corresponding member to correct the chorus, and the chorus of the choir 6 can be brought close to an ideal state.

  In addition, the head-mounted display 2 can specify the difference content and the difference amount and notify the conductor 3 when the level shift, the tempo shift, and the pitch shift occur. It becomes possible to give more specific correction instructions to the members of the choir 6.

  In addition, this invention is not limited to the said embodiment, A various change is possible. In the above-described embodiment, the voice uttered by the choir 6 is judged to be different from the ideal music voice, and the conductor 3 is notified when there is a difference. However, the present invention is not limited to the determination of the difference in the voice of the choir, and may be configured to determine the difference in the performance voice by, for example, an orchestra or a brass band.

  In the above embodiment, the head mounted display 2 includes the short-range wireless communication module 57 and receives audio information from the microphone 12. However, the present invention is not limited to this configuration. Therefore, the head mounted display 2 and the microphone 12 may be connected via a wired cable, and audio information may be communicated via the wired cable. A device (receiver) that receives audio information from the microphone 12 and converts the format may be provided separately. In this case, the head mounted display 2 receives the audio data information whose format has been converted from the receiver together with the microphone identification information, whereby the above-described processing is executed.

In the above-described embodiment, the voice immediately after reception is compared with the voice of the music, and when it is determined that the two are different, the conductor 3 is made to visually recognize the content of the difference. However, the present invention is not limited to this configuration. For example, in the case where the same music is repeatedly sung, if the difference has occurred in the past, the difference information is notified to the conductor 3 in advance. Also good. As a result, the conductor 3 can call attention to the members of the choir 6 in advance. Hereinafter, modifications of the present embodiment will be described.
<Modification>

  Hereinafter, modifications of the present embodiment will be described with reference to the drawings. FIG. 11 is a flowchart of the difference notification process in the modification. FIG. 12 is a subroutine flowchart of the difference pre-display process in the modified example. The difference notification process is activated and executed by the CPU 61 when the power of the head mounted display 2 is turned on. In the following, description of the same parts as those of the above-described embodiment will be omitted or simplified.

  In the modification, the head mounted display 2 displays the contents of the difference information stored in the difference information storage area 493 in advance and is recognized by the conductor 3 before the portion corresponding to the difference is choral by the choir 6. Let Since the occurrence of level shift, tempo shift, and pitch shift is assumed to be repeated at the same location in the music, the conductor 3 recognizes the location where the shift is likely to occur in advance and It is possible to call attention to the members.

  A flowchart of the difference notification process will be described with reference to FIG. As shown in FIG. 11, in the CPU 61 of the head mounted display 2, first, information from the acceleration sensor 14 is referred to, and the visual field direction of the conductor 3 wearing the head mounted display 2 is specified (S11). Next, it is determined whether audio information is received from the microphone 12 (S13). When no audio information is received from the microphone (S13: NO), the process returns to S11 without performing any special process, and the process of specifying the visual field direction of the conductor 3 wearing the head-mounted display 2 is executed continuously. The

  When voice information is received from the microphone 12 (S13: YES), storage of voice data information obtained by converting the received voice information is started (S15), and then the voice that is the basis of the received voice information is It is selected whether the music is sung (S17). At the same time, it is specified which performance part in the music sound data information corresponds to the sound of the sound information being received. When the corresponding sound is specified, the elapsed time from the start of performance of the music sound data information to the timing at which the sound is played is stored in the counter of the RAM 48. Thereafter, the counter is updated over time. Next, a process for displaying the difference information corresponding to the selected music (difference pre-display process) is executed (S71).

The difference pre-display process will be described with reference to FIG. In the difference pre-display process, a predetermined time (for example, 5 s) is added to the elapsed time specified by the counter of the RAM 48 among the difference information (see FIG. 7) corresponding to the music selected in S17 (see FIG. 11). or phase違箇Tokorojo report that matches the time is stored is determined. Then, if the same phase違箇Tokorojo paper is determined to be stored, it is determined that the difference in the past at the positions has occurred (S81: YES). In this case, there is a possibility that the same different speech is uttered repeatedly. Therefore, the microphone identification information, the difference contents, and the difference amount information associated with the corresponding different portions are displayed and visually recognized by the conductor 3 (S83). As a result, a member who is highly likely to utter a different sound is specified before a predetermined time (5 s) before the timing at which the different sound is likely to be uttered, and the conductor 3 together with information on the content and amount of the difference Can be recognized.

  The display process described above is continuously executed until the predetermined time (5 s) has elapsed (S85: NO). If the predetermined time has elapsed (S85: YES), the display process is stopped (S87). Then, the difference pre-display process ends, and the process returns to the difference notification process (see FIG. 11).

  On the other hand, if the corresponding difference information is not stored in the difference information corresponding to the music selected in S17 (see FIG. 11) (S81: NO), the difference prior display process is performed without performing any special process. And return to the difference notification process (see FIG. 11).

  In the difference notification process, a difference specifying process (S19) is executed after the difference pre-display process. In the difference specifying process, the audio information immediately after reception is compared with the music information to determine whether there is a difference. If it is determined that there is a difference (S21: YES), the contents of the difference are displayed (S23), and the conductor 3 is visually recognized.

  The above-described processing is repeatedly executed until the audio information transmitted from the microphone 12 is completed (S29: NO). When reception of the audio information is completed (S29: YES), the generated difference information is stored as the difference information. The difference information in the area 493 is stored (S31), and the difference notification process is terminated.

  As described above, in the modified example, it is possible to notify the conductor 3 in advance of a place where the member of the choir 6 is likely to make a wrong voice. Can call attention to the members of the choir 6 in advance. Thereby, it becomes possible to improve the efficiency of choral practice more.

  Note that the CPU 61 that performs the process of S13 in FIG. 8 corresponds to the “sound acquisition unit” of the present invention, and the CPU 61 that performs the process of specifying the sound arrival direction from the microphone identification information that is the transmission source of the audio information received in S13. This corresponds to the “direction specifying means” of the present invention. The flash memory 49 having the voice-related storage area 491 in FIG. 5 corresponds to the “first storage unit” of the present invention, and the CPU 61 that performs the processing of S15 in FIG. 8 corresponds to the “first storage control unit” in the present invention. The CPU 61 that performs the process of S19 corresponds to the “sound matching determination unit” of the present invention, and the CPU 61 that performs the process of S23 corresponds to the “display control unit” of the present invention. The flash memory 49 provided with the music sound storage area 492 of FIG. 5 corresponds to the “second storage means” of the present invention.

  The CPU 61 that performs the process of extracting the amplitude component in order to determine the level shift in S41 of FIG. 9 or to determine the tempo shift in S49 corresponds to the “acquired sound amplitude extracting means” of the present invention. The CPU 61 that performs the process of determining whether or not a level shift has occurred or the CPU 61 that performs the process of determining whether or not a tempo shift has occurred in S49 corresponds to the “amplitude matching determination unit” of the present invention. The CPU 61 that performs the process of extracting the frequency component in order to determine the pitch deviation in FIG. 6 corresponds to the “acquired sound frequency extracting means” of the present invention, and the CPU 61 that performs the process of determining whether the pitch deviation occurs in S57. Corresponds to the “frequency match determination means” of the present invention, and the CPU 61 that performs the process of calculating the difference amount in S45, S53, and S61 corresponds to the “difference calculation means” of the present invention.

  The flash memory 49 provided with the difference information storage area 493 of FIG. 5 corresponds to the “fifth storage means” of the present invention, and the CPU 61 that performs the process of storing the difference information in S33 of FIG. It corresponds to “memory control means”. The flash memory 49 having an area for storing the difference amount in the difference information storage area 493 of FIG. 5 corresponds to the “third storage means” and the “fourth storage means” of the present invention, and in S33 of FIG. The CPU 61 that performs the process of storing the “corresponding” corresponds to “third storage control means” and “fourth storage control means” of the present invention. The CPU 61 that performs the process of updating the cumulative number in S33 of FIG. 8 corresponds to the “first cumulative unit” of the present invention.

It is a schematic diagram which shows the outline | summary of the choral practice system 1 in which the head mounted display 2 is used. FIG. 2 is a diagram showing an external configuration of a head mounted display 2 attached to a conductor 3. 3 is a schematic diagram showing an electrical configuration of a head mounted display 2. FIG. 6 is a schematic diagram illustrating a process in which image light 4 is formed on the display unit 40. FIG. 3 is a schematic diagram showing a storage area of a flash memory 49. FIG. It is a schematic diagram which shows the acquisition sound information memorize | stored in the audio | voice related storage area 491. FIG. It is a schematic diagram which shows the difference information memorize | stored in the difference information storage area | region 493. It is a flowchart of a difference notification process. It is a subroutine flowchart of a difference specific process. 3 is a schematic diagram showing an example of an image visually recognized by a conductor 3 via a head mounted display 2. FIG. It is a flowchart of the difference notification process in a modification. It is a subroutine flowchart of the difference prior display process in the modification.

DESCRIPTION OF SYMBOLS 1 Chorus practice system 2 Head mounted display 3 Conductor 6 Choir 12, 121, 122, 123, 124, 125 Microphone 40 Display part 43 Short-range radio | wireless communication part 46 Control part 61 CPU
49 Flash memory 57 Short-range wireless communication module 58 Short-range wireless communication control circuit 491 Audio-related storage area 492 Music sound storage area 493 Difference information storage area 494 Audio storage area 495 Video storage area

Claims (9)

Sound acquisition means for acquiring sound;
Direction specifying means for specifying the direction of arrival of the sound acquired by the sound acquisition means;
First storage control means for storing the sound acquired by the sound acquisition means and the arrival direction specified by the direction specifying means in association with each other in a first storage means;
Whether the acquired sound information, which is the sound information stored in the first storage means, and the music sound information, which is the information of the sound of the predetermined music and is stored in the second storage means, coincide with each other. Sound matching judgment means for judging whether or not
Display control means for displaying on the display means the direction of arrival corresponding to the acquired sound information determined to be different when the sound match determination means determines that both are different;
With
The sound matching judgment means
From the acquired sound information stored in the first storage unit, an acquired sound amplitude extracting unit that extracts an acquired sound amplitude that is an amplitude component;
Determining whether or not the acquired sound amplitude extracted by the acquired sound amplitude extracting means matches the music sound amplitude that is an amplitude component of the music sound information stored in the second storage means; A head-mounted display comprising: amplitude matching determination means for determining whether or not the acquired sound information matches the music sound information . Sound acquisition means for acquiring sound;
Direction specifying means for specifying the direction of arrival of the sound acquired by the sound acquisition means;
First storage control means for storing the sound acquired by the sound acquisition means and the arrival direction specified by the direction specifying means in association with each other in a first storage means;
Whether the acquired sound information, which is the sound information stored in the first storage means, and the music sound information, which is the information of the sound of the predetermined music and is stored in the second storage means, coincide with each other. Sound matching judgment means for judging whether or not
Display control means for displaying on the display means the direction of arrival corresponding to the acquired sound information determined to be different when the sound match determination means determines that both are different;
With
The sound matching judgment means
From the acquired sound information stored in the first storage unit, an acquired sound amplitude extracting unit that extracts an acquired sound amplitude that is an amplitude component;
It is determined whether or not the timing matches between the acquired sound amplitude extracted by the acquired sound amplitude extracting means and the music sound amplitude that is an amplitude component of the music sound information stored in the second storage means. Amplitude matching judgment means for judging whether or not the acquired sound information and the music sound information match.
Head mounted display with Sound acquisition means for acquiring sound;
Direction specifying means for specifying the direction of arrival of the sound acquired by the sound acquisition means;
First storage control means for storing the sound acquired by the sound acquisition means and the arrival direction specified by the direction specifying means in association with each other in a first storage means;
Whether the acquired sound information, which is the sound information stored in the first storage means, and the music sound information, which is the information of the sound of the predetermined music and is stored in the second storage means, coincide with each other. Sound matching judgment means for judging whether or not
Display control means for displaying on the display means the direction of arrival corresponding to the acquired sound information determined to be different when the sound match determination means determines that both are different;
With
The sound matching judgment means
From the acquired sound information stored in the first storage unit, an acquired sound frequency extracting unit that extracts an acquired sound frequency that is a frequency component;
Determining whether or not the acquired sound frequency extracted by the acquired sound frequency extracting means matches the music sound frequency that is a frequency component of the music sound information stored in the second storage means; Accordingly, f head-mounted display and a frequency matching determination means for determining whether or not said acquired sound information and the musical tone information is matched. A first difference calculating means for calculating a first difference amount which is a degree of difference when it is determined that the amplitude match determining means is different;
The display control means includes
Wherein when the first difference amount is calculated by the first difference calculation means, head according to the calculated the first difference amount to claim 1 or 2, characterized in that to be displayed on said display means Mount display. A second difference calculating means for calculating a second difference amount which is a degree of difference when it is determined that the frequency match determining means is different;
The display control means includes
4. The head mounted display according to claim 3 , wherein when the second difference amount is calculated by the second difference calculation unit, the calculated second difference amount is displayed on the display unit. 5. . When the first difference amount is calculated by the first difference calculating unit, the first difference amount is associated with the acquired sound information that is the basis of the acquired sound amplitude determined to be different. A third storage control means for storing the third storage means together with the direction;
The display control means includes
5. The head mounted display according to claim 4, wherein when the first difference amount is stored in the third storage means, the first difference amount is displayed on the display means. When the second difference amount is calculated by the second difference calculation means, the second difference amount is associated with the acquired sound information that is the basis of the acquired sound frequency that is determined to be different. A fourth storage control means for storing in the fourth storage means together with the direction;
The display control means includes
6. The head mounted display according to claim 5 , wherein when the second difference amount is stored in the fourth storage means, the second difference amount is displayed on the display means. When the sound coincidence determining means determines that the acquired sound information and the music sound information are different, the different location information which is information for specifying the performance location of the predetermined music corresponding to the different location A fifth storage control means for storing in the fifth storage means together with the direction of arrival associated with the acquired sound information determined to be different,
The display control means includes
In the case where the different location information is stored in the fifth storage means, before the person who plays the predetermined music plays the performance location of the predetermined music specified by the different location information. , head-mounted display according to the DOA associated with the said different part information to any one of claims 1 to 7, characterized in that to be displayed on the display means. A first summation is made for each direction of arrival corresponding to the acquired sound information determined to be different, the number of differences being the number of times that the acquired sound information and the music sound information are determined to be different by the sound match determining means. With a means of accumulation,
The display control means includes
Wherein when the difference count is cumulative at first cumulative means, head according to any one of claims 1 to 8, characterized in that to be displayed on said display means the difference count for each said direction of arrival Mount display.

Images