JP6498090B2 - Karaoke device and karaoke program - Google Patents

Description

  The present invention relates to a karaoke apparatus that enjoys singing according to accompaniment, and a karaoke program that implements a karaoke function by being mounted on an information processing apparatus such as a karaoke apparatus or a game apparatus.

  Conventionally, in a banquet hall or the like, karaoke is performed to sing along with accompaniment. In a karaoke apparatus for performing karaoke, a function of assisting singing by displaying lyrics in synchronization with a performance is well known. The user can check the lyrics to be sung by visually recognizing the lyrics displayed on the monitor screen even if the lyrics are not stored. Various improvements have been made to the lyrics display function of such a karaoke apparatus.

  Patent Document 1 discloses a karaoke apparatus capable of providing lyrics that are easy to visually recognize in an environment where a user can enjoy singing using a display unit having a large screen. In this karaoke apparatus, the magnitude | size of the image information displayed in the display area of a display part is changed based on the position of the detected singer. By including the lyric information in the image information, it is possible to display the lyric at a position in the display area suitable for the position of the singer.

  Patent Document 2 discloses a karaoke system that uses a head-mounted display that is worn on a user's head. By displaying the lyrics on the head mounted display, the degree of freedom of the user's position is increased, and the user can enjoy karaoke at a desired position.

Japanese Patent Laying-Open No. 2015-69042 JP 2001-22362 A

  In recent years, with the advancement of image processing technology, an environment in which a virtual space using an HMD (head-mounted display) can be used even in a general home has been prepared. This is because of an improvement in arithmetic processing capability of a CPU, GPU or the like mounted on the information processing apparatus, an increase in storage capacity, and the like. In the virtual space, the video can be displayed according to the movement of the head of the user wearing the HMD, so that the user can enjoy the feeling of being in the virtual space. Collecting. In Patent Document 2, although HMD is used, it is limited to displaying lyrics by HMD.

  By the way, in the virtual space, it is possible to enjoy a virtual feeling as if the user wearing the HMD is on the spot. The applicant has newly developed a karaoke apparatus that enjoys singing using a virtual space. During development, the issue was how to display lyrics in the virtual space. That is, when a predetermined location in the virtual space is used as a display location for lyrics as in the conventional karaoke apparatus, unrealistic lyrics are displayed in the virtual space, and a sense of incongruity in the virtual space occurs. In the virtual space, it is important for the user to enter the projected world (immersive feeling), and if the user feels uncomfortable with the virtual space, the immersive feeling is impaired. The present invention is premised on such a problem, and it is an object of the present invention to enable singing while fully enjoying the virtual space by displaying lyrics in the virtual space without impairing the user's immersive feeling. It is said.

Therefore, the karaoke apparatus according to the present invention adopts the following configuration.
Performance processing for performing based on performance information,
Detection processing for detecting the arrangement information including the relative direction of the operation device with respect to the headset, which is information related to the arrangement of the operation device ;
In the background image displayed on the display unit arranged in the headset, a lyrics display object is arranged corresponding to the arrangement information detected by the detection process, and the lyrics of the music played in the performance process are displayed in the lyrics display object. Display processing to be executed.

Furthermore, in the karaoke apparatus according to the present invention,
The arrangement information detected by the detection process includes the distance between the headset and the operation device,
The display process is characterized in that the lyrics display object is arranged in a size corresponding to the distance in the arrangement information detected by the detection process.

Furthermore, in the karaoke apparatus according to the present invention,
The arrangement information detected by the detection process includes the inclination of the operating device with respect to the headset,
The display process is characterized in that the lyrics display object is arranged at an inclination corresponding to the inclination in the arrangement information detected by the detection process.

Furthermore, in the karaoke apparatus according to the present invention,
The display process regulates the inclination of the lyrics display object when the inclination in the arrangement information detected by the detection process exceeds a predetermined value.

Furthermore, in the karaoke apparatus according to the present invention,
The display process
Follow-up mode that arranges the lyrics display object corresponding to the placement information detected by the detection process,
A fixed mode in which a lyrics display object is arranged at a predetermined position on a coordinate system of a background image displayed on the display unit can be selected.

Furthermore, in the karaoke apparatus according to the present invention,
The display process is characterized in that an operation object indicating an operation device is arranged and displayed corresponding to the arrangement information detected by the detection process.

Furthermore, the karaoke apparatus according to the present invention is
When the arrangement information detected by the detection process is out of a predetermined range, the lyrics display object is not arranged in the background video.

Furthermore, in the karaoke apparatus according to the present invention,
The display process is characterized in that the transparent state of the lyrics display object can be changed, and the transparent state of the lyrics display object is changed according to the progress of the performance information.

The karaoke program according to the present invention is
Performance processing for performing based on performance information,
Detection processing for detecting the arrangement information including the relative direction of the operation device with respect to the headset, which is information related to the arrangement of the operation device ;
In the background image displayed on the display unit arranged in the headset, a lyrics display object is arranged corresponding to the arrangement information detected by the detection process, and the lyrics of the music played in the performance process are displayed in the lyrics display object. And display processing to be executed by an information processing apparatus.

  The karaoke apparatus or karaoke program according to the present invention provides a new form of lyric display for assisting singing in karaoke in which singing is performed in accordance with accompaniment. Specifically, the arrangement of the operation device in the real space is detected, and the lyrics display object is arranged and displayed in correspondence with the detected arrangement, and in particular, a mode for allowing the user to experience the virtual space Is effective.

The figure which shows the game system used by this embodiment The block diagram which shows the game device used by this embodiment The perspective view which shows HMD (head mounted display) used by this embodiment. Front view and side view showing a controller used in the present embodiment Flow chart showing the music playback process of this embodiment The figure which shows the data structure of the music information used by this embodiment Table showing correspondence between operation unit, function, and initial setting of controller of this embodiment Schematic diagram for explaining visual field image formation in the present embodiment Diagram for explaining the arrangement of the controller and the display relationship between the lyrics display object and the controller object Lyric display object, front view of controller object, side view Actual field of view (follow-up, non-transparent, controller object display) Actual field of view (follow-up, non-transparent, controller object hidden) Actual field of view (follow-up, transmission, controller object display) Actual field of view (fixed, non-transparent, controller object display) Actual field of view (follow-up, non-transparent, controller object display) The figure for demonstrating a 2nd modification The figure for demonstrating a 5th modification

  The present invention will be described taking an example of using a game system. FIG. 1 is a diagram showing a game system used in the present embodiment. The game system includes a game device 1, an HMD 3, and a controller 4. The game apparatus 1 can provide visual information to the user by displaying a game screen on the monitor 22 in a normal usage pattern. In the present embodiment, visual information (field-of-view video) is provided to the user by displaying an image on the HMD 3 (head mounted display). The game apparatus 1 can realize different functions by changing the program. In this embodiment, it is possible to perform karaoke using the virtual space by HMD3 by starting the karaoke program. Note that the game device 1 that has activated the karaoke program corresponds to the karaoke device of the present invention. The user listens to the performance sound reproduced by the game apparatus 1 through the headphones 32 and sings using the microphone 33. The singing voice input to the microphone 33 is mixed with the performance sound and emitted from the headphones 32.

  Further, the user can instruct the game apparatus 1 with various commands using the controller 4 (corresponding to the “operation device” of the present invention). The game apparatus 1 and the controller 4 are wirelessly connected, and the user can perform an operation without being bothered by a cable. In the present embodiment, the video displayed on the HMD 3 is changed according to the movement of the user's head so that the user can experience the virtual space. The movement of the user's head is detected by capturing an image with the camera 21 and detecting the arrangement of the HMD 3 in the real space. In the present embodiment, the arrangement of the controller 4 in the real space is also detected and used in the virtual space.

  FIG. 2 is a block diagram showing the game apparatus 1 used in the present embodiment. The game apparatus 1 includes a CPU 10 for controlling each component in an integrated manner, and a memory 11 for temporarily storing information necessary for executing various programs. The CPU 10 and the memory 11 constitute a control unit in the game apparatus 1. In addition, the game apparatus 1 of the present embodiment includes an acoustic control unit 15 that inputs and outputs various sounds. The sound control unit 15 performs a performance based on the performance information included in the music information when the karaoke program is executed. In addition, a microphone 33 is connected to the acoustic control unit 15, and the input singing voice is mixed with the played performance sound and is acoustically output from the headphones 32 </ b> R and 32 </ b> L.

  In addition, the game apparatus 1 includes video reproduction means for displaying lyrics video and background video on the monitor 22. This video playback means includes a video playback unit 13 that plays back video based on video information, a video RAM 12 that temporarily stores video to be played back, displays lyrics on the played back video, or provides video effects. A video control unit 14 is provided. The video controller 14 displays and outputs the reproduced video on the monitor 22 or the right-eye display 31R and the left-eye display 31L of the HMD 3. Different images can be displayed and output on the monitor 22 and the HMD 3, respectively. Further, the right-eye display 31 </ b> R and the left-eye display 31 </ b> L of the HMD 3 can be stereoscopically displayed to the user by displaying and displaying an image having parallax.

  Further, the video control unit 14 is connected to the camera 21 and can capture a video shot by the camera 21. In particular, in the present embodiment, the arrangement of the HMD 3 and the arrangement of the controller 4 in the real space are detected from the video imaged by the camera 21 as described with reference to FIG. In order to improve the accuracy of the arrangement detection, the HMD 3 is provided with LEDs 35 and 36, and the controller 4 is provided with an LED 47. By recognizing the LEDs 35, 36, and 47 from the captured video, it is possible to accurately detect the arrangement of the HMD 3 and the controller 4 in the real space. Further, the lighting of the LEDs 35, 36, and 47 makes it possible to detect the arrangement even in a dark environment.

  The game apparatus 1 also includes a hard disk 19 as a storage unit that stores various programs and various information used in the programs. Further, it also has a medium reproducing unit 20 for reproducing a disk medium, and can execute a program stored in the disk medium. The game apparatus 1 includes a LAN communication unit 18 as communication means connected to the LAN 40. The LAN 40 is connected to a router 41 in the home and can communicate with the Internet. In the example of FIG. 2, the game apparatus 1 has a connection form for communicating with the server apparatus 5 that manages various information via the LAN communication unit 18 and the router 41. In addition, the game apparatus 1 includes a first wireless communication unit 16 that can implement a wireless LAN, and can be connected to the Internet wirelessly. Furthermore, the game apparatus 1 includes a second wireless communication unit 17 of Bluetooth (registered trademark) standard as short-range wireless communication. The game apparatus 1 performs wireless communication with the controller 4 using the second wireless communication unit 17.

  FIG. 3 is a perspective view showing an HMD 3 (head mounted display) used in the present embodiment. The figure on the left side is a perspective view of the user wearing the HMD3 as seen from the rear, and the figure on the right side is a perspective view of the user wearing the HMD3 as seen from the front. The HMD 3 includes a right-eye display 31R, a left-eye display 31L (corresponding to the “display unit” of the present invention), an HMD housing 37, and a headband 34 that is attached to the user's head. Yes. In the present embodiment, the HMD casing 37 and the headband 34 constitute a headset. Four LEDs 35a to 35d are provided on the front surface of the HMD casing 37 for detecting the arrangement. The headband 34 is also provided with four LEDs 36a to 36d for detecting the arrangement. By providing the LED for detecting the arrangement around the HMD 3 in this manner, the arrangement of the head in the real space (including the direction in which the head is facing) is included regardless of the direction the user faces. ) Can be detected.

  Headphones 32 </ b> R and 32 </ b> L are attached to the headband 34 so that the user can listen to the sound. In addition, a microphone 33 for capturing the user's singing voice via an arm is attached to the left headphone 32L. With such a configuration, the user can listen to the performance sound or the like with the headphones 32R and 32L regardless of the direction, and can sing without holding the microphone 33 by hand. ing.

  FIG. 4 is a front view and a side view showing a controller used in the present embodiment. The controller 4 of this embodiment is wirelessly connected to the game apparatus 1 and can give various instructions to the game apparatus 1. 4A is a front view of the controller 4, FIG. 4B is a side view of the controller 4 of FIG. 4A viewed from the right side, and FIG. 4C is FIG. It is the side view which looked at the controller 4 of A) from the upper side. The controller 4 is formed by having two grips 41R and 41L that a user holds with both hands and a connecting portion 42 provided therebetween.

  The left grip 41L is provided with a cross key 44 for inputting up, down, left and right directions in the game operation. A button group 45 is provided on the right grip 41R. In this example, the button group 45 includes four buttons with “A” to “D”. The connection unit 42 is provided with two analog sticks 43R and 43L. The user can use the analog sticks 43R and 43L to indicate multi-step directions. Above the controller 4, a right first button 46R1, a right second button 46R2, a left first button 46L1, and a left second button 46L2 are provided. These buttons are provided at positions that are easy to operate with the index finger when the user grasps the grips 41R and 41L.

  Further, as shown in FIG. 4C, the connection portion 42 is provided with an LED 47 for detecting the arrangement. As shown in FIG. 1, the LED 47 is provided at a position where it is easy to photograph with the camera 21 installed in the vicinity of the monitor 22 when the user faces the monitor 22. The game apparatus 1 detects the arrangement (including the direction of the controller 4) of the controller 4 in the real space by identifying the shape and color of the LED 47 from the video imaged by the camera 21.

  As described above, it is possible to sing in the virtual space using the configuration of the game system described above. Therefore, the game apparatus 1 starts a karaoke program. The karaoke program can be downloaded from the server device 5 and stored in the hard disk 19 or can be stored in a disk medium. The music playback process is started by starting the karaoke program in the game apparatus 1. Note that the karaoke program is based on the premise that the user uses the HMD 3 while the video output by the karaoke program is output to the HMD 3 and the sound is output to the headphones 32.

  FIG. 5 is a flowchart showing the music reproduction process of the present embodiment. First, a song to be sung is selected using the user interface displayed on the HMD 3 (S101). In the present embodiment, after music is selected, the corresponding music information is received from the server device 51, but may be stored on the game device 1 side.

  FIG. 6 is a diagram showing a data structure of music information used in the karaoke program. The music information is configured to include meta information given for performing music selection and the like and actual information for performing performance and displaying lyrics. The meta information includes a song ID for managing the song, a song name, a singer name, a songwriter name, a composer name, section identification information, and the like. The section identification information is information indicating a section type corresponding to the progression of music, such as a prelude, A melody, B melody, and interlude. By using this section identification information, it is possible to determine the section type currently being played during music playback.

  The actual information includes performance information, lyrics information, reference pitch information, and background video information. When the music is reproduced, the performance information as the singing accompaniment can be acoustically output from the headphones 32R and 32L by causing the acoustic control unit 15 to perform the performance information. In addition, when the music is reproduced, it is possible to assist the user in singing by reproducing the lyric information by the video reproducing unit 13 and outputting the video to the HMD 3.

  The background video information is information used as a background video displayed on the HMD 3. The background video information of the present embodiment is information obtained by capturing an actual landscape, a concert video, or the like. In this embodiment, since a virtual space is formed using a background video, background video information is formed by shooting a plurality of directions with a plurality of cameras from a certain viewpoint and connecting the videos. By displaying the video in the background video information according to the arrangement of the HMD 3 detected by the camera 21, the user can experience the virtual space. In the present embodiment, it is possible to experience a virtual space with a stereoscopic effect by displaying an image having parallax on the right-eye display 31R and the left-eye display 31L. The background video information of this embodiment is configured to be included in the music information, and the music and the background video have a one-to-one relationship. The background video information is not limited to such a form, and may be a form independent of the music information. In this case, it is conceivable to use background video information corresponding to the genre of the music information or use background video information selected by the user.

  After the music is selected, reproduction is started based on the selected music information (S102). The reproduction of the music information is a process of causing the sound control unit 15 to perform the performance information included in the music information and causing the video reproduction unit 13 to reproduce the lyrics information and the background video information. During music reproduction, the user can perform operations using the controller 4. FIG. 7 is a table showing the correspondence between the operation unit, function, and initial setting of the controller 4 of the present embodiment. The first left button 46L1 is assigned to the operation unit for switching between transparent / non-transparent of the lyrics display object 61. Each time the left first button 46L1 is pressed, the transparent / non-transparent of the lyrics display object 61 is switched alternately. The second left button 46L2 is an operation unit for switching between a fixed mode for fixing the lyrics display object 61 at a predetermined position in the virtual space and a follow-up mode for following the arrangement of the controller 4 in the real space. Each time the left second button 46L2 is pressed, the tracking mode is switched to the tracking mode, or the tracking mode is switched to the fixing mode. The first right button 46R1 is assigned to an operation unit for switching display / non-display of the controller object 62 (corresponding to the “operation object” of the present invention). Every time the right first button 46R1 is pressed, the display and non-display of the controller object 62 are alternately switched.

  At the start of music reproduction, the initial settings are applied to these operation units. Note that the user may be able to change the setting of the operation unit and the initial setting for each function of the controller 4. When the reproduction of the music is started (S102), the initial setting shown in FIG. 7 is read and an image (view image 60) displayed on the HMD 3 is formed. The visual field image 60 is formed using background video information corresponding to the music. FIG. 8 is a schematic diagram for explaining the formation of the visual field image 60 in the present embodiment. As described with reference to FIG. 6, the background video information is a video formed by photographing a plurality of directions with a plurality of cameras from a certain viewpoint and connecting the videos. FIG. 8 schematically shows background video information around the user wearing the HMD 3. Here, the background video information is displayed only on the user's side for easy understanding on the drawing, but in reality, the video image surrounds the user in a hemispherical shape or a full spherical shape. The video displayed on the HMD 3 is determined based on the arrangement of the HMD 3 detected by the camera 21. That is, an area in the background video information corresponding to the arrangement of the HMD 3, specifically, the direction in which the HMD 3 faces is cut out as the view video 60. Therefore, the visual field image 60 changes depending on the movement of the head of the user wearing the HMD 3, and the user can experience a virtual space formed by the background video information.

  Further, in the present embodiment, the lyrics display object 61 and the controller object 62 can be displayed on the visual field image 60 based on the arrangement of the controller 4 detected by the camera 21. The arrangement detection of the controller 4 in the present embodiment is performed with respect to the coordinate system XYZ based on the background video information shown in FIG. 8, the positions in the X direction, the Y direction, the Z direction, the X axis, the Y axis, and the Z axis. The amount of rotation around is detected as arrangement information of the controller 4. In the present embodiment, similarly for the HMD 3, the position in the X direction, the Y direction, the Z direction, and the amount of rotation about the X axis, the Y axis, and the Z axis with respect to the coordinate system XYZ based on the background video information. Is detected as the arrangement information of the HMD3. Based on the arrangement information of the HMD 3 and the arrangement information of the controller 4, it is possible to detect the relative arrangement of the controller 4 with respect to the HMD 3. As described above, the arrangement information of the controller 4 and the arrangement information of the HMD 3 in this embodiment are the positions in the real space (positions in the X direction, Y direction, and Z direction) and directions (around the X axis, Y axis, and Z axis). A plurality of items such as (rotation amount) are included, but as the arrangement information, items that are not detected or are not used may be provided.

  The detected relative arrangement between the two is used to display the controller object 62 displayed in the visual field image 60. When the controller object 62 is displayed in the visual field image 60, the user can observe the controller 4 held in his / her hand in the virtual space. Furthermore, in this embodiment, the lyrics display object 61 is arranged corresponding to the controller object 62. The lyric display object 61 is an object for displaying the reproduced lyric during the music reproduction, and is arranged corresponding to the controller object 62, and therefore moves in accordance with the movement of the controller 4.

  FIG. 9 is a diagram for explaining the arrangement of the controller 4 and the display relationship between the lyrics display object 61 and the controller object 62. On the HMD 3, background video information cut out based on the arrangement of the HMD 3 detected by the camera 21 is displayed as the visual field video 60. Actually, the visual field image 60 having parallax is displayed on the right-eye display 31R and the left-eye display 31L of the HMD 3 so that the user can experience a virtual space based on a stereoscopic image. FIG. 9 is a display example when the controller 4 is arranged in front of the user wearing the HMD 3, and the controller object 62 and the lyrics display object 61 are displayed in the view image 60 using the arrangement of the controller 4 detected in the real space. Is displayed.

  FIG. 10 is a front view and a side view of the lyrics display object 61 and the controller object 62. The lyrics display object 61 and the controller object 62 are formed as three-dimensional objects by computer graphics. Moreover, both are arrange | positioned by the predetermined positional relationship. The left side of FIG. 10 is a front view of the lyrics display object 61 and the controller object 62, and the right side of FIG. 10 is a front view of the lyrics display object 61 and the controller object 62. The lyric display object 61 is a rectangular plate-like object, and lyric characters are displayed on the front. On the other hand, the controller object 62 is a plate-like object imitating the controller 4. In the present embodiment, the lyrics display object 61 is arranged above the controller object 62 with a predetermined angle when viewed from the side.

  After the reproduction of the music starts, the dynamic visual field image 60 is formed by repeatedly executing S111 to S116 (follow-up mode) and S121 to S124 (fixed mode) in the frame period. In this embodiment, since the follow-up mode is set as an initial state, the process on the follow-up mode side (S111 to S116) is executed as a result of the mode determination (S110). First, the arrangement of the HMD 3 is detected by the camera 21 (S111), and an area to be cut out as the visual field image 60 in the background video information is determined using the direction in which the HMD 3 faces (S112). Next, the arrangement of the controller 4 is detected by the camera 21 (S113), and the arrangement of the controller 4 relative to the HMD 3 is calculated (S114). Based on the calculated relative arrangement, the arrangement of the controller object 62 in the visual field image 60 is determined (S115). In the display process (S116), the controller object 62 is displayed in the visual field image 60 determined in S112, and the lyrics display object 61 is displayed at a position corresponding to the controller object 62. On the lyric display object 61, lyric information is displayed in accordance with the progress of music reproduction. The display of the lyric information enables the lyric to be sung by confirming the color of the displayed lyric in the same manner as in ordinary karaoke.

  When the lyrics display object 61 is switched to transparent by the operation of the first left button 46L1, in the display process (S116), the lyrics display object 61 is displayed in a transparent state, and behind the lyrics display object 61. The visual field image 60 is displayed so that it can be easily seen. When the lyrics display object 61 is not transparent, the lyrics characters displayed on the lyrics display object 61 are easy to read. On the other hand, when the lyrics display object 61 is transparent, the lyrics characters are difficult to read, but the background image is easily visible. In the present embodiment, the lyrics display object 61 can be switched between transparent and non-transparent by the user's operation, so that it is possible to freely switch between giving priority to reading the lyrics or giving priority to the visibility of the background video. It is possible.

  When the controller object 62 is switched to non-display by the operation of the first right button 46R1, the controller object 62 is not displayed in the display process (S116). By displaying the controller object 62, the controller object 62 that is similar to the controller 4 held by the user and that follows the movement of the controller 4 can be visually recognized in the virtual space, thereby improving the sense of immersion in the virtual space. It is possible. However, the user who wants to enjoy the background video may feel annoyed because the controller object 62 blocks the field of view. For this reason, in the present embodiment, it is possible to switch between display and non-display of the controller object 62 by a user operation.

  FIGS. 11 to 13 show an actual visual field image 60 in the follow-up mode. In each figure, a broken line and a symbol surrounding the lyrics display object 61 and the controller object 62 are added to facilitate understanding. The same applies to FIGS. 14 and 15 described later.

  FIG. 11 shows a visual field image 60 when the lyrics display object 61 is set to be non-transparent and the controller object 62 is set to display. The user can view the controller object 62 corresponding to the controller 4 to be grasped in the view image 60 and sing while confirming the lyrics displayed on the lyrics display object 61 positioned above the controller object 62. It is. Since the lyrics display object 61 is non-transparent, the lyrics characters displayed on the lyrics display object 61 are easy to read.

  FIG. 12 is a visual field image 60 when the lyrics display object 61 is set to be non-transparent and the controller object 62 is set to non-display. In this case, the controller object 62 is not displayed in the visual field image 60, but the lyrics display object 61 also moves following the movement of the controller 4 held by the user. Since the visual field image 60 is not obstructed by the controller object 62, the user can enjoy the virtual space by the visual field image 60.

  FIG. 13 is a visual field image 60 when the lyrics display object 61 is set to be transparent and the controller object 62 is set to display. In this setting, since the lyrics display object 61 is in a transparent state, the user can visually recognize an image located behind the lyrics display object 61. In addition to the settings described with reference to FIGS. 11 to 13, the lyrics display object 61 can be set to be transparent, and the controller object 62 can be set to be non-displayed. In the present embodiment, the periphery of the lyric character displayed on the lyric display object 61 is non-transparent, but may be made transparent up to the last position of the lyric character.

  In the follow-up mode described above, the lyrics display object 61 (which may be accompanied by the controller object 62) can be arranged at a desired position in the visual field image 60 by the user moving the controller 4 in the real space. It is. Therefore, when there is a part that the user wants to pay attention to in the visual field image 60, the lyrics display object 61 can be arranged so as to avoid the part. Furthermore, the lyrics display object 61 can be removed from the visual field image 60 by moving the controller 4, and it is possible to cope with the case where lyrics display is not required or only the background image is desired. is there. Further, since the lyrics display object 61 is a plate-like object, the range occupied by the lyrics display object 61 can be reduced by a slight operation of tilting the controller 4. Further, since the lyrics display object 61 is displayed according to the relative arrangement of the controller 4 and the HMD 3, the size of the lyrics display object 61 can be changed by changing the distance between the controller 4 and the HMD 3. It is. As in a normal space, when the user wants to see a larger image, the lyrics display object 61 is enlarged and displayed by bringing the controller 4 closer to the face.

  On the other hand, as a result of the mode determination (S110), when the fixed mode is set, the processing of S121 to S124 is executed. The fixed mode of the present embodiment is a mode in which the lyrics display object 61 is fixed and displayed at a predetermined position on the coordinate system of the background video information shown in FIG. Accordingly, the lyrics display object 61 is displayed at a predetermined position in the virtual space formed by the background video information regardless of the position of the controller 4. In this case, first, by detecting the arrangement of the HMD 3 with the camera 21 (S121), an area to be cut out as the visual field image 60 in the background video information is determined (S122). Next, the arrangement of the controller 4 is detected by the camera 21 (S123). Then, a display process (S123) is performed in which the controller object 62 is displayed in the visual field image 60 determined in S122 and the lyrics display object 61 is arranged at a predetermined position in the virtual space formed by the background image information. In the fixed mode, similarly to the follow-up mode, the display process (S123) is executed in accordance with the setting of transparent / non-transparent of the lyrics display object 61 and display / non-display of the controller object 62.

  FIG. 14 is a diagram showing an actual visual field image 60 in the fixed mode. In this example, the lyrics display object 61 is arranged at a predetermined position (on the lawn) in the background video information. The lyrics display object 61 is arranged at a predetermined position in a predetermined virtual space regardless of the arrangement of the controller 4 in the real space. In this example, the lyrics display object 61 is displayed in a non-transparent manner. The user can visually recognize the lyric display object 61 by looking at the lyric display object 61 in the background video information (turning the HMD 3). In the fixed mode, the user can observe the lyrics display object 61 like a signboard arranged in a virtual space. In the example of FIG. 14, the controller object 62 is set to be displayed, and the controller object 62 is displayed following the arrangement of the controller 4 in the real space.

  In the fixed mode described above, the lyrics display object 61 is displayed at a predetermined position in the virtual space formed by the background video information. The user can check the lyrics by displaying the lyrics display object 61 in the visual field image 60 by pointing the HMD 3 at the position of the lyrics display object 61. In the present embodiment, the lyric display object 61 is positioned at a predetermined position in the virtual space (a predetermined position in the coordinate system of FIG. 8), but may be changed to an arbitrary position. For example, it is conceivable to fix the position of the lyrics display object 61 when the follow mode is changed to the fixed mode. The user uses the follow mode to move the lyrics display object 61 to a position where it is desired to be fixed, and the lyrics display object 61 is fixed by switching from the follow mode to the fixed mode, thereby displaying the lyrics at a desired position in the virtual space. The object 61 can be fixedly displayed. In the fixed mode of the present embodiment, the lyric display object 61 is displayed at the same location, but the position of the lyric display object 61 may be changed according to the progress of the music.

  The field-of-view image 60 displayed on the HMD 3 is formed by executing S111 to S116 (in the follow-up mode) or S121 to S124 (in the fixed mode) for each frame, and is virtual for the user wearing the HMD3. It is possible to experience the space. At that time, the lyrics display object 61 allows the user to observe the lyrics to be sung. When it is determined that the performance information has been finished for the music (S117: Yes), the process returns to the beginning of the music reproduction process, and the user selects the music to be reproduced next.

  As mentioned above, although one Embodiment of this invention was described, this invention is not limited only to this Embodiment, It is possible to employ | adopt various modifications. Various modifications will be described below.

(First modification)
In the embodiment described above, a form (in the follow-up mode) in which the lyrics display object 61 is arranged in the virtual space following the arrangement of the controller 4 in the real space is adopted. In this form, when the controller 4 has a predetermined inclination or more with respect to the HMD 3, the lyrics of the lyrics display object 61 are difficult to read. FIG. 15 is a visual field image 60 when the controller 4 is tilted with respect to the HMD 3. As can be seen from the visual field image 60, when the controller 4 is tilted, the lyrics display object 61 is also tilted, and the lyrics characters displayed on the lyrics display object 61 are difficult to read. When tilted to the side, the lyrics display object 61 is observed from the side, and the lyrics characters cannot be read. For this reason, in the first modification, when the controller 4 has a predetermined inclination or more with respect to the HMD 3, the inclination of the lyrics display object 61 is restricted so as not to follow the controller 4. With such a configuration, even when the controller 4 is tilted more than necessary, it is possible to allow the user to check the lyrics toward the side where the lyrics display object 61 can display the lyrics characters. When the controller 4 becomes smaller than the predetermined inclination, the restriction is released, and the inclination of the lyrics display object 61 follows the controller 4.

(Second modification)
In the embodiment described above, the transparent / non-transparent of the lyrics display object 61 is switched by the user's operation. However, the transparent / non-transparent of the lyrics display object 61 may be automatically switched. For example, by automatically switching the lyric display object 61 to a transparent state in a section where singing such as prelude, interlude, and posterior is not performed, the user can enjoy a visual field image that is not obstructed by the lyric display object 61 during the section. Become. Such a section can be determined by the section identification information in the music information described with reference to FIG. Alternatively, the period in which the user is not singing may be determined based on the presence or absence of the singing voice input to the microphone 33, and the lyrics display object 61 may be switched to the transparent state.

  In addition, when the controller 4 is switched from the fixed mode to the follow mode, if the controller 4 is located near the HMD 3, a large lyrics display object 61 may be displayed in front of the user and surprise the user. Therefore, the transparent / non-transparent of the lyrics display object 61 may be switched according to the distance between the controller 4 and the HMD 3. For example, when the controller 4 is located within a predetermined distance from the HMD 3, the lyric display object 61 is made transparent so that troubles such as when the large lyric display object 61 is displayed in front of the eyes are suppressed. It becomes possible.

  Alternatively, in the background video information, when the lyrics display object 61 is located in a predetermined area, the lyrics display object 61 may be switched to the transparent state. FIG. 16 is a diagram for explaining the second modified example, for example, assuming a virtual space in which singing is enjoyed together with a singer. In the visual field image 60, singer who enjoys singing together is displayed. FIG. 16A shows a case where the lyric display object 61 is positioned in a region not superimposed on the singer, and FIG. 16B shows a case where the lyric display object 61 is positioned in a region superimposed on the singer. In the state of FIG. 16A, the lyric display object 61 is set in a non-transparent state, and the displayed lyric characters are easily read. On the other hand, when the lyrics display object 61 is superimposed on the singer, the lyrics display object is switched to the transparent state so that the singer displayed behind the lyrics display object 61 is easily visible. It should be noted that it is preferable to change and move the predetermined area through which the lyrics display object 61 is transparent according to the progress of the reproduction of the background video information.

(Third Modification)
In the above-described embodiment, the background video information is a form using an actual video shot by a camera, but the background video may be formed as a three-dimensional object by computer graphics. In such a form, it is also possible to move freely in the formed virtual space. While the above-described embodiment is a fixed viewpoint, in the third modified example, it is possible to experience the virtual space from a free viewpoint. The viewpoint is moved by, for example, assigning the left analog stick 43L of the controller 4 for movement in the virtual space and moving the virtual space in the direction in which the left analog stick 43L is tilted, or a predetermined path as reproduction progresses. It is possible to move the user's viewpoint. When the second modification described in the previous paragraph is applied to the third modification, the target for setting the lyrics display object 61 in the transparent state is not a predetermined area but a predetermined three-dimensional object. For example, when the lyric display object 61 is superimposed on the video of a singer formed of a three-dimensional object, the lyric display object 61 is switched to a transparent state so that the three-dimensional object to be observed is easily visible. Is possible.

(Fourth modification)
In the above embodiment, the lyrics display object 61 and the controller object 62 are displayed using the relative direction of the controller 4 with respect to the HMD 3, the distance between the HMD 3 and the controller 4, and the inclination of the controller 4 with respect to the HMD 3. Only the relative direction of the controller 4 to the HMD 3 may be used to display the lyrics display object 61 and the controller object 62. In the visual field image 60, it is possible to display the lyrics display object 61 and the controller object 62 at a position corresponding to the relative direction of the controller 4 with respect to the HMD 3. Furthermore, the size can be changed by adding a distance, and the appearance can be changed by adding a tilt, thereby improving the virtual reality.

(5th modification)
In the above-described embodiment, the form using the HMD 3 has been described. However, the present invention is not necessarily limited to the form using the HMD 3. For example, the present invention can be applied to karaoke in which background video and lyrics are displayed on a monitor as in the past. FIG. 17 is a diagram for explaining the fifth modification example, in which the singing is performed while watching the screen displayed on the monitor 22 (corresponding to the “display unit” of the present invention). The plane parallel to the display surface of the monitor 22 is the XY plane, and the depth direction is the Z axis. Also in such a form, the arrangement of the controller 4 is detected, and the display position of the lyrics display object 61 displayed on the monitor 22 is changed according to the arrangement of the controller 4. As the arrangement of the controller 4 to be detected, the lyrics display object 61 can be freely moved in the background image displayed on the monitor 22 by using at least a position on a plane (XY plane) parallel to the monitor plane. Is possible. In addition to the position of the controller 4, the position of the controller 4 in the Z-axis direction or the rotation amount of the controller 4 around the X axis, the Y axis, or the Z axis is detected and displayed on the monitor 22. It may be reflected in the display form of the lyrics display object 61 to be displayed.

(Sixth Modification)
In the above-described embodiment, the arrangement detection of the HMD 3 and the controller 4 is a form using the video taken by the camera 21, but the arrangement detection of the HMD 3 and the controller 4 is not limited to such a form. Alternatively, a form of detection using various centers such as a gyro may be employed. In the above-described embodiment, the controller 4 for the game device 1 is used as the operation device. However, the operation device is not limited to the controller 4, and various kinds of operations are performed on the game device 1 (information processing device). It is possible to employ a device that can issue a command. For example, a glove-like device or the like worn by the user on the hand may be used as the operating device.

  As described above, the present embodiment has been described by taking the game system as an example, but the present invention is not limited to the game system, and can be applied to various information processing apparatuses such as a conventional karaoke apparatus and a personal computer. Further, a karaoke program that is executed by the game apparatus 1 and various information processing apparatuses of the present embodiment and that can realize the functions of the present invention also belongs to the category of the present invention.

1: Game device 34: Headband 3: HMD 35, 36: LED
4: Controller (operating device) 37: HMD chassis 5: Server device 41: Router 10: CPU 41L: Left grip 11: Memory 41R: Right grip 12: Video RAM 42: Connection unit 13: Video playback unit 43L: Left analog Stick 14: Video control unit 43R: Analog stick 15: Sound control unit 44: Cross key 16: First wireless communication unit 45: Button group 17: Second wireless communication unit 46L1: Left first button 18: LAN communication unit 46L2: Left second button 19: Hard disk 46R1: Right first button 20: Medium playback unit 46R2: Right second button 21: Camera 47: LED
22: Monitor 51: Server device 31L: Display for left eye 60: Field of view video 31R: Display for right eye 61: Lyric display object 32 (32R, 32L): Headphone 62: Controller object 33: Microphone

Claims (9)

Performance processing for performing based on performance information,
Detection processing for detecting the arrangement information including the relative direction of the operation device with respect to the headset, which is information related to the arrangement of the operation device ;
In the background image displayed on the display unit arranged in the headset, a lyrics display object is arranged corresponding to the arrangement information detected by the detection process, and the lyrics of the music played in the performance process are displayed in the lyrics display object. And performing a display process. The arrangement information detected by the detection process includes the distance between the headset and the operation device,
2. The karaoke apparatus according to claim 1, wherein the display process arranges the lyric display object with a size corresponding to the distance in the arrangement information detected by the detection process. The arrangement information detected by the detection process includes the inclination of the operating device with respect to the headset,
Display processing, the karaoke apparatus according to claim 1 or claim 2 in response to the inclination in the arrangement information detected by the detection process, characterized in that arranged tilting the lyrics display object. The karaoke apparatus according to claim 3 , wherein the display process regulates the inclination of the lyrics display object when the inclination in the arrangement information detected by the detection process exceeds a predetermined value. The display process
Follow-up mode that arranges the lyrics display object corresponding to the placement information detected by the detection process,
According to any one of claims 1 to 4, characterized in that the fixed mode to place the lyrics display object to a predetermined position on the coordinate system of the background image displayed on the display unit, a selectable Karaoke equipment. The karaoke apparatus according to any one of claims 1 to 5 , wherein the display process arranges an operation object indicating the operation apparatus corresponding to the arrangement information detected by the detection process. The karaoke apparatus according to any one of claims 1 to 6 , wherein when the arrangement information detected by the detection process is out of a predetermined range, the lyrics display object is not arranged in the background video. Display process is a capable of changing the transmission state of the lyric display object, in accordance with the progress of the performance information, any one of claims 1 7, characterized in that to change the transmission state of the lyric display object The karaoke apparatus according to item 1. Performance processing for performing based on performance information,
Detection processing for detecting the arrangement information including the relative direction of the operation device with respect to the headset , which is information related to the arrangement of the operation device ;
In the background image displayed on the display unit arranged in the headset, a lyrics display object is arranged corresponding to the arrangement information detected by the detection process, and the lyrics of the music played in the performance process are displayed in the lyrics display object. A karaoke program characterized by causing an information processing apparatus to execute display processing.