JP2006171215A - Data processor and program for conducting guidance display associated with music playing and lyrics display - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enable a user to conduct an appropriate action according to the operation and the conditions of video cameras or the like, in a karaoke system in which imaging is conducted together with the playing of a karaoke music. <P>SOLUTION: In a karaoke system 1, automatic playing and lyrics display of a karaoke music are conducted and guidance display to be presented to the user is conducted for the action or the condition of video cameras 15 at the present moment, or in the near future, in accordance with the sequence data which are included in karaoke data 1001 to instruct the actions of the video cameras or the like or the instruction data transmitted to a karaoke machine 10 by a controller 11 in accordance with the user's operations. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to a data processing technique in a karaoke system.

  There is a technique for imaging a user who sings along with the performance of karaoke music.

For example, Patent Document 1 discloses a karaoke provided with a video camera that performs zooming and panning operations according to performance of karaoke music according to camera control data that is time-series data prepared in advance corresponding to performance data of karaoke music. A device is proposed.
JP 2002-311969 A

  In the case of the conventional technique described in Patent Document 1, since the user cannot know the operation of the video camera in advance, it is difficult to perform an appropriate operation such as determining a pose according to the operation of the video camera.

  In view of the above situation, the present invention enables a user to perform an operation suitable for the operation and state of an imaging-related device such as a video camera in a karaoke system in which imaging is performed along with the performance of karaoke music. It aims to provide a means.

  In order to achieve the above object, the present invention provides performance instruction data output means for outputting performance instruction data for instructing performance of music, and lyrics display instruction data for instructing display of lyrics. At least one of lyrics display instruction data storage means for storing the timing data indicating the timing at which the image is to be displayed, the imaging device, the processing device for processing the image data, the moving device for moving the imaging device, and the lighting device Processing instruction data storage means for storing processing instruction data for instructing a process to be executed in association with timing data indicating a timing at which the process should be executed in the progress of the music; and the performance instruction data output means In parallel with the output of the performance instruction data by, the processing instruction data is associated with the processing instruction data. The lyric display in parallel with the processing instruction data output means for outputting at a timing according to the timing data, the output of the performance instruction data by the performance instruction data output means and the output of the processing instruction data by the processing instruction data output means The lyrics designated by the instruction data are displayed at a timing corresponding to the timing data associated with the lyrics display instruction data, and guidance according to the process instruction data output by the process instruction data output means is displayed. There is provided a data processing apparatus comprising: display instruction data output means for outputting display instruction data for instructing display at a timing according to timing data associated with data.

  The present invention also provides performance instruction data output means for outputting performance instruction data for instructing performance of music and lyrics display instruction data for instructing display of lyrics at a timing at which the lyrics should be displayed in the progress of the music. Lyric display instruction data storage means for storing in association with the timing data indicating, and the position of the imaging device, the direction of the imaging device, the processing of the imaging device, the position of the lighting device, the direction of the lighting device, the processing of the lighting device and the image data Among the processes of the processing apparatus that performs the processing of the above, state data input means for receiving state data indicating a change in the current state or a state scheduled within a predetermined period of at least one, and a performance instruction by the performance instruction data output means In parallel with the data output, the lyrics designated by the lyrics display instruction data are associated with the lyrics display instruction data. And display instruction data output means for outputting display instruction data for instructing to display guidance according to the state data received by the state data input means while displaying at timing according to the timing data. A data processing apparatus is provided.

  In any of the above-described data processing devices, the display means for receiving the display instruction data by the data processing device displays the lyrics of the karaoke music, and the current state of the imaging device, processing device, moving device or lighting device and Since guidance display for notifying the user of processing to be executed in the near future is performed at the same time, the user can perform the behavior necessary for generating preferable image data in accordance with the operation of the devices.

  Further, in a preferred aspect, the display instruction data output means includes, as the display instruction data, a character, a figure, an illumination device indicating an imaging device or a lighting device that is executing a specific process or scheduled to execute a specific process within a predetermined period, It may be configured to output data instructing display of a symbol, a combination thereof, a combination of these and a color, or a display by lighting a lamp.

  According to the data processing apparatus having such a configuration, the display unit notifies the user of an apparatus that is executing the process or is scheduled to be executed in the near future. For example, the user turns the body toward the imaging apparatus that is performing the imaging process. Appropriate behavior can be performed.

  Further, in another preferred aspect, the display instruction data output means, as the display instruction data, according to a current state of an imaging device, a processing device, a moving device, or a lighting device or a state scheduled after a predetermined time, Data for instructing display of different modes may be output.

  According to the data processing apparatus having such a configuration, the display unit notifies the user of the details of the current or future state of the imaging apparatus or the like in the display mode. Depending on the situation, it is possible to perform a suitable behavior.

  In another preferred embodiment, the data processing device includes position data input means for receiving position data indicating a user's position, and the display instruction data output means adds the lyrics and the guidance as the display instruction data. Further, it may be configured to output data instructing simultaneous display of the position indicated by the position data.

  According to the data processing apparatus having such a configuration, the current position of the user is displayed by the display unit. As a result, the user can easily grasp his / her position in relation to the position of the imaging apparatus or the like, and the operation of the imaging apparatus or the like can be performed. The corresponding behavior can be performed more easily.

  In another preferred embodiment, the data processing device includes image data input means for receiving image data of a user, and the display instruction data output means includes the image as the display instruction data in addition to the lyrics and the guidance. Data for instructing simultaneous display of images indicated by the data may be output.

  According to the data processing apparatus having such a configuration, as a result of displaying the user's current image by the display means, the user can correct his / her behavior while confirming the image obtained as a result of his / her behavior.

  The present invention also provides a program that causes a computer to execute processing performed by the data processing apparatus.

[1. First Embodiment]
[1.1. Configuration of karaoke system]
FIG. 1 shows a state in which a karaoke system 1 according to an embodiment of the present invention is installed in a karaoke box. FIG. 2 shows the configuration of the karaoke system 1. However, in FIG. 1, the infrared receivers 20a to 20l shown in FIG. 2 are omitted for simplification of the drawing. The karaoke system 1 includes a karaoke device 10, a controller 11 for giving a user instruction to the karaoke device 10, picks up a user's voice and outputs a sound signal indicating the voice to the karaoke device 10 at the same time intermittently. Microphones 12a and b having a function of emitting ultrasonic waves of a frequency and continuously irradiating infrared rays of a predetermined wavelength, an amplifier 13 for amplifying a sound signal output from the karaoke apparatus 10, and a sound signal output from the amplifier 13 14L and R that convert and output sound into sound, video cameras 15a to 15d that sequentially generate and output image data indicating captured images, and camera movements that change the positions and directions of the video cameras 15a to 15d by a motor or the like Devices 16a to 16d, lighting devices 17a to 17h for irradiating a user with light of a specified color and brightness, karaoke Displays 18a-d for displaying images in accordance with image data output from the device 10, ultrasonic receivers 19a-d for receiving ultrasonic waves and outputting electrical signals indicating the waveforms of the received ultrasonic waves to the karaoke apparatus 10, and karaoke A media drive 21 for storing data output from the apparatus 10 in a storage medium such as a flash memory is provided. The karaoke apparatus 10 is connected to the Internet 22.

  The karaoke system 1 includes a plurality of microphones, speakers, video cameras, camera moving devices, lighting devices, displays, ultrasonic receivers, and infrared receivers as described above. When it is not necessary to distinguish between them, they are simply called as the microphone 12. In addition, the number of microphones 12 and the like provided in the karaoke system 1 is an example, the number of ultrasonic receivers 19 is four or more, and the direction of the user's body can be specified by the infrared receiver 20 as will be described later. It can be changed freely except for the large number.

  Each of the microphones 12a and 12b includes ultrasonic generators 121a and 121b that generate ultrasonic waves, and intermittently generates ultrasonic waves as described above, but the ultrasonic waves generated by the ultrasonic generators 121a and 121b are generated. The frequencies of the sound waves are different from each other, and it is possible to distinguish which ultrasonic wave is emitted from any of the microphones 12a and b. The ultrasonic generators 121a and 121b are arranged at a time interval sufficiently longer than the time required for the ultrasonic waves emitted from the microphones 12a and 12b to reach the ultrasonic receivers 19a to 19d in the karaoke room. Repeat occurrence. More specifically, for example, the ultrasonic generators 121a and 121b emit ultrasonic waves for 0.5 seconds, do not emit ultrasonic waves for 1 second, and then repeat the same processing. This is because, in the processing of the position / direction calculation unit 205 described later, among the ultrasonic waves received by the ultrasonic receivers 19a to 19d having different distances from the microphones 12a and 12b one after another, the microphone 12a and This is because it can be easily distinguished whether it is emitted from b.

  In addition, the microphones 12a and b are respectively provided with infrared generators 122a and b that generate infrared rays, and continuously irradiate infrared rays as described above. However, the wavelengths of infrared rays generated by the infrared generators 122a and b are as follows. Are different from each other, and it is possible to distinguish which infrared rays are emitted from any of the microphones 12a and b. FIG. 3 is a diagram illustrating a state in which the infrared generator 122 is disposed in the microphone 12. The infrared generator 122 is arranged in a ring shape at the junction between the main body and the head of the microphone 12, and a large number of infrared light emitting diodes 1221 capable of emitting infrared rays of the same wavelength are arranged on the outer peripheral surface thereof. . The microphone 12 can irradiate infrared rays in various directions from the infrared light emitting diodes 1221 arranged in a ring shape in this way. However, if infrared rays can be emitted from the microphone 12 in various directions, the arrangement method of the infrared light emitting diodes 1221 does not need to be ring-shaped.

  Then, the structure of the karaoke apparatus 10 is demonstrated. The karaoke apparatus 10 includes a storage unit 101, an input / output I / F (Interface) 102, and a control unit 103. The storage unit 101 stores a program for instructing various processes by the control unit 103 in advance, and temporarily stores data generated by the control unit 103. Furthermore, the storage unit 101 stores a plurality of karaoke data 1001 in advance.

  Hereinafter, the contents of the karaoke data 1001 will be described. FIG. 4 schematically shows the configuration of the karaoke data 1001. In FIG. 4, as the karaoke data 1001 stored in the storage unit 101, three karaoke data 1001 relating to different music pieces, that is, karaoke data 1001a to 1001c are illustrated. Karaoke data 1001 is data for instructing various processes related to automatic performance of karaoke music and image generation displayed on the display 18. Karaoke data 1001 includes management data, sequence data, and source data group. The management data is data indicating an identification number and a song name for identifying karaoke data.

  As shown in FIG. 4, the sequence data includes a sound track for instructing automatic performance of karaoke music, a lighting track for instructing operation of the lighting device 17, a camera track for instructing operations of the video camera 15 and the camera moving device 16, It includes a visual track that directs image data processing.

  The sound track is performance control data for instructing automatic performance of karaoke music, and is, for example, data in SMF (Standard MIDI File) format. FIG. 5 illustrates the contents of a sound track. However, in FIG. 5, the SMF expression format is simplified for the sake of explanation. The sound track includes a plurality of event data with timing including timing data and event data associated with each other. Timing data is data indicating the timing of music. This is not limited to the timing data included in the sound track, and the same applies to the timing data included in any of the tracks described below. On the other hand, the contents of the event data differ depending on which track is included in the event data. Hereinafter, some of them will be exemplified, but it goes without saying that the types of event data are not limited to those described below.

  The event data included in the sound track is data for instructing the pronunciation of a musical sound and adding an effect to the sound to be generated. For example, the event data [1ch: note on: D4: 74] indicates that a musical tone having a tone pitch D4 assigned to the first channel is to be generated with a tone strength of 74. Here, the channel is an individual performance part included in the music. Also, for example, event data [1ch: delay: 0.5 seconds] instructs to add a 0.5 second delay effect to the sound of the first channel. Further, for example, event data [3ch: reproduction: S514. wav] instructs to reproduce the sound data stored in the sound data file “S514.wav” as the third channel.

  The event data included in the lighting track is a collection of timed event data that instructs the operation of the lighting device 17. Hereinafter, the operation of the lighting device 17 controlled by the event data will be described as being the color and intensity of the illumination. However, for example, the control of other operations such as changing the focus intensity is instructed by the event data. May be. FIG. 6 illustrates the contents of the lighting track. For example, the event data [lighting 1: color: red] instructs the lighting device 17 assigned to the lighting 1 to select red as the lighting color. In the present embodiment, it is assumed that lighting devices 17a to 17h are assigned to each of the lighting devices 1 to 8. Further, for example, the event data [lighting 1: intensity: 0 → 5: 3.2 seconds] indicates that the lighting intensity of the lighting 1, that is, the lighting device 17 a, from the strength indicated by “0” to “5” It is instructed to change the strength indicated by "" in 3.2 seconds. In the following description, the numerical value indicating the intensity of illumination assumes an integer value of “0” to “9”, and a large integer value indicates stronger illumination. “0” indicates that the illumination is turned off. Therefore, for example, the event data [illumination 1: intensity: 0] instructs the illumination device 17a to turn off the illumination. Thus, the event data that does not specify the time required for the change instructs that the change be performed instantaneously.

  The camera track is a collection of timed event data for instructing the operations of the video camera 15 and the camera moving device 16. Hereinafter, the operation of the video camera 15 controlled by the event data will be described as the start / end of the imaging process and the change of the zoom strength. However, other operations such as a change of the focus strength are controlled. You may make it instruct | indicate by event data. The operation of the camera moving device 16 controlled by the event data will be described as changing the position of the video camera 15 and changing the direction in the left-right direction. However, other operations such as changing the direction in the vertical direction are controlled by the event. You may make it instruct | indicate by data. FIG. 7 illustrates the contents of the camera track. For example, the event data [Camera 1: Position: 4] moves the position of the video camera 15 to the position indicated by “4” with respect to the camera moving device 16 to which the video camera 15 assigned to the camera 1 is attached. I have indicated that. In the present embodiment, it is assumed that video cameras 15a to 15d are assigned to the cameras 1 to 4, respectively. The numerical value indicating the camera position is an integer value of “1” to “9”, and as the numerical value increases, the position of the video camera 15 changes from right to left as viewed from the user. To do.

  For example, the event data [Camera 1: Zoom: 7] instructs the video camera 15 assigned to the camera 1, that is, the video camera 15 a to change the zoom magnification to the magnification indicated by “7”. is doing. The numerical value indicating the zoom magnification is an integer value from “1” to “9”, and the larger the numerical value is, the higher the zoom magnification is. Further, for example, the event data [camera 1: direction: 3] indicates that the imaging direction of the video camera 15a is the direction indicated by “3” with respect to the camera moving device 16 to which the video camera 15a is attached, that is, the camera moving device 16a. It is instructed to change. The numerical value indicating the imaging direction of the video camera 15 takes an integer value of “1” to “9”, and the imaging direction of the video camera 15 changes from right to left as the numerical value increases. . When the numerical value indicating the camera position matches the numerical value indicating the camera direction, the video camera 15 faces the vicinity of the center of the karaoke room. Therefore, for example, when event data [Camera 1: Position: 4 → 6: 4.2 seconds] and [Camera 1: Direction: 4 → 6: 4.2 seconds] are transmitted to the camera moving device 16a at the same timing. The camera moving device 16a moves the position of the camera moving device 16a and changes the imaging direction in accordance with these event data. As a result, the camera moving device 16a moves from right to left as viewed from the user, and always moves the karaoke room. It turns to the vicinity of the center. The event data [Camera 1: ON] and [Camera 1: OFF] are event data for instructing the start and end of the imaging process of the camera moving device 16a, respectively.

  The visual track is a collection of timed event data instructing generation of image data indicating an image to be displayed on the display 18. In the present embodiment, the image displayed on the display 18 is an image obtained by superimposing a plurality of display layers. The display layer is directed from the back to the front, and the background layer, the first layer, and the second layer. ,..., Arranged in the order of the # layer and the lyrics layer. However, “#” is an arbitrary natural number. The karaoke apparatus 10 generates image data for display by performing processing on the source image data specified for each display layer in accordance with the visual track and then synthesizing them. However, as will be described later, the image actually displayed on the display 18 is obtained by superimposing the guidance display image data on the image data generated according to the visual track.

  FIG. 8 illustrates the contents of the visual track. For example, event data [background layer: source: S713. jpeg] instructs to use the still image data stored in the still image data file “S713.jpeg” as the source of the image data to be arranged in the background layer. In the following, it is assumed that a file with the extension “.jpeg” is a still image data file, and a file with the extension “.mpeg” is a moving image data file. Therefore, for example, event data [background layer: source: S151. mpeg] instructs to sequentially use image data obtained by reproducing moving image data stored in the moving image data file “S151.mpeg” as a source of image data to be arranged in the background layer. Similarly, the event data [first layer: source: camera 1] sequentially uses image data generated by the imaging processing of the camera 1, that is, the video camera 15a, as the source of image data arranged in the first layer. I am instructing.

  Also, for example, event data [first layer: window size: 7], [first layer: window shape: rectangular], and [first layer: window position: (120, 80)] are respectively stored in the first layer. A window indicated by size “7” is opened, the window shape is changed to a rectangle, and the representative point of the window (for example, the intersection of diagonal lines if the window shape is a rectangle) is the coordinates on the screen of the display 18 ( 120, 80) is instructed to change the window position. When a window is opened, the image placed in that layer is displayed only inside the window. Since the outside of the window is transmissive display, an image of a lower layer is displayed in that portion.

  Further, for example, event data [first layer: fade-in: 3.6 seconds] is obtained by performing fade-in processing over 3.6 seconds on image data arranged in the first layer, and event data [background Layer: Color tone: Color → Sepia: 3.6 seconds] instructs the image data arranged in the background layer to be subjected to a color change process from color to sepia over 3.6 seconds. The processing of these image data is an example, and the visual track can include event data for instructing various types of processing such as a dissolve process and a morph process.

  Event data [lyric layer: (32, 140): “hour”: yellow: display] indicates that the character “hour” is displayed in yellow at the position indicated by the coordinates (32, 140) on the lyrics layer. is doing. Further, the event data [lyric layer: (32, 140): erase] instructs to erase the character displayed at the position indicated by the coordinates (32, 140) on the lyrics layer, that is, to hide it. ing.

  As described above, the display in each display layer indicated by the visual track is combined, and image data indicating an image as shown in FIG. 9 is generated. In FIG. 9, a display part 181 shows a display part by a background layer, a display part 182 shows a display part by a first layer, and a display part 183 shows a display part by a lyrics layer.

  The source data group included in the karaoke data 1001 (see FIG. 4) is a still data instructed to be displayed by the sound data instructed to be played by the event data included in the sound track and the event data included in the visual track. It is a set of various files for storing image data or moving image data for which a reproduction instruction is given.

  Returning to FIG. 2, the description of the configuration of the karaoke apparatus 10 is continued. The input / output I / F 102 includes a storage unit 101 and a control unit 103 included in the karaoke device 10, a controller 11, a microphone 12, an amplifier 13, a video camera 15, a camera moving device 16, a lighting device 17, a display 18, and an ultrasonic receiver. 19, an interface for inputting / outputting analog signals and digital signals between the media drive 21 and the Internet 22. The input / output I / F 102 includes an A / D (Analog to Digital) converter and a D / A (Digital to Analog) converter, and performs conversion between an analog signal and a digital signal as necessary.

  The control unit 103 generates sound data indicating the performance sound of the karaoke music according to the sound track included in the karaoke data 1001, and performs various processing processes on the sound data indicating the sound picked up by the microphone 12, as a result. The obtained sound data is synthesized with sound data indicating the performance sound of karaoke music, and then output to the amplifier 13. Further, the control unit 103 controls the operation of the lighting device 17 according to the lighting track included in the karaoke data 1001. The control unit 103 controls the operation of the video camera 15 and the camera moving device 16 according to the camera track included in the karaoke data 1001. Further, the control unit 103 generates image data by performing various kinds of processing processing and synthesis processing on the user image data, the image data included in the source data group of the karaoke data 1001, and the like according to the visual track included in the karaoke data 1001. The generated image data is output to the display 18. Further, the control unit 103 causes the storage unit 101 to store sound data and image data generated by the processing process and the synthesis process as video data.

  The configuration unit of the control unit 103 includes a sound data block 201 related to sound data processing, an image data block 202 related to image data processing, an operation control block 203 related to operation control of the video camera 15, the camera moving device 16 and the illumination device 17, karaoke. Event data reproduction unit 204 that transmits event data to sound data block 201, image data block 202, and operation control block 203 according to sequence data included in data 1001, and microphone 12 according to a signal input from ultrasonic receiver 19 A position / direction calculation unit 205 that calculates each position is divided. The control unit 103 includes an oscillator (not shown), and the processing of the components of the control unit 103 is synchronized based on a clock signal generated by the oscillator.

  The sound data block 201 includes a sound source unit 2011 that outputs sound data indicating various musical sounds according to event data transmitted from the event data reproducing unit 204, and a source of karaoke data 1001 according to event data transmitted from the event data reproducing unit 204. A sound data reproduction unit 2012 that reproduces sound data included in the data group is provided. The sound data block 201 includes sound data indicating the sound picked up by the microphone 12 and sound data transmitted from the sound source unit 2011 and the sound data reproducing unit 2012 according to the event data transmitted from the event data reproducing unit 204. A sound data processing unit 2013 is provided for outputting to the amplifier 13 sound data obtained by performing processing such as various effect processing on each of them and then synthesizing them.

  The image data block 202 includes a moving image data reproducing unit 2021 that reproduces moving image data included in the source data group of the karaoke data 1001 according to the event data transmitted from the event data reproducing unit 204, and an event transmitted from the event data reproducing unit 204. A lyric screen generation unit 2022 that sequentially generates image data indicating a lyric screen according to the data, and guidance to the user according to event data transmitted from the event data reproduction unit 204 and microphone coordinate data (described later) transmitted from the position / direction calculation unit 205 In accordance with event data transmitted from the event data reproducing unit 204, the guidance screen generating unit 2023 that sequentially generates image data indicating the display screen of the video, the moving image data reproducing unit 2021, the lyrics screen generating unit 2022, and the guidance screen generating unit 202 And image data transmitted from each of the video cameras 15 and an image data processing unit that performs various processing such as effect processing on still image data included in the source data group of the karaoke data 1001 and then synthesizes them. 2024 is provided. The image composition process includes, for example, a process of superimposing another image on an image and a process of compositing both images so that another image can be seen through the image. The image data generated by the image data processing unit 2024 is transmitted to the display 18.

The guidance indicated by the image data generated by the guidance screen generation unit 2023 is a display that provides various information as exemplified below to a user who is singing.
(A) Which video camera 15 is currently used for displaying on the display 18 by the video camera 15.
(B) When the video camera 15 that captures an image used for display on the display 18 is changed, when is the change made?
(C) When the video camera 15 that captures an image used to display the display 18 is changed, which video camera 15 is the changed video camera 15?
(D) Where is the current position of each video camera 15?
(E) Which is the current imaging direction of each video camera 15?
(F) What is the current zoom magnification of each video camera 15?
(G) Where is the user's current location?
(H) In which color each of the illumination devices 17 is currently irradiating light.
(I) At what intensity each of the illumination devices 17 emits light.
(J) When the lighting device 17 starts or stops the light irradiation, when is the start or stop being performed?
(K) When various types of processing such as dissolve processing are performed on the image captured by the video camera 15, which type of processing is performed.
(L) When various types of processing such as dissolve processing are performed on an image captured by the video camera 15, at which time zone the processing is performed.

  A specific guidance display example will be described together in the operation description to be described later. In addition to the above, for example, there are various guidance contents such as in which time zone the change is made when the color or intensity of the light irradiated by the lighting device 17 gradually changes. It is done.

  The operation control block 203 includes a camera operation control unit 2031, a movement operation control unit 2032, and a movement operation control unit 2032 that perform operation control of the video camera 15, the camera moving device 16, and the lighting device 17 in accordance with event data transmitted from the event data reproducing unit 204. An illumination operation control unit 2033 is provided.

  The event data reproducing unit 204 transmits each event data to each component unit according to the content at the timing indicated by the timing data included in the sequence data of the karaoke data 1001 as follows.

(A) Event data for instructing pronunciation of a musical tone included in a sound track → sound source unit 2011.
(B) Event data instructing reproduction of sound data included in the sound track → sound data reproduction unit 2012.
(C) Event data instructing effect processing for sound included in the sound track → sound data processing unit 2013.
(D) Event data included in the lighting track → lighting operation control unit 2033.
(E) Event data regarding the camera position and the imaging direction of the camera included in the camera track → the movement operation control unit 2032.
(F) Event data regarding camera zoom magnification and ON / OFF included in the camera track → camera operation control unit 2031.
(G) Event data instructing reproduction of moving image data included in the visual track → moving image data reproducing unit 2021.
(H) Event data instructing display of lyrics included in the visual track → lyric screen generation unit 2022.
(I) Event data other than event data instructing display of lyrics included in the visual track → image data processing unit 2024.

In addition to the above, the event data reproducing unit 204 sends the following event data to the guidance screen generating unit 2023 at a timing earlier by a predetermined time, for example, 10 seconds than the timing indicated by the timing data included in the sequence data of the karaoke data 1001. Send.
(A) Event data included in the lighting track.
(B) Event data included in the camera track.
(C) Event data that designates one of the video cameras 15 as a source among the event data included in the visual track.

  As described above, the event data reproduction unit 204 gives the guidance screen generation unit 2023 the timing indicated by the timing data, that is, the timing when the lighting device 17 is turned on / off and the video camera 15 is actually switched. Since the event data is transmitted at an early timing, the guidance screen generation unit 2023 can generate image data for guidance display that informs the user in advance of the occurrence of those events.

  The position / direction calculation unit 205 receives sound data indicating ultrasonic waves from each of the ultrasonic receivers 19a to 19d disposed above the karaoke room (see FIG. 1). The ultrasonic waves picked up by the ultrasonic receivers 19a to 19d are ultrasonic waves emitted intermittently from the ultrasonic generators 121a and 121b provided in the microphones 12a and 12b, respectively. As described above, the ultrasonic waves generated from each of the ultrasonic generators 121a and 121b have different frequencies from each other, and the position / direction calculation unit 205 receives two different frequencies from each of the ultrasonic receivers 19a to 19d. Sound data indicating a sound mixed with ultrasonic waves is received. Therefore, the position / direction calculation unit 205 filters the sound data received from the ultrasonic receiver 19 with a band-pass filter that passes only the components of the frequency band that includes one frequency and does not include the other frequency for each of the two frequencies. To do. The sound data obtained by such filtering processing is data indicating the ultrasonic waves emitted from the ultrasonic generator 121a and data indicating the ultrasonic waves emitted from the ultrasonic generator 121b, respectively.

The position / direction calculation unit 205 generates eight sets of data as follows by performing the above-described filtering process on the sound data received from each of the ultrasonic receivers 19a to 19d.
(A) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121a and received by the ultrasonic receiver 19a (referred to as “ultrasonic data aa”).
(B) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121a and received by the ultrasonic receiver 19b (referred to as “ultrasonic data ab”).
(C) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121a and received by the ultrasonic receiver 19c (referred to as “ultrasonic data ac”).
(D) Data indicating an ultrasonic wave emitted from the ultrasonic generator 121a and received by the ultrasonic receiver 19d (referred to as “ultrasonic data ad”).
(E) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121b and received by the ultrasonic receiver 19a (referred to as “ultrasonic data ba”).
(F) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121b and received by the ultrasonic receiver 19b (referred to as “ultrasonic data bb”).
(G) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121b and received by the ultrasonic receiver 19c (referred to as “ultrasonic data bc”).
(H) Data indicating the ultrasonic wave emitted from the ultrasonic generator 121b and received by the ultrasonic receiver 19d (referred to as “ultrasonic data bd”).

  The ultrasonic data aa, ab, ac, and ad are data indicating the same ultrasonic wave, but the distance between the ultrasonic generator 121a provided in the microphone 12a and each of the ultrasonic receivers 19a to 19d is different. The timing at which each of the ultrasonic receivers 19a to 19d receives them is different. The same applies to the ultrasonic data ba, bb, bc and bd. The position / direction calculation unit 205 calculates the positions of the microphones 12a and 12b in the karaoke room based on the timing difference. Since the calculation method of the positions of the microphones 12a and b in the karaoke room is the same, the calculation method will be exemplified below by taking the case of calculating the position of the microphone 12a as an example.

FIG. 10 is a view of the karaoke room as viewed from above. In FIG. 10, the positions of the ultrasonic receivers 19a to 19d are point A (0, 0, r), point B (0, q, r), point C (p, q, r) and point D ( p, 0, r). Further, the position of the microphone 12a is shown as a point X (x, y, z). Here, p, q, and r are constants (meters) indicating the width, depth, and height of the karaoke room. On the other hand, x, y and z are variables (meters) indicating the coordinates of the microphone 12a. Here, the following equation holds. Where t _A , t _B , t _C and t _D indicate the time (seconds) required for the sound emitted from the point X to reach each of the points A to D, and S propagates the sound in the air. Indicates the speed (meter / second).

By the way, as already described, the ultrasonic generator 121 does not emit ultrasonic waves for 1 second after continuing to emit ultrasonic waves for 0.5 seconds, for example. Therefore, the position / direction calculation unit 205 continues for about 1 second from each of the ultrasonic receivers 19a to 19d after the period in which the ultrasonic data is not received from any of the ultrasonic receivers 19a to 19d continues for about 1 second. The ultrasonic data aa to ad are received. The ultrasonic data received as such is data indicating the same ultrasonic wave emitted from the ultrasonic generator 121a. For example, the position / direction calculation unit 205 sets the rising timing of the value of the ultrasound data aa, that is, the reception timing of the leading data that changes across the predetermined value in which the value of the ultrasound data aa exceeds the level of the noise component from the bottom to the top. a reference to calculate the time until the rise timing of each of the ultrasound data ab~ad (seconds) as t _1, t ₂ and t _3.

For example, when the point X is at the position shown in FIG. 10, the following equation holds.

  By substituting Equations 1 to 4 into Equations 5 to 7, a ternary simultaneous equation consisting of three equations is obtained. Here, since x, y, and z have positive values, the position / direction calculation unit 205 can uniquely calculate x, y, and z by solving the simultaneous equations. The above is the method by which the position / direction calculation unit 205 calculates the position of the microphone 12.

The position / direction calculation unit 205 has coordinate data X _a (x _a , y _a , z _a ) and X _b (x _b , y _b , z _b ) indicating the positions of the microphones 12a and b calculated as described above. (Hereinafter referred to as “microphone coordinate data”) are sequentially transmitted to the guidance screen generation unit 2023. The microphone coordinate data is used by the guidance screen generation unit 2023 to generate image data indicating the position of the user.

  By the way, the method by which the position / direction calculation unit 205 calculates the position of the microphone 12 is not limited to the above. For example, when the position of the microphone 12 in the height direction is small enough to be ignored, only the position of the microphone 12 in the two-dimensional plane, that is, x and y may be obtained. In that case, since there are two variables, the number of ultrasonic receivers 19 may be three or more. Further, instead of providing the ultrasonic generator 121 in the microphone 12, the ultrasonic generator 121 is arranged at the position of the ultrasonic receivers 19 a to 19 d in FIG. 10, the display 18 is provided in the microphone 12, and the position / direction calculating unit 205. May be configured to receive ultrasonic data from the microphone 12 so that the position / direction calculation unit 205 calculates the position of the microphone 12 according to a calculation method similar to the above.

  The position / direction calculation unit 205 receives the sound data indicating the ultrasonic waves from the ultrasonic receiver 19 as described above, while the infrared receiver 20a disposed at the same height of the wall surface of the karaoke room (see FIG. 1). The reception notification data indicating which wavelength of infrared light is being received is received at a sufficiently short interval, for example, an interval of 10 milliseconds, from each of those receiving infrared rays.

  FIG. 11 is a view of the karaoke room as seen from above as in FIG. 10, but shows the arrangement position of the infrared receiver 20 instead of the ultrasonic receiver 19. In FIG. 11, a point X indicates the position of the microphone 12a, and a solid line arrow extending from the point X indicates the front direction of the user who is using the microphone 12a. Moreover, the dotted line extended from the point X has shown the boundary of the range which the infrared rays irradiated from the microphone 12a reach the wall of a karaoke room. That is, a part of infrared rays irradiated from the infrared generator 122 of the microphone 12a is blocked by the user's body and does not reach the wall of the karaoke room. Therefore, only the infrared rays irradiated in the direction indicated by the solid-line double-sided arrow attached around the point X in FIG. 11 reach the wall of the karaoke room. As a result, in this case, the infrared receivers 20a to 20f transmit the reception notification data to the position / direction calculation unit 205, but the infrared receivers 20g to 20l do not transmit the reception notification data.

  When receiving the reception notification data from some of the infrared receivers 20 as described above, the position / direction calculation unit 205 receives the calculated coordinate data indicating the position of the microphone 12a and the infrared reception of the transmission source of the received reception notification data. Based on the coordinate data indicating the position of the container 20, the front direction of the user is estimated. That is, the position / direction calculation unit 205 assumes that the range in which infrared rays reach from the point X is a range indicated by a dotted double-sided arrow attached around the point X, and extends from the center of the range, that is, from the point X. It is estimated that the user is facing in the direction of the dotted arrow. The position / direction calculation unit 205 sequentially transmits data indicating the estimated direction (hereinafter referred to as “user direction data”) to the guidance screen generation unit 2023. The user direction data is used by the guidance screen generation unit 2023 to generate image data indicating the user direction. The user direction estimated by the position / direction calculation unit 205 causes an error between the actual direction as shown in FIG. 11, but by increasing the number of infrared receivers 20, the error can be reduced to a practical one. Can be reduced to a range.

  By the way, the method by which the position / direction calculation unit 205 estimates the direction of the user is not limited to the above. For example, an azimuth meter may be provided in the microphone 12, and the user's direction may be estimated based on the azimuth measured by the azimuth meter.

  Further, as described above, instead of specifying the position of the microphone 12 using ultrasonic waves and estimating the direction of the user using infrared rays, it is conceivable to perform them simultaneously by image analysis processing. Specifically, for example, an image having a function of arranging a video camera that performs imaging in the floor direction on the ceiling of a karaoke room and extracting an image that satisfies a predetermined condition from image data obtained by imaging processing of the video camera. An analysis unit is provided in the karaoke apparatus 10. The image analysis unit extracts a substantially circular image having a size corresponding to a human head from image data transmitted from a video camera arranged on the ceiling, and specifies the position as the position of the user. Similarly, the image analysis unit extracts an image that matches the color and shape of the microphone from the image data transmitted from the video camera, and specifies the position as the position of the microphone 12. In that case, for example, the accuracy of image extraction of the microphone 12 can be improved by giving a characteristic to the color of the microphone 12. Further, by making the colors of the microphones 12 a and b different from each other, the image analysis unit can specify which image corresponds to which microphone 12. The image analysis unit estimates the direction from the identified user position to the position of the microphone 12 as the front direction of the user, and sequentially transmits data indicating the user position and direction to the guidance screen generation unit 2023. .

  Further, instead of specifying or estimating the position and direction of the user by image analysis processing, a similar result can be obtained by providing an acceleration sensor in the microphone 12. In this case, the microphone 12 is provided with a calculation unit that estimates the moving direction and moving distance from the initial position based on the acceleration detected by the acceleration sensor and its direction. When using the karaoke system 1, the user stands at a predetermined position in the karaoke room with a microphone 12 in a predetermined direction, and presses a predetermined button on the microphone 12. At the same time as setting the position, it instructs the start of calculation. Thereafter, when the microphone 12 is moved by the user's movement, the calculation unit estimates the moving direction and moving distance, and sequentially adds them to the initial position, thereby estimating the current position and direction of the microphone 12. The microphone 12 sequentially transmits data indicating the position and direction thus estimated to the guidance screen generation unit 2023.

[1.2. Operation of karaoke system]
Hereinafter, the operation of the karaoke system 1 will be described. In the following description, it is assumed that two users use the karaoke system 1 at the same time, but the number of users who can use the karaoke system 1 at the same time can be freely changed up to the number of microphones 12 included in the karaoke system 1.

[1.2.1. Selection operation by the user]
FIG. 12 illustrates a screen displayed on the display 18 when the user starts using the karaoke system 1. When the user instructs the karaoke apparatus 10 to start operation using the controller 11, the karaoke apparatus 10 first displays the screen of FIG. 12A on the display 18, and allows the user to select either the manual mode or the automatic mode. Prompt to select. The automatic mode is a process in which the karaoke device 10 controls the video camera 15, the camera moving device 16, and the lighting device 17 and selects one of the video cameras 15 as a source of image data by the image data processing unit 2024. This mode is performed according to event data included in karaoke data 1001. On the other hand, the manual mode is a mode in which the karaoke apparatus 10 performs these processes in accordance with instructions from the user to the karaoke apparatus 10 using the controller 11.

  More specifically, when the manual mode is selected by the user on the screen of FIG. 12A, the event data reproducing unit 204 transmits event data included in the sequence data to each component of the karaoke apparatus 10. Event data included in the illumination track and the camera track is not transmitted to the operation control block 203. Also, the event data playback unit 204 does not transmit, to the image data processing unit 2024, event data that designates one of the video cameras 15 as the source of each display layer among the event data included in the visual track. Instead, the operation control block 203 and the image data processing unit 2024 follow the instruction data transmitted from the controller 11 according to the user's operation, and control the video camera 15 and the like as the source of the image data used for the processing. Perform the selection process.

  In the instruction data transmitted from the controller 11 in response to the user's operation, in addition to instructing the karaoke apparatus 10 to execute the process immediately, for example, “switch from camera 1 to camera 4 after 5 seconds” in the near future. Also included is a process to be performed and an instruction for a process with a certain processing time such as “change the color of the illumination 2 from red to blue in 5 seconds”.

  Regarding the processes other than the control process of the video camera 15 and the selection process of the video camera 15, the karaoke apparatus 10 performs these processes in accordance with the event data in both the manual mode and the automatic mode.

  When the user selects one of the modes using the controller 11 on the screen of FIG. 12A and performs an operation of pressing the “OK” button, the karaoke apparatus 10 displays the screen of FIG. 18, and prompts the user to select whether or not to select each of the camera and illumination as guidance display targets.

  In the screen of FIG. 12B, the user uses the controller 11 to perform an operation of checking or unchecking whether or not to perform guidance display for each of the camera and lighting, and then pressing the “OK” button. When the operation is performed, the karaoke apparatus 10 displays the screen of FIG. 12C on the display 18 and prompts the user to select a guidance display method. In this embodiment, as an example, the guidance display method is selected from “display with characters”, “display with bars”, “display with layout (1)”, and “display with layout (2)”. It shall be possible. The display corresponding to each of these methods will be described in the “guidance display mode” below.

[1.2.2. Sequence data playback processing]
When the user performs an operation of selecting one of the methods using the controller 11 on the screen of FIG. 12C and then performs an operation of pressing the “OK” button, the karaoke apparatus 10 displays the screen shown in FIG. A screen is displayed on the display 18 and prompts the user to select a song. When the karaoke apparatus 10 displays the screen of FIG. 12D, the karaoke device 10 reads out data indicating the music title from the management data included in each of the karaoke data 1001 stored in the storage unit 101, displays them as a list, and displays it to the user. Selectable.

  When the user performs an operation of selecting one of the songs using the controller 11 on the screen of FIG. 12D and then performs an operation of pressing the “OK” button, the karaoke device 10 is first selected by the user. The sequence data included in the karaoke data 1001 corresponding to the tune name is read from the storage unit 101. In the following processing, the karaoke apparatus 10 uses sequence data and source data group included in the read karaoke data 1001.

  When the automatic mode is selected by the user, the event data reproduction unit 204 performs prefetch processing of event data to be transmitted to the guidance screen generation unit 2023. That is, the event data playback unit 204 uses (b) event data included in the illumination track, (b) event data included in the camera track, and (c) based on the start timing of playback of the karaoke music according to the sound track. Among the event data included in the visual track, each of the event data that designates one of the video cameras 15 as a source is a timing earlier by a predetermined time, for example, 10 seconds than the timing according to the associated timing data Then, the process of sequentially transmitting to the guidance screen generation unit 2023 is started. Thereafter, when the playback timing of karaoke music is reached, the event data playback unit 204 displays the event data included in the sequence data at a timing according to the associated timing data, according to the type of event data. A process of sequentially transmitting to components other than the generation unit 2023 is started. As a result, the guidance screen generation unit 2023 can obtain information such as the switching timing of the video camera 15 from the event data playback unit 204 at a timing earlier than the automatic performance of the karaoke music by a predetermined time.

  On the other hand, when the manual mode is selected by the user, as described above, the event data reproducing unit 204 does not transmit the event data of (A) to (C) to the guidance screen generating unit 2023, and the guidance screen generating unit 2023 receives the same instruction | indication by a user from the controller 11, instead of receiving those event data from the event data reproduction | regeneration part 204. FIG. Therefore, the event data reproduction unit 204 does not perform prefetching and transmission of event data to the guidance screen generation unit 2023 described above, and the event data included in the sequence data at a timing according to the associated timing data. A process of sequentially transmitting to components other than the guidance screen generator 2023 corresponding to the type of event data is started.

  The sound data block 201 performs automatic performance of karaoke music according to the event data transmitted from the event data reproducing unit 204, and also includes sound data indicating voice picked up by the microphone 12 and sound data indicating automatic performance of karaoke music. The sound data generated by performing the processing process and synthesizing them are sequentially output to the amplifier 13. At that time, conversion from a digital signal to an analog signal is performed by the input / output I / F 102. The amplifier 13 amplifies the sound signal input from the sound data block 201 via the input / output I / F 102. The sound signal amplified by the amplifier 13 is generated by the speaker 14. The sound thus generated is a sound obtained by effect processing and mixing processing of the performance sound of karaoke music and the user's singing sound.

  The operation control block 203 also provides control data for instructing the video camera 15 to turn on / off the imaging process and change the zoom magnification in accordance with the event data transmitted from the event data reproducing unit 204, and to the camera moving device 16. Control data for instructing the movement of the camera 15 and the change of the imaging direction and the control data for instructing the illumination device 17 to change the color and intensity of the illumination are transmitted. In accordance with the control data, the video camera 15, the camera moving device 16, and the illumination device 17 each perform a designated operation, and as a result, a series of image data transmitted from the video camera 15 to the image data processing unit 2024 changes. It will be rich.

  Further, the image data block 202 generates image data as follows according to the event data transmitted from the event data reproducing unit 204, and transmits the generated image data to the display 18. First, the moving image data reproducing unit 2021 receives event data instructing reproduction of moving image data from the event data reproducing unit 204, reads out moving image data instructed by the received event data from the source data group, and reproduces the read moving image data. Are sequentially transmitted to the image data processing unit 2024. The lyric screen generation unit 2022 receives the event data instructing the lyric display from the event data reproduction unit 204, and generates image data for displaying the specified character in the specified position and manner according to the received event data. The generated image data is sequentially transmitted to the image data processing unit 2024. In addition, the guidance screen generation unit 2023 instructs the event data reproduction unit 204 to perform selection processing of the video camera 15, movement processing of the video camera 15 by the camera moving device 16, processing for changing the illumination color or the like by the lighting device 17, and the like. In accordance with the received event data, image data indicating the results of those processes and processes scheduled to be executed in the near future is generated, and the generated image data is sequentially transmitted to the image data processing unit 2024.

  The image data processing unit 2024 operates on the image data transmitted from each of the video camera 15 and the moving image data reproduction unit 2021 and the still image data included in the source data group according to the event data received from the event data reproduction unit 204. Image data of each display layer is generated by performing the processing. The image data processing unit 2024 superimposes the generated image data of each display layer in the order of the background layer, the first layer, the second layer, and so on to generate new image data. Further, the image data processing unit 2024 superimposes the lyrics layer image data transmitted from the lyrics screen generation unit 2022 and the guidance display image data transmitted from the guidance screen generation unit 2023 on the generated image data. Thus, image data including a background image, a user image, lyrics, and guidance is generated. The image data processing unit 2024 sequentially transmits the image data thus generated to the display 18.

[1.2.3. Guidance display mode]
FIG. 13 is a diagram illustrating a screen displayed on the display 18 when the karaoke apparatus 10 is performing sequence data reproduction processing. When “display with text” is selected by the user on the screen of FIG. 12C, a screen including a guidance display as shown in FIG. 13A is displayed. In the screen of FIG. 13 (a), in the vicinity of the lyric display, the display object 1301 showing the video camera 15 scheduled to perform imaging processing at the utterance timing of the lyric and the lighting device 17 at the utterance timing of the lyrics. A display object 1302 is displayed that indicates the color of the light that is to be irradiated by either of the characters.

  The display object 1301 and the display object 1302 specify event data for the guidance screen generation unit 2023 to specify the source of image data, event data for instructing ON / OFF of the imaging process of the video camera 15, and the color and intensity of illumination. The event data is generated based on the instruction data transmitted from the controller 11 in response to a user operation instead of the event data.

  According to the example of FIG. 13A, after the user who is singing sings the lyrics of “The flow of time”, the video camera 15 that captures the image of the user is the video camera 15a (corresponding to the camera 1). At the same time as the video camera 15c (corresponding to the camera 3) is switched, it is possible to know in advance that the color of the illumination changes while singing the lyrics of “The flow of time”.

  When “display with bar” is selected by the user on the screen of FIG. 12C, a screen including a guidance display as shown in FIG. 13B is displayed. In the screen of FIG. 13B, in the vicinity of the lyric display, the display object 1311 showing the video camera 15 to be imaged at the utterance timing of the lyric with a bar, and light irradiation at the utterance timing of the lyrics as well. A display object 1312 showing a lighting device 17 to be performed by a bar is shown. Moreover, the display thing 1313 shown by FIG.13 (b) is a display which shows the present timing in a karaoke music. In the display object 1311, the thickness of the bar indicates the size of the zoom magnification of the video camera 15, for example. In addition, the diagonal line drawn from the camera 1 bar to the camera 3 bar included in the display object 1311 causes the image data transmitted from the camera 1 to fade out from the camera 3 in that time zone, for example. It shows that a process for fading in image data to be transmitted, that is, a dissolve process is performed. Since there are a plurality of types of such processing, the types may be distinguished by the color of the bar or diagonal line, or the type of processing may be displayed in the vicinity of the bar or diagonal line. In the display object 1312, the thickness of the bar indicates, for example, the intensity of irradiation light from the illumination device 17. Further, the color of the bar indicates the color of the irradiation light of the lighting device 17.

  The display object 1311, the display object 1312, and the display object 1313 include event data for the guidance screen generation unit 2023 to instruct the source of the image data, event data for instructing to change the zoom magnification of the video camera 15 and ON / OFF of the imaging process, Generated based on event data for instructing various types of processing on image data, event data for instructing the color and intensity of illumination, or instruction data transmitted from the controller 11 in response to a user operation instead of the event data Is.

  According to the example of FIG. 13B, in addition to the switching timing of the video camera 15 that captures the image and the timing of changing the color of the illumination, the user can change the zoom magnification and illumination intensity of each video camera 15, a plurality of times. Detailed information such as a time zone during which processing using image data transmitted from the video camera 15 is performed can be known in advance during singing.

  In the screen of FIG. 12C, when “display in layout (1)” is selected by the user, a screen including a guidance display as shown in FIG. 13C is displayed. In the screen of FIG. 13C, in the vicinity of the lyric display, the video camera 15 that is scheduled to perform imaging processing at the utterance timing of the lyric, the illumination device 17 that is scheduled to perform light irradiation, and the current user position And a display object 1321 and a display object 1322 indicating the direction. The squares included in the display object 1321 and the display object 1322 indicate the position of the video camera 15 that performs the imaging process. A square indicating the video camera 15 moves in the display object 1321 and the display object 1322 as the camera moving device 16 moves. The circles included in the display object 1321 and the display object 1322 indicate the position of the illuminating device 17 that irradiates light, and at the same time indicate the color of the irradiated light by the color of the circular display. A combined figure of a circle and a triangle shown near the center of the display object 1321 and the display object 1322 is a figure indicating the current position and direction of the user, and the display position and direction are always changed according to the user's movement. Change.

  The display object 1321 and the display object 1322 are event data for the guidance screen generation unit 2023 to instruct the source of the image data, event data to instruct ON / OFF of the imaging process of the video camera 15, and an event to instruct the movement of the video camera 15. Data, event data indicating the color and intensity of illumination, or instruction data transmitted from the controller 11 in response to a user operation instead of the event data, and microphone coordinate data transmitted from the position / direction calculation unit 205 And generated based on user direction data.

  According to the example of FIG. 13C, the user can determine at which timing the video camera 15 arranged at which position and the lighting device 17 arranged at which position and at which timing. Whether or not to irradiate light with colors can be easily and visually known in advance during singing.

  In the screen of FIG. 12C, when “display in layout diagram (2)” is selected by the user, a screen including a guidance display as shown in FIG. 13D is displayed. In the screen of FIG. 13D, the entire screen is viewed as if the karaoke room is viewed from above, a display object 1331 indicating each of the lighting devices 17, a display object 1332 indicating each of the video cameras 15, and a display indicating the user. Object 1333 is shown in the corresponding position. The display object 1331 indicates the intensity and color of the light currently irradiated by the corresponding lighting device 17 according to the size and color. In addition, when the corresponding lighting device 17 finishes the light irradiation in the near future, the display object 1331 is displayed in a blinking manner, for example, in order to notify the user of the end in advance. Further, in order to accurately notify the user of the timing when the illumination device 17 ends the light irradiation, the remaining time until the irradiation end timing is “3!” “2!” “1!” In the vicinity of the display object 1331. In this way, a countdown display may be performed. The display object 1334 exemplifies such display.

  When there is a lighting device 17 that starts light irradiation in the near future, a display object 1331 corresponding to the lighting device 17 is displayed in a blinking manner. However, in order to distinguish from the display object 1331 corresponding to the illumination device 17 that will end the irradiation in the near future, the display object 1331 corresponding to the illumination device 17 that starts the irradiation in the near future is, for example, a hollow (circular outline). Only colored). Also in this case, the remaining time until the irradiation start timing may be displayed in the vicinity of the display object 1331 as “3!”, “2!”, “1!”.

  The display object 1332 indicates the position, imaging direction, and zoom magnification of the video camera 15 that is currently performing imaging processing, depending on the position, direction, and size thereof. Therefore, when the position and imaging direction of the video camera 15 are changed by the camera moving device 16, the position and direction of the display object 1332 on the screen change accordingly. Similarly to the display object 1331 indicating the lighting device 17, the display object 1332 is displayed in a blinking manner, for example, in order to notify the user in advance when the corresponding video camera 15 ends the imaging process in the near future. Is done. Similar to the case of the display object 1331, the remaining time until the end timing of the imaging process may be counted down in the vicinity of the display object 1332 in order to accurately notify the user of the end timing.

  In addition, when there is a video camera 15 that starts imaging processing in the near future, the display object 1332 corresponding to the video camera 15 is displayed blinking with a hollow as in the case of the display object 1331. Similar to the case of the display object 1331, the remaining time until the start timing of the imaging process may be counted down in the vicinity of the display object 1332 in order to accurately notify the user of the start timing.

  The display object 1333 is a figure indicating the current position and direction of the user, and the display position and direction always change according to the movement of the user. The display object 1331 to the display object 1334 are event data for the guidance screen generation unit 2023 to instruct the source of image data, event data for instructing to change the zoom magnification of the video camera 15 and ON / OFF of the imaging process, and Event data for instructing movement and change of imaging direction, event data for instructing the color and intensity of illumination, or instruction data transmitted from the controller 11 in response to a user operation instead of these event data, and position direction This is generated based on microphone coordinate data and user direction data transmitted from the calculation unit 205.

  According to the example of FIG. 13D, the user can easily know the current operation status of the lighting device 17 and the video camera 15 and the details of the position and imaging direction of the video camera 15 visually. If there is a video camera 15 and a lighting device 17 that will end or start operation in the near future, the timing can be known in advance during singing.

  Various forms of guidance display other than those shown in FIG. 13 are conceivable. For example, when the color of light irradiated by the lighting device 17 changes, in the guidance display shown in FIG. 13D, the near future lighting near the display object 1331 indicating the current state of the lighting device 17 A display object indicating the state of the device 17 may be displayed. Further, instead of blinking the display object to indicate the video camera 15 or the like that will end or start processing in the near future, for example, the shade of the display object may be changed.

  As described above, according to the karaoke system 1, along with the lyrics of the karaoke music, the display 18 displays guidance indicating the current state of the video camera 15, the camera moving device 16, and the lighting device 17 and their state in the near future. Therefore, the user can schedule an operation according to a change in the state of the video camera 15 or the like in advance during singing. As a result, the user can easily perform operations that have been difficult according to the prior art, such as posing at a timing suitable for switching the video camera 15, and can comfortably enjoy singing and operations associated with singing. it can. Further, since the video of the user who operates in synchronization with the operation timing of the video camera 15 or the like is displayed on the display 18, it is preferable for the viewer.

  When the performance of the karaoke music piece is completed, the user uses the sound data generated by the sound data processing unit 2013 of the karaoke apparatus 10 and the image data generated by the image data processing unit 2024 as video data via the media drive 21. Can be stored in a storage medium such as a flash memory, or transmitted to a desired server device or the like via the Internet 22. As a result, the user can repeatedly enjoy the video data generated by the karaoke system 1 at home or send it to a friend.

  By the way, the karaoke apparatus 10 may be realized by dedicated hardware, or may be realized by causing a general-purpose computer to execute processing according to an application program. When the karaoke apparatus 10 is realized by a general-purpose computer, each component of the control unit 103 includes a CPU (Central Processing Unit) provided in the general-purpose computer and a DSP (Digital Signal Processor) operating under the control of the CPU in the application program. It is realized as a function of a general-purpose computer by performing processing according to each included module in parallel.

  In the embodiment described above, the position and direction of the lighting device 17 are described as being fixed. However, the position and direction of the lighting device can be changed, and the change is notified to the user in advance by a guidance display. You may do it. A display method similar to that for the video camera 15 can be used.

  In the above-described embodiment, the image data displayed on the display 18 is superimposed on the image data of the guidance display generated by the guidance screen generation unit 2023, thereby displaying the lyrics display and the guidance display at the same time. However, for example, an LED (Light Emitting Diode) corresponding to each of the video camera 15 and the lighting device 17 is provided in the vicinity of the display 18 and is turned on, flickering, turned off, etc. On the other hand, the operations of the video camera 15, the camera moving device 16, and the lighting device 17 may be notified in advance.

  In the above-described embodiment, the display 18 has been described as being disposed on the wall surface of the karaoke room. However, for example, the display 18 formed of a small liquid crystal may be attached to the microphone 12.

[2. Second Embodiment]
In said 1st Embodiment, the karaoke system 1 is comprised by the combination of various apparatuses, such as the karaoke apparatus 10 and the microphone 12 connected to the karaoke apparatus 10. FIG. However, it can be freely changed which of the components of the karaoke system according to the present invention is combined and arranged in the same housing. 14 is a block diagram showing a configuration of a karaoke system 2 in which the karaoke system according to the present invention is realized by an arrangement different from that of the karaoke system 1 shown in FIG. Hereinafter, the karaoke system 2 will be described as a second embodiment of the present invention.

  However, since the configuration and operation of the karaoke system 2 are similar to those of the karaoke system 1 in many respects, only the points where the karaoke system 2 is different from the karaoke system 1 will be described below. In FIG. 14, among the constituent parts of the karaoke system 2, constituent parts that are the same as those of the karaoke system 1 are given the same reference numerals. Further, in FIG. 14, for simplification of the drawing, an arrow indicating a detailed data flow between the components is omitted, and only one component of the same kind composed of a plurality of components is shown.

  The karaoke system 2 includes a sound data processing device 31, an image data processing device 32, and a central management device 33 instead of the karaoke device 10 of the karaoke system 1. The sound data processing device 31 includes a control unit 1031 that includes components included in the sound data block 201 in the karaoke system 1, a storage unit 1011, and an input / output I / F 1021. The image data processing device 32 includes a configuration unit included in the image data block 202 in the karaoke system 1, a control unit 1032 including an ultrasonic receiver 19, an infrared receiver 20, and a position / direction calculation unit 205, a storage unit 1012, and an input / output unit. An I / F 1022 is provided. The central management device 33 includes a control unit 1033 including an event data reproduction unit 204 in the karaoke system 1, a storage unit 1013, and an input / output I / F 1023.

  Input / output I / Fs 1021 to 1023 are interfaces for inputting / outputting data to / from other devices in the same manner as the input / output I / F 102 of the karaoke apparatus 10. Similar to the storage unit 101 of the karaoke apparatus 10, the storage units 1011 to 1013 store programs for instructing various processes by the control units 1031 to 1033 in advance, and temporarily generate data generated by the control units 1031 to 1033. Remember. Further, for example, the storage unit 1013 of the central management device 33 stores a plurality of karaoke data 1001 in advance.

  In addition, the video camera 15, the camera moving device 16, and the lighting device 17 in the karaoke system 2 include a camera operation control unit 2031, a moving operation control unit 2032, and a lighting operation control unit 2033, respectively.

  In the karaoke system 2, when the user instructs playback of the karaoke music using the controller 11, the central management device 33 follows the sequence data included in the karaoke data 1001 corresponding to the music selected by the user, according to the contents of the event data, Event data is transmitted to each of the sound data processing device 31, the image data processing device 32, the video camera 15, the camera moving device 16, and the lighting device 17. The central management device 33 transmits source data required for processing instructed by the event data to the sound data processing device 31 or the image data processing device 32 prior to or in parallel with the transmission of the event data. However, instead of the central management device 33 transmitting the source data, the sound data processing device 31 and the image data processing device 32 store the source data necessary for each processing in the storage unit 1011 and the storage unit 1012 in advance. Also good.

  The sound data processing device 31 performs sound data reproduction processing and processing processing according to the event data received from the central management device 33, and outputs the generated sound data to the amplifier 13. The sound data processing device 31 transmits the generated sound data to the central management device 33, and the central management device 33 stores the sound data transmitted from the sound data processing device 31 in the storage unit 1013.

  The image data processing device 32 performs image data reproduction processing and processing processing according to the event data received from the central management device 33, and transmits the generated image data to the display 18. The image data processing device 32 transmits the generated image data to the central management device 33, and the central management device 33 stores the image data transmitted from the image data processing device 32 in the storage unit 1013.

  As described above, the sound data and the image data sequentially stored in the storage unit 1013 are integrated as video data, and are reproduced, transmitted, stored in a storage medium, and the like according to a user instruction.

  Also in the karaoke system 2 described above, in the same way as the karaoke system 1 in the first embodiment, the image displayed on the display 18 includes a guidance screen generation in addition to the lyric display image generated by the lyric screen generation unit 2022. As a result of synthesizing the guidance display image generated by the unit 2023, the user sings the current state of the video camera 15, the camera moving device 16, and the lighting device 17 and their state in the near future together with the lyrics of the karaoke music. You can know in advance. As a result, the user can schedule an operation corresponding to the state change of the video camera 15 or the like in advance.

  By the way, the sound data processing device 31, the image data processing device 32, and the central management device 33 may be realized by dedicated hardware, or may be realized by causing a general-purpose computer to execute processing according to an application program. Good. When the sound data processing device 31, the image data processing device 32, and the central management device 33 are realized by a general-purpose computer, each component of the control unit 1031, the control unit 1032, and the control unit 1033 includes a CPU included in the general-purpose computer and the CPU. A DSP operating under control performs the processing according to each module included in the application program in parallel, and is realized as a function of a general-purpose computer.

It is the figure which showed a mode that the karaoke system which concerns on 1st Embodiment of this invention was installed in the karaoke box. It is the block diagram which showed the structure of the karaoke system which concerns on 1st Embodiment of this invention. It is the figure which illustrated a mode that the infrared generator was arranged in the microphone concerning a 1st embodiment of the present invention. It is the figure which showed typically the structure of the karaoke data which concern on 1st Embodiment of this invention. It is the figure which illustrated the contents of the sound track concerning a 1st embodiment of the present invention. It is the figure which illustrated the contents of the lighting track concerning a 1st embodiment of the present invention. It is the figure which illustrated the contents of the camera track concerning a 1st embodiment of the present invention. It is the figure which illustrated the contents of the visual track concerning a 1st embodiment of the present invention. It is the figure which illustrated the screen displayed on the display of the karaoke system which concerns on 1st Embodiment of this invention. It is the figure which looked at the karaoke room which has arrange | positioned the ultrasonic receiver which concerns on 1st Embodiment of this invention from upper direction. It is the figure which looked at the karaoke room which has arrange | positioned the infrared receiver which concerns on 1st Embodiment of this invention from upper direction. It is the figure which illustrated the screen displayed on the display of the karaoke system which concerns on 1st Embodiment of this invention. It is the figure which illustrated the screen displayed on the display of the karaoke system which concerns on 1st Embodiment of this invention. It is the block diagram which showed the structure of the karaoke system which concerns on 2nd Embodiment of this invention.

Explanation of symbols

  1, 2 ... Karaoke system, 10 ... Karaoke device, 11 ... Controller, 12 ... Microphone, 13 ... Amplifier, 14 ... Speaker, 15 ... Video camera, 16 ... Camera moving device, 17 ... Lighting device, 18 ... Display, 19 ... Ultrasonic wave receiver, 20 ... Infrared receiver, 21 ... Media drive, 22 ... Internet, 31 ... Sound data processing device, 32 ... Image data processing device, 33 ... Central management device, 101, 1011, 1012, 1013 ... Storage unit , 102, 1021, 1022, 1023, input / output I / F, 103, 1031, 1032, 1033, control unit, 121, ultrasonic generator, 122, infrared generator, 181, display portion, 182, display portion, 183 ... display part 201 ... sound data block 202 ... image data block 203 ... operation control block 04 ... Event data playback unit, 205 ... Position / direction calculation unit, 1001 ... Karaoke data, 1221 ... Infrared light emitting diode, 2011 ... Sound source unit, 2012 ... Sound data playback unit, 2013 ... Sound data processing unit, 2021 ... Movie data playback , 2022 ... Lyric screen generator, 2023 ... Guidance screen generator, 2024 ... Image data processing unit, 2031 ... Camera operation controller, 2032 ... Moving operation controller, 2033 ... Lighting operation controller.

Claims (8)

Performance instruction data output means for outputting performance instruction data for instructing the performance of music;
Lyrics display instruction data storage means for storing lyrics display instruction data for instructing display of lyrics in association with timing data indicating the timing at which the lyrics should be displayed in the progression of the music;
Processing instruction data for instructing processing to be executed by at least one of an imaging device, a processing device that processes image data, a moving device that moves the imaging device, and a lighting device is executed in the progress of the music. Processing instruction data storage means for storing in association with timing data indicating the timing to be performed;
In parallel with the performance instruction data output by the performance instruction data output means, processing instruction data output means for outputting the processing instruction data at a timing according to timing data associated with the processing instruction data;
In parallel with the output of the performance instruction data by the performance instruction data output means and the output of the processing instruction data by the processing instruction data output means, the lyrics indicated by the lyrics display instruction data are associated with the lyrics display instruction data. Display at a timing according to the timing data associated with the processing instruction data, and display at a timing according to the timing data associated with the processing instruction data. A data processing device comprising: display instruction data output means for outputting display instruction data for instructing. Performance instruction data output means for outputting performance instruction data for instructing the performance of music;
Lyrics display instruction data storage means for storing lyrics display instruction data for instructing display of lyrics in association with timing data indicating the timing at which the lyrics should be displayed in the progression of the music;
Current state of at least one of the position of the imaging device, the direction of the imaging device, the processing of the imaging device, the position of the lighting device, the direction of the lighting device, the processing of the lighting device and the processing of the processing device that processes the image data State data input means for receiving state data indicating a change in state scheduled within a predetermined period;
In parallel with the output of the performance instruction data by the performance instruction data output means, the lyrics indicated by the lyrics display instruction data are displayed at a timing according to the timing data associated with the lyrics display instruction data, and A display instruction data output means for outputting display instruction data for instructing to display guidance according to the state data received by the state data input means. The display instruction data output means includes, as the display instruction data, a character, a figure, a symbol, or a combination thereof, indicating an imaging device or a lighting device that is executing a specific process or is scheduled to execute a specific process within a predetermined period. The data processing apparatus according to claim 1 or 2, wherein data instructing display of combination of these and color or display by lighting of a lamp is output. The display instruction data output means instructs the display instruction data to display different modes depending on the current state of the imaging device, processing device, moving device, or lighting device or a state scheduled after a predetermined time. The data processing apparatus according to claim 1, wherein: Position data input means for receiving position data indicating the position of the user,
The display instruction data output means outputs, as the display instruction data, data instructing simultaneous display of a position indicated by the position data, in addition to the lyrics and the guidance. The data processing apparatus described. Comprising image data input means for receiving user image data;
The display instruction data output means outputs, as the display instruction data, data instructing simultaneous display of an image indicated by the image data in addition to the lyrics and the guidance. The data processing apparatus described. A process of outputting performance instruction data for instructing the performance of the music;
A process of storing lyrics display instruction data instructing display of lyrics in association with timing data indicating a timing at which the lyrics should be displayed in the progress of the music;
Processing instruction data for instructing processing to be executed by at least one of an imaging device, a processing device that processes image data, a moving device that moves the imaging device, and a lighting device is executed in the progress of the music. Processing associated with timing data indicating the timing to be performed;
In parallel with the output of the performance instruction data, an output process for outputting the processing instruction data at a timing according to timing data associated with the processing instruction data;
In parallel with the output of the performance instruction data and the output of the processing instruction data, the lyrics indicated by the lyrics display instruction data are displayed at a timing according to the timing data associated with the lyrics display instruction data. Causing the computer to execute a process of outputting display instruction data instructing to display guidance corresponding to the process instruction data output in the output process at a timing corresponding to the timing data associated with the process instruction data. A program characterized by that. A process of outputting performance instruction data for instructing the performance of the music;
A process of storing lyrics display instruction data instructing display of lyrics in association with timing data indicating a timing at which the lyrics should be displayed in the progress of the music;
Current state of at least one of the position of the imaging device, the direction of the imaging device, the processing of the imaging device, the position of the lighting device, the direction of the lighting device, the processing of the lighting device and the processing of the processing device that processes the image data An input process for receiving state data indicating a state change scheduled within a predetermined period;
In parallel with the output of the performance instruction data, the lyrics designated by the lyrics display instruction data are displayed at a timing corresponding to the timing data associated with the lyrics display instruction data and received in the input process A program for causing a computer to execute a process of outputting display instruction data instructing to display guidance according to data.

Images