JP2012027340A - Karaoke apparatus and method of outputting still picture of karaoke singer - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve entertainment by surely extracting a still picture at a timing of excitement of users of a karaoke apparatus.SOLUTION: A karaoke apparatus 10 includes: a speaker 108 which reproduces music data for allowing a user A to sing; a display section 109 capable of displaying picture data with the reproduction of the music data by music reproduction means; and at least one omnidirectional camera 400 which captures a view of a prescribed range including users B and C other than the user A and the user A to generate picture data of the prescribed range. This karaoke apparatus calculates the number of other users B and C whose faces are turned to the direction of the user A based on the picture data of the prescribed range generated by the omnidirectional camera 400 and determines the timing of the excitement of the users A, B and C by time-shift of the number of persons calculated.

Description

  The present invention relates to a karaoke apparatus that provides a karaoke performance reproduction service, and a karaoke singer's still image output method for outputting a karaoke singer's still image in the karaoke apparatus.

  In the karaoke apparatus, in addition to the reproduction of the karaoke performance music, various services accompanying this have already been proposed. For example, a karaoke apparatus that shoots a singer's appearance with a single video camera and displays it on a display has been proposed in the past (for example, see Patent Document 1). This conventional karaoke device receives one video camera, a servo head for freely changing the orientation of the video camera within a predetermined range, and a radio beacon signal from a wireless microphone of the singer. Drive control means for driving the servo head so that the direction of the video camera is aligned with the position of the source of the radio signal.

  In recent years, in order to improve the entertainment, the video during the singing shot as described above is recorded and uploaded to the server, and the singer and other users can view the singing video (operation) on a personal computer or mobile phone. Services that can be viewed and enjoyed from a terminal have already been provided. Normally, in such a service, a still image of one scene in each moving image is appropriately extracted and displayed as a thumbnail on a web page for selecting a moving image to be browsed.

  A still image to be selected as a thumbnail from uploaded videos is originally a still image at a timing when the group of users including a singer is most excited in the sense of a representative image representing the content of the video. As such a technique for selecting and extracting a specific still image from a moving image, for example, methods described in Patent Document 2 and Patent Document 3 have already been proposed. In these conventional techniques, a place where the volume is high is determined as the rising timing, and a still image at the rising timing is extracted.

JP-A-10-240276 Japanese Patent No. 4198331 Patent No. 4435130

  However, when the techniques described in Patent Document 2 and Patent Document 3 are applied to the selection of a still image in the karaoke video, since the climax timing is determined based on the volume, the viewer in the recorded karaoke video can There is a possibility that it will be determined as a climax timing even when the chatter is chatting loudly without listening to the song of the singer. Therefore, it has been difficult to accurately determine the climax timing of the user of the karaoke apparatus and to reliably obtain an optimal still image at the timing.

  An object of the present invention is to provide a karaoke device and a method for outputting a karaoke singer's still image, which can improve entertainment by reliably extracting a still image at a rising timing of a user of the karaoke device.

  In order to achieve the above object, the first invention is a karaoke apparatus that provides a reproduction service of karaoke performance music using music data and video data, and reproduces the music data for singing by a singer. The music reproduction means, the display means capable of displaying the video data in accordance with the reproduction of the music data by the music reproduction means, a viewer other than the singer, and a predetermined range including the singer The face is directed to the singer by at least one video camera that captures the field of view and generates video data of the predetermined range and the video data of the predetermined range generated by the video camera. The calculating means for calculating the number of viewers, and the time of the number of persons calculated by the calculating means, the group including the singers and the viewers And having a determining means for determining a gully timing.

  In the first invention of the present application, in order to improve amusement, a swell of a group of singers and viewers is detected. First, at least one video camera captures a predetermined range of visual field including a singer and a viewer, and generates video data of the predetermined range. Then, based on the generated video data, the calculation means calculates the number of viewers whose faces are facing the singer.

  Here, the time when the group is excited is when the audience is enjoying the song with the singing of the singer, and at that moment, the viewer's line of sight is usually facing the singer. It is. Therefore, in the first invention of the present application, the determining means determines the excitement timing based on the temporal transition of the number of viewers calculated by the calculating means. Thereby, the moment with the largest number of viewers whose faces are facing the singer can be determined as the climax timing of the group. As a result, for example, out of the generated video data, a still image at the rising timing is extracted and uploaded to the server as a representative image of the service provision time, or displayed as a representative image on the display unit of the karaoke device, It is possible to provide various services. Thereby, the entertainment property of a karaoke apparatus can further be improved.

  A second invention is the microphone according to the first invention, which is possessed by the singer and for inputting a voice signal of karaoke singing by the singer, and a sign signal generating means which is provided in the microphone and generates a sign signal. And one moving image shooting camera is provided, and the entire range around the field of view of the predetermined range including the microphone and the singer is photographed, and the sign signal generator generates the sign signal. The video data of the predetermined range including a sign signal is generated, and the calculating means includes the microphone and the song based on the sign signal included in the video data of the predetermined range generated by the one video camera. The position of the performer is specified, and predetermined face recognition processing is performed on video data in a range other than the position of the singer in the predetermined range. Wherein by determining the orientation of the viewer's face, face and calculates the number of the viewer facing the direction of the singer Te.

  In the second invention of the present application, during karaoke performance, the music data is reproduced by the music reproducing means and the video data is displayed by the display means, and the singer sings with the microphone in accordance with the reproduction and display. Further, only one moving image shooting camera is provided. At the time of singing, the one video camera captures a predetermined field of view including the singer and the microphone, and generates video data of the predetermined range. At this time, by using a video camera with a wide field of view (for example, a camera equipped with a fish-eye lens and capable of shooting 360 ° around the entire circumference) capable of shooting the entire circumference centering on its own device, all the singers and viewers can It is always included in the video data generated by the video camera.

  Here, the microphone is provided with a sign signal generating means for generating a sign signal. Therefore, a sign signal corresponding to the position of the microphone possessed by the singer is always recorded together with the appearance of the singer in the generated video data in the predetermined range. Accordingly, in response to this, the calculation means specifies the position of the microphone and the singer using the sign signal included in the video data, and performs face recognition processing on the video data other than the singer's position for viewing. The orientation of the person's face. Accordingly, the number of viewers whose faces are facing the singer can be reliably calculated without using a plurality of cameras.

  According to a third aspect of the present invention, in the second aspect of the present invention, at least one partial still image including the singer at the climax timing determined by the determining means is selected from the predetermined range of video data with respect to the center of the entire circumference. A video processing unit that cuts out a fan-shaped arc and corrects the fan-shaped subarc into a straight line, corrects the fan into a quadrangle, and forms a single still image; and Still image output means for outputting a still image to a server connected to the karaoke apparatus over a network.

  When generating a still image at the rising timing determined by the determining means, the still image obtained from the video data of the video camera with a wide field of view as described above may be distorted toward the end of the field of view. . Therefore, in the third invention of the present application, the video processing means cuts out a partial still image including the singer at the rising timing from the video data in the predetermined range into a fan shape having a subarc with respect to the center of the entire circumference. A predetermined correction process is performed on the cut-out partial still image. As the correction process, the cut out arc-shaped subarc is corrected to a straight line, and the fan-shape is made a quadrangle. Thus, a normal still image in which the distorted state is corrected can be output to the server as a representative image by the still image output means. As a result, each user who accesses the server can view and enjoy the still image.

  In a fourth aspect of the present invention, in the third aspect of the present invention, a determination is made as to whether a predetermined face recognition process is performed on the at least one still image corrected by the video processing means and the singer's face can be recognized. The still image output means includes a still image determined by the determination means to be able to recognize the singer's face, out of the at least one still image corrected by the video processing means. It outputs to the said server, It is characterized by the above-mentioned.

  There is a case where the singer happens to face downward or sideways at the rising timing when the group of singers and viewers is rising. A still image at such a timing is not necessarily preferable as a representative image because the singer's face is not correctly recorded. Accordingly, in the fourth invention of the present application, the determination means determines whether or not the singer's face can be recognized for each of the at least one still image after the correction processing. Then, the still image output means outputs the still image determined to be able to recognize the singer's face to the server. Thereby, only the still image in which the singer's face is correctly recorded can be reliably output to the server as a representative image.

  5th invention has the singer ID input means which inputs singer ID from the outside in the said 4th invention, The said video processing means is further the said specified singer from the video data of the said predetermined range. The partial image data including the position of the singer is cut out and subjected to a predetermined correction process, and the still image output means recognizes the singer ID input from the singer ID input means and the singer's face by the determination means. The still image determined to be possible and the partial video data after the correction processing by the video processing means are associated with each other and output to the server connected to the karaoke apparatus via a network.

  Thereby, the still image in which the singer's face is reflected, the singer ID of the singer, and the video sung by the singer are uploaded to the server. As a result, the singer and each other user can search and browse the still image using the singer ID, or browse and enjoy the corresponding singing video.

  In order to achieve the above object, a sixth aspect of the invention is for generating and outputting a still image including a singer of a karaoke performance piece, which is executed by a computer provided in the karaoke apparatus for reproducing the karaoke performance piece. A method for outputting a still image of a karaoke singer, and acquiring a predetermined range of video data including a viewer other than the singer and the singer, which is captured and generated by at least one video camera; , A calculation procedure for calculating the number of viewers whose face is facing the singer by the video data in the predetermined range acquired in the acquisition procedure, and the time of the number of persons calculated in the calculation procedure From the determination procedure for determining the climax timing of the group including the singer and the viewer, and the predetermined range of video data acquired in the acquisition procedure, An extraction procedure for extracting at least one still image including the singer at the rising timing determined by the determination procedure, the still image extracted by the extraction procedure, or a still image after correcting the still image Is output to a server connected to the karaoke apparatus.

  In the sixth invention of the present application, at least one video camera captures a predetermined range of visual field including a singer and a viewer, and generates video data of the predetermined range. Based on the generated video data, the calculating means calculates the number of viewers whose faces are facing the singer. The time when the group is excited is when the viewer is enjoying the singing of the singer, so at that moment the viewer's line of sight is facing the singer. Therefore, the determination means determines the excitement timing based on the temporal transition of the number of viewers calculated in the calculation procedure. Thereby, the moment with the largest number of viewers whose faces are facing the singer can be determined as the climax timing of the group. As a result, among the video data acquired by the acquisition procedure, a still image at the rising timing is extracted by the extraction procedure, and the extracted still image (or a still image obtained by correcting the extracted still image) is represented as a representative image of service provision time. Can be uploaded to the server in the output procedure. As a result, the entertainment of the karaoke apparatus can be further improved.

  ADVANTAGE OF THE INVENTION According to this invention, the still image of the rise timing of the user of a karaoke apparatus can be extracted reliably, and amusement can be improved.

It is a figure which represents roughly the karaoke room in which the karaoke apparatus of one embodiment of this invention was installed. It is a side view showing the external appearance of a microphone. It is a functional block diagram showing the whole structure of the karaoke system provided with the karaoke apparatus. It is explanatory drawing showing the process of obtaining the moving image data of a singer by image-processing the image | video input from the perimeter camera. It is a figure which represents typically the image acquired when the inside of a karaoke room is image | photographed with the all-around camera. It is another figure which represents typically the image acquired when the inside of a karaoke room is image | photographed with the all-around camera in the state where one viewer does not look at a singer. It is a figure which represents roughly the karaoke room in the state where one viewer does not look at a singer. It is explanatory drawing which shows an example of a timing log. It is a flowchart showing the detail of the process sequence performed by the control part of an apparatus main body. It is a flowchart showing the detail of the procedure of a log creation process. It is a figure showing the example of a display of the thumbnail uploaded to the host server. It is a flowchart showing the detail of the process sequence performed by the control part of an apparatus main body in the modification which extracts the still image in which a singer's face is reflected. 12 is a flowchart showing details of a processing procedure executed by a control unit of the apparatus main body in a modification example in which extraction is performed in consideration of a sound level.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram schematically showing a karaoke room in which the karaoke apparatus of the present embodiment is installed.

  In FIG. 1, a karaoke apparatus 10 is installed in a karaoke room KR such as a karaoke store. The karaoke device 10 is a device that provides a karaoke performance reproduction service using Musical Instrument Digital Interface (MIDI) data and video data as music data. In the one shown in FIG. 1, users A to C perform karaoke singing. The karaoke apparatus 10 includes an apparatus main body 100 called a commander, a remote controller 200, a microphone (hereinafter abbreviated as a microphone) 300, and a single all-around camera 400 (moving picture camera). The apparatus main body 100, the remote controller 200, and the microphone 300 will be described in detail later.

  In this example, the all-around camera 400 is provided downward in the center of the ceiling of the karaoke room KR, for example. The all-around camera 400 captures a fixed range of visual field including the microphone 300 and the singers A to C, and generates video data of the range. Specifically, the omnidirectional camera 400 has a fisheye lens, and obtains video data in which a hemispherical field of 360 degrees on the left and right and 90 degrees on the upper and lower sides is centered on the camera 400. Due to the function of the fisheye lens, the image in the field of view of the omnidirectional camera 400 has a property that a closer object appears in the center of the circle and a farther object appears in the periphery of the circle. In addition, as a price for obtaining a wide viewing angle with a fisheye lens, all objects are distorted in a fan shape. Therefore, when the entire karaoke room KR is photographed by the all-around camera 400, an image distorted in a fan shape as a whole is acquired.

  FIG. 2 is a diagram illustrating the appearance of the microphone 300. In FIG. 2, the microphone 300 is for converting the voice of karaoke singing by the user into an audio signal and inputting it.

  The microphone 300 has a microphone housing 302 in which a microphone element 301 is built. A power switch 303 is provided on the top of the microphone housing 302. At the bottom of the microphone housing 302, a light emitting diode (LED) 304 serving as a marker signal generating unit that emits marker light as a marker signal of a predetermined color (in this example, green), and a green light emitted from the LED 304 are displayed. A translucent light diffusion sphere 305 that uniformly diffuses the marker light is provided.

  FIG. 3 is a functional block diagram showing the overall configuration of a karaoke system including the karaoke apparatus 10 described above.

  In FIG. 3, the karaoke system 1 includes the karaoke apparatus 10 installed in the karaoke room KR and a host server 20. The karaoke apparatus 10 and the host server 20 are connected to each other so as to be able to transmit and receive information via a network NW such as a communication network.

  The karaoke apparatus 10 includes the apparatus main body 100, the remote controller 200, the microphone 300, and the all-around camera 400. The apparatus main body 100 and the remote controller 200 are connected to each other so as to be able to transmit / receive information to / from each other via a network such as a wireless or wired LAN. The apparatus main body 100 and the microphone 300 are connected by a wireless line or a wired line.

  The apparatus main body 100 includes a control unit 101, a mass storage device 103, an operation unit 104, a reception unit 105, a sound source 106, a sound control unit 107, a speaker 108, a display unit 109, and a communication control unit 110. And have.

  The control unit 101 includes a CPU (not shown) and a memory such as a RAM and a ROM. The control unit 101 executes various programs stored in advance in the ROM or the large-capacity storage device 103 while using the temporary storage function of the RAM. Thereby, the entire apparatus main body 100 is controlled.

  In particular, the control unit 101 performs predetermined image processing on the captured image in the karaoke room KR obtained by the omnidirectional camera 400, and a plurality of images of a singer with the microphone 300 (moving images as an aggregate). A plurality of still images to be configured (the same applies hereinafter) is generated, and the image is stored in the mass storage device 103 and displayed on the display unit 109 (details will be described later).

  The mass storage device 103 is composed of, for example, a hard disk drive (HDD). The mass storage device 103 stores various information such as MIDI data, background video data, and lyrics data. Also, the large-capacity storage device 103 sequentially stores moving image data when the user sings.

  The operation unit 104 includes, for example, a plurality of keys and switches. The user can perform various operations such as a reservation operation for a karaoke performance using the operation unit 104 or an operation unit 204 of the remote controller 200 described later.

  The receiving unit 105 receives the singer's voice signal output from the microphone 300.

  The sound source 106 reproduces the MIDI data read from the mass storage device 103 by the control unit 101 and outputs it to the audio control unit 107. The audio control unit 107 amplifies the MIDI data output from the sound source 106 and the audio signal input from the microphone 300 via the receiving unit 105 and outputs the amplified signal to the speaker 108. The speaker 108 outputs the MIDI data and audio signal output from the audio control unit 107 as audio.

  Hereinafter, the sound source 106, the audio output unit 107, and the speaker 108 are appropriately omitted and referred to as “sound source 106 etc.”. The sound source 106 and the like constitute music reproducing means for reproducing music data.

  The display unit 109 is composed of a liquid crystal display, for example, and functions as display means for displaying various videos. In particular, the display unit 109 synchronizes with the reproduction of the MIDI data by the sound source 106 or the like, in other words, the background video read from the mass storage device 103 as the MIDI data is reproduced by the sound source 106 or the like. Data and telops corresponding to the lyrics data can be displayed.

  The communication control unit 110 controls information communication with the remote controller 200 and the host server 20.

  The remote controller 200 is an operation terminal for a user to perform various operations such as a reservation operation for a karaoke performance song. The remote controller 200 includes a control unit 201, a storage device 203, an operation unit 204, a display unit 209, and a communication control unit 210.

  The control unit 201 includes a memory such as a CPU, RAM, and ROM (not shown). The control unit 201 executes various programs stored in advance in the ROM or the storage device 203 while using the temporary storage function of the RAM. As a result, the entire remote controller 200 is controlled.

  The storage device 203 is composed of, for example, a nonvolatile memory and stores various types of information. The operation unit 204 is composed of, for example, a plurality of keys and switches. Using the operation unit 204 or the operation unit 104 of the karaoke apparatus 100, the user can perform various operations such as a reservation operation for karaoke performance songs. The display unit 209 is composed of, for example, a liquid crystal display and performs various displays.

  The communication control unit 210 controls information communication with the apparatus main body 100 and the host server 20.

  The host server 20 can upload the moving image data of the user's singing as a compressed moving image file (details will be described later). The moving image data uploaded to the host server 20 can be viewed from a specific user terminal on a predetermined Web page (see also FIG. 11 described later).

  Here, as one of the features of the present embodiment, the position of the microphone 300 is specified based on the marker light included in the video image in the karaoke room KR obtained by the all-around camera 400, and the position of the microphone 300 is determined. The included partial video is cut out and the video (karaoke post video) data of the singer with the microphone 300 is acquired. At this time, the video signal obtained by imaging with the omnidirectional camera 400 is significantly different from the normal visual field of human beings, and therefore cannot be used as it is for the karaoke post movie. For this reason, it is necessary to perform a predetermined process on the video signal obtained by imaging with the omnidirectional camera 400.

  FIGS. 4A to 4F are explanatory diagrams showing a process of obtaining video data of a singer by performing image processing on the video input from the all-around camera 400. This process is executed by the control unit 101 of the apparatus main body 100.

  As shown in FIG. 4A, first, an image in the karaoke room KR acquired by the all-around camera 400 is input. Here, only the images of the users A to C and the microphone 300 are shown, and the images of the table, the apparatus main body 100, and the like are omitted.

  Thereafter, as shown in FIG. 4B, color filter processing is performed to remove components other than a component of a predetermined color (in this example, green) in the video input from the all-around camera 400. Specifically, only the green component is extracted from the video input from the all-around camera 400 through a color filter (here, a green filter). The color filter is a pixel data file obtained by CPU calculation processing that passes only the G values of RGB or passes only the pixel values of YUV UV within a certain range.

  Thereafter, as shown in FIG. 4C, the image data that has been subjected to color filter processing is passed through a luminance filter, and only pixel values that have a luminance of a certain level or higher are passed, thereby binarizing the image data. I do. Thereby, only a pixel “close to pure green and having a certain level of brightness” in the image indicates “1”, and other pixels indicate “0”.

  Thereafter, as shown in FIG. 4D, area determination is performed on the video input from the all-around camera 400. Specifically, according to a map defined in the shape of a spider web in advance, the angle direction (corresponding to left and right in human vision) is divided into 8 (A to H) and the distance direction (in human visual depth). All the pixel values of the bitmap are added for each area for each area that has been divided into a total of 28 (three) (1-3) or four (1-4). The value with the largest added value (area G2 in the figure) detects the marker light and is determined to be the area (direction) where the singer to be photographed is.

  Thereafter, as shown in FIG. 4E, the shape of the image is corrected by fan scan for the area selected in the process shown in FIG. Specifically, a process of rearranging the pixels in the area according to the scanning line shown in FIG. As a result, the fan-shaped diameter direction is deformed and corrected to a Y-axis rectangular shape and the circumferential direction is an X-axis rectangular shape. Here, the scanning line close to the central part is short and the scanning line close to the circumferential part is long, but it is supplemented with pixels of the same value and enlarged, or the pixels are omitted and reduced to obtain line data of a fixed length. . A scanning line coordinate pattern is prepared in advance for each of the areas A1 to H4.

  When the conversion process is completed for all the scanning lines, a final output image (an image of area G2 in the drawing) as shown in FIG. 4F is obtained. This image is obtained by cutting out only the peripheral region of the marker light captured by the all-round camera 400 and converting and correcting the circular viewing angle of the curved fisheye lens into a normal rectangular viewing angle. In other words, a sector having a subarc arc with respect to the center of the entire circumference is cut out, and the sector is corrected to a quadrangle by correcting the cut out arc of the sector into a straight line. Since it is such correction, as a result, a result equivalent to that when a normal camera is pointed at the singer (user A in this example) having the microphone 300 is obtained. In this embodiment, the continuous correction | amendment image | video which contains the user A who is a singer is acquired by performing the said conversion correction | amendment with respect to the image | video while a singer (user A in this example) is singing. Memorized.

  Since these processes are performed on each image (still image) constituting the moving image, the process is performed at a rate of 30 frames per second. However, for example, at a frequency of 1 frame per 10 frames due to calculation capability. Thinning may be performed such as processing.

  In this embodiment, after specifying the position of the singer (user A in this example) as described above, other users (users B and C in this example) look at the singer. The rising timing of the group consisting of these users A, B, and C is detected depending on whether or not it is directed.

  That is, in the above description, the processing for the image of the area G2 has been described as an example, but the other 27 areas, that is, the areas A1 to A3, B1 to B3, C1 to C4, D1 to D4, E1 to E3. , F1 to F3, G1, G3, G4, and H1 to H4, it is possible to obtain an image equivalent to a case where a normal camera is directed to each area by the same method. As a result, an image of a predetermined range around the user A who is a singer (for example, a range excluding areas C4, D4, G4, and H4 corresponding to the four corners of the karaoke room KR, where it is difficult for humans to exist) is obtained. It is possible to detect whether there is another user (viewer) within the predetermined range by using a known face recognition technique or the like. In this example, the presence of the user B and the user C is recognized using the face recognition technology described above. FIG. 5 conceptually shows a state where an image of a predetermined range around the singer (user A) is acquired in this way.

  And in this embodiment, users other than a singer (in this example, users B and C) using the image acquired about the predetermined range around a singer (user A) as shown in FIG. Detects whether or not the singer (user A) is turning his gaze. The example of FIG. 5 shows a state in which the faces of all the two users B and C are facing the user A and are looking toward the user A, as indicated by broken arrows. This corresponds to the state shown in FIG.

  On the other hand, for example, among the users B and C, the face of the user C is not facing the user A who is a singer, and only one user B is looking toward the user A. There may be certain conditions. Also in this case, the above-described state is recognized by acquiring a predetermined range of images around the singer (user A) as shown in FIG. The In the actual space, the state is as shown in FIG.

  As described above, in the present embodiment, when the user A is singing as a singer, the face is directed toward the user A at each timing every predetermined period (for example, several tens of msec). Detect the number of other users who are. The detected number of people is recorded in a timing log in time series. Then, the timing when the number of other users (viewers) who are turning their faces to the singer is the highest is determined to be the rising timing of this group.

  FIG. 8 is an explanatory diagram showing an example of the timing log used for determining the rising timing. As shown in FIG. 8, the timing log includes a “time” column indicating the time (in other words, recording time) at which each data is acquired, for example, in units of msec, and a song identified based on the marker light as described above. The “singer” field that represents the listener, the “viewer” field that represents the resident user other than the singer, that is, the viewer, identified by the face recognition, and the singer of the viewer is the face A column of “viewer facing the singer” indicating a viewer who is facing is provided as a record column. As illustrated, in this example, each data is recorded in time series from left to right.

  For example, while user A is singing, “A” is recorded in the “singer” column at all timings of the time range. At the timing when both the users B and C are present in the karaoke room KR where the user A sings, “B, C” is recorded in the “viewer” column. For example, only “C” is recorded when user B leaves the room to go to the toilet and only user C is present as a viewer.

  In addition, when the user A is singing, at the timing when both the users B and C are facing the user A, “B, C” is displayed in the “viewer facing the singer” column. Is recorded (corresponding to the states of FIGS. 5 and 1). On the other hand, user C is facing user A when user A is singing, but user B is facing the wall and not facing user A, for example. Only “C” is recorded in the “viewer facing the singer” column (corresponding to the state of FIGS. 6 and 7), and “none” is indicated when no one is facing the user A. To be recorded.

  In the example shown in FIG. 8, two users B and C are facing the user A who is a singer at a timing located in the center in the left-right direction in the figure. Therefore, it is determined that this timing is the “rising timing”.

  And as stated previously, in this embodiment, the image | video containing the said singer (user A in said example) while the singer is singing is continuously acquired and memorize | stored. The video thus stored, that is, the moving image is an aggregate of a large number of still images, but in this embodiment, the still image at the timing determined as the climax timing represents the video obtained by photographing the singer. Extracted as a thumbnail image.

  FIG. 9 is a flowchart showing details of a processing procedure executed by the control unit 101 in order to execute the above method.

  In FIG. 9, when the reproduction of the MIDI data corresponding to the karaoke performance music is started, this flow is started. That is, the background video data and the lyrics data are displayed on the display unit 109 in synchronization with the reproduction of the MIDI data by the sound source 106 or the like. Then, the singing of the karaoke performance music by the singer is performed, and the voice of the karaoke singing is input from the microphone 300.

  First, in step S10, video data in the karaoke room KR taken by the all-around camera 400 is acquired. This procedure constitutes the acquisition procedure described in each claim.

  Thereafter, in step S25, the position of the microphone 300 is specified based on the marker light included in the video data in the karaoke room KR. The processing in step S25 corresponds to the image processing shown in FIGS. 4 (a) to 4 (c). In step S30, partial video data including the position of the microphone 300 is cut out from the video data in the karaoke room KR. The process in step S30 corresponds to the area determination process shown in FIG.

  Thereafter, in step S <b> 35, the partial video data including the position of the microphone 300 is corrected, and an image showing the singer holding the microphone 300 is obtained. The processing in step S35 corresponds to the shape correction of the image shown in FIGS. 4 (e) and 4 (f). In step S40, the corrected image is displayed in a partial area of the display unit 109, and is stored in the mass storage device 103 in association with the photographing time.

  Thereafter, in step S100, a log creation process for creating the timing 6 described with reference to FIG. 8 is executed. FIG. 10 is a flowchart showing the detailed procedure of step S100.

  In FIG. 10, first, in step S110, video data within a predetermined range (a range excluding the four corners of the karaoke room KR in the above example) from the position of the singer is corrected. Note that this correction process only needs to execute the same correction process as the above-described step S35, and thus detailed description thereof is omitted.

  Thereafter, in step S120, a known face recognition process is performed on the video data in the predetermined range corrected in step S110 to detect the viewer's appearance.

  Then, the process proceeds to step S130, and the orientation of each viewer's face is detected for the viewer detected in step S120. For this detection, as described above, the face contour of each viewer and the position of the nose / mouth are specified by a known appropriate face recognition process and other image analysis processes, and the face orientation is determined in the karaoke room KR. It may be calculated which direction it is facing.

  Thereafter, the process proceeds to step S140, and the number of viewers facing the singer is counted based on the detection result in step S130. Then, the number of viewers present in the karaoke room KR detected in step S120, the number of viewers facing the counted singers, and the shooting time as data in the timing log. Record. Note that the created timing log is accumulated and stored in the mass storage device 103 so that it can be referred to. These steps S120, S130, and S140 constitute the calculation procedure described in each claim and function as calculation means. Thereafter, the process proceeds to step S45 (see FIG. 9).

  Returning to FIG. 9, in step S45, it is determined whether or not the reproduction of the MIDI data corresponding to the karaoke performance song has been completed. When the reproduction of the MIDI data corresponding to the karaoke performance song is completed, the determination at step S45 is satisfied, and the routine goes to step S50. On the other hand, when the reproduction of the MIDI data corresponding to the karaoke performance song is not finished, the determination in step S45 is not satisfied, and the process returns to step S10 and the same procedure is repeated. Thus, steps S10 to S40 and step S100 are repeated while the reproduction of the karaoke performance song is not completed, and the corrected video (moving image) is stored in the large capacity storage device 103 in step S40 every time step S40 is passed. The log creation processing in step S100 is executed for each image (still image) constituting the moving image. That is, step S10 to step S40 and step S100 are repeated for each image (still image).

  In step S <b> 50, the moving image posting instruction operation screen is displayed in a partial area of the display unit 109. In addition, the video posting instruction operation screen includes a display for prompting the input of the ID of the operator (user A who is a singer, but may be operated by the users B and C), that is, the singer ID. It is. Thereafter, the process proceeds to step S55.

  In step S55, it is determined whether posting of a karaoke video is instructed by the operator on the video posting instruction operation screen of the display unit 109. That is, a singer ID is input in response to the display prompting the input of the singer ID (for example, by the operation unit 104 of the apparatus main body 100 or the operation unit 204 of the remote controller 200), and an operation of an appropriate posting instruction button or the like. It is determined whether or not In addition, the said operation part 104 or the operation part 204 functions as a singer ID input means as described in each claim. When the posting of the karaoke video is instructed, the determination at step S55 is satisfied, and the routine proceeds to step S200. When the posting of the karaoke video is not instructed, the determination at step S55 is not satisfied, and this flow ends.

  In step S200, the column of “viewer facing the viewer” in the timing log (see FIG. 8) created by the log creation process in step S100 is referred to, and the song is directed toward the singer at each timing. Get the number of viewers who are. Then, the timing at which the number of persons is maximum is determined as the rising timing. This step S200 constitutes the determination procedure described in each claim and functions as a determination means.

  Thereafter, the process proceeds to step S210, and corresponds to the climax timing determined in step S200 from among a plurality of image data including the appearance of the singer stored in the mass storage device 103 in association with the shooting time in step S40. The obtained image (still image) is acquired, and the image is set as a thumbnail image (representative image). Step S210 and the above-described steps S30 and S35 function as video processing means described in each claim. Thereafter, the process proceeds to step S60.

  In step S60, moving image data using a plurality of output videos after the correction processing sequentially stored in the large-capacity storage device 103 in step S40 when the above-described steps S10 to S40 and step S100 are repeated, and acquired in step S210. The thumbnail image and the singer ID input in step S55 are uploaded to the host server 20 in a manner associated with each other. This step S60 constitutes the output procedure described in each claim and functions as a still image output means. When step S60 is completed, this flow ends.

  FIG. 11 shows a display example of the thumbnail uploaded to the host server 20 on the aforementioned predetermined web page. In the display example shown in FIG. 11, the thumbnail image sung by a singer (for example, one of users A to C. In the above example, the sung date, the singing date and time (the above-mentioned excitement timing) Are displayed together with the number of times the karaoke video has been played, evaluation, etc. By clicking, for example, this thumbnail image (or the corresponding operation unit or the like) on the web page screen, all the videos of the singers uploaded in step S60 can be reproduced and viewed. In this example, after the browsing performed in this way, the evaluation by the browsing user with respect to the karaoke video (“delicious”, “interesting”, “cute”, “crying” is also written and displayed (detailed) (The illustration is omitted).

  As described above, in this embodiment, in order to improve the entertainment, a group of singers (user A in the above example) and viewers (users B and C in the above example) is excited. To detect. As mentioned above, when the group is excited, the audience is enjoying the song with their singing, and at that moment the audience's line of sight is facing the singer. Is normal. Therefore, face recognition processing is performed on the video data in the karaoke room KR generated based on the photographing result of the all-around camera 400, and “viewer facing the singer” at each timing based on the recognition result. Are recorded as a timing log (see step S100 in FIGS. 8 and 9). Then, the rising timing is determined based on the temporal transition of the number of persons (see step S200). Thereby, the moment with the largest number of viewers whose faces are facing the singer can be determined as the climax timing of the group. As a result, a still image at the rising timing is extracted from the video data generated by shooting with the all-around camera 400 (see step S210) and uploaded to the host server 20 as a representative image of the service providing time. Yes (step S60). As a result, each user who has accessed the host server 20 can view and enjoy the still image (see FIG. 11). This thumbnail can also be displayed on the display unit 109 of the karaoke apparatus 10. By performing such various services, the entertainment of the karaoke apparatus 10 can be further improved.

  In the present embodiment, in particular, one omnidirectional camera 400 captures a predetermined range of visual field including the singer and the micro 300 and generates video data of the predetermined range. At this time, by using the omnidirectional camera 400 having a wide field of view capable of photographing the entire circumference around the camera, the singer (user A in the above example) and the viewers (the above users B and C) can be obtained. All of them are always included in the video data generated by the camera. In the generated video data in the predetermined range, marker light corresponding to the position of the microphone 300 possessed by the singer is always recorded together with the appearance of the singer. Therefore, the positions of the microphone 300 and the singer are specified using the marker light included in the video data (see step S25 in FIG. 9), and the face recognition process is performed on the video data other than the singer's position, thereby allowing the viewer Is determined (see step S120 and step S130 in FIG. 10). Accordingly, the number of viewers whose faces are facing the singer can be reliably calculated with only one of the all-round cameras 400 without using a plurality of cameras.

  Here, when the still image is generated, the still image obtained from the video data of the omnidirectional camera 400 having a wide field of view as described above is distorted toward the end of the field of view. In the present embodiment, in particular, in step S35 of FIG. 9, the partial still image including the singer is cut out from the video data in the predetermined range into a fan shape having a subarc with respect to the center of the entire circumference, and the cut out partial still image is included. A predetermined correction process is performed on the image (as described above, the cut arc-shaped subarc is corrected to a straight line so that the fan shape is a square). As a result, a thumbnail at a rising timing composed of a normal still image in which the distorted state is corrected can be output to the host server 20 as a representative image in step S60.

  The present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the spirit and technical idea of the present invention. Hereinafter, such modifications will be described.

(1) When extracting a still image in which a singer's face is reflected For example, when a singer happens to face down or to the side at a climax when the group of singers and viewers is rising There is. A still image at such a timing is not necessarily preferable as a representative image because the singer's face is not correctly recorded. This modification is for dealing with such a case.

  FIG. 12 is a flowchart showing details of the processing procedure executed by the control unit 101 in this modification, and corresponds to FIG. 9 described above. Steps equivalent to those in FIG. 9 are denoted by the same reference numerals, and description thereof is omitted or simplified.

  In FIG. 12, in this modification, step S210 in FIG. 9 is omitted, and step S220, step S230, step S240, step S250, and step S260 are newly provided.

  The processes in steps S10 to S40 and S200 are the same as those in FIG. When step S200 ends, the process proceeds to newly provided step S220.

  In step S220, it is determined whether or not there are a plurality of rising timings based on the determination result of the rising timing in step S200. That is, in the timing log, it is determined whether or not the number of viewers facing the singer is the same number at a plurality of timings.

  If the rising timing is limited to one and there are not a plurality of timings, the determination at step S220 is not satisfied, and the routine goes to step S250. In step S250, an image (still image) corresponding to the one climax timing is selected from a large number of image data including the appearance of the singer stored in the mass storage device 103 in association with the shooting time in step S40. And the image is used as a thumbnail image. Thereafter, the process proceeds to step 60.

  On the other hand, if there are a plurality of climax timings at step S220, the determination at step S220 is satisfied, and the routine goes to step S230. In step S230, an image (still image) corresponding to the plurality of climax timings is selected from a large number of image data including the appearance of the singer stored in the mass storage device 103 in association with the shooting time in step S40. get. Then, it is determined whether there is any one that can recognize the singer's face by using a known face recognition method among the plurality of still images. In addition, this step S220 functions as determination means described in each claim.

  If there is something that can recognize the singer's face, the determination in step S230 is satisfied, and the process proceeds to step S240, where the recognizable still image is set as a thumbnail image. If there are a plurality of still images in which the singer's face can be recognized, any one of them may be used as a thumbnail. Thereafter, the process proceeds to step 60.

  On the other hand, in step S230, when there is no still image which can recognize a singer's face, it moves to step S260. In step S260, any one of the plurality of still images acquired in step S230 is set as a thumbnail. Thereafter, the process proceeds to step S60. Incidentally, in response to the fact that the singer's face could not be recognized, in this step S260, a thumbnail is not set, a display signal is output to the display unit 109, an appropriate error display is performed, and this flow is terminated. It may be. Even if there is only one climax timing and the determination in step S220 is not satisfied, the same determination as in step S230 is performed. If the determination is not satisfied, a thumbnail is not set and an error is generated as described above. It is good also as a display.

  Note that the processing in step S60 is the same as that in FIG. Further, Step S250, Step S240, Step S260 and Step S30 and Step S35 described above function as the video processing means described in each claim.

  In this modification, when a thumbnail is set and output to the host server 20, a still image in which the singer's face is correctly recorded can be used as a thumbnail.

(2) When extracting in consideration of the audio level For example, there is a case where the singer does not sing by chance at the exciting timing when the group of singers and viewers is excited. That is, when the viewer pays attention all at once due to gestures, gestures, facial expressions, etc. that are not singing by the singer, this is a case where excitement occurs due to actions other than singing by the singer. Since the still image at such timing does not sing, it is not necessarily preferable as a representative image of the karaoke song moving image. This modification is for dealing with such a case.

  FIG. 13 is a flowchart showing details of a processing procedure executed by the control unit 101 in this modification, and corresponds to FIG. 9 and FIG. The same steps as those in FIG. 12 are denoted by the same reference numerals, and description thereof is omitted or simplified.

  In FIG. 12, in this modification, steps S300 and S310 are provided instead of steps S230 and S240 in FIG.

  That is, as shown in FIG. 13, when the determination in step S220 is satisfied, the process proceeds to step S300. In step S300, an image (still image) corresponding to the plurality of climax timings from among a large number of image data including the appearance of the singer stored in the mass storage device 103 in association with the shooting time in step S40. To get. Then, it is determined whether among the plurality of still images, there is an audio level from the microphone 300 at the timing that is less than a predetermined value. That is, in this modification, each image data stored in the large-capacity storage device 103 is stored in a form associated with the sound level of the microphone 300 at that time in advance.

  If there is an audio level that is less than the predetermined value, the determination in step S300 is satisfied, and the process moves to step S310, and a still image when the audio level is equal to or higher than the predetermined value is set as a thumbnail image. When there are a plurality of still images having an audio level equal to or higher than a predetermined value, any one of them may be used as a thumbnail. Thereafter, the process proceeds to step 60.

  On the other hand, if there is no audio level lower than the predetermined value in step S300, the process proceeds to step S260, and as described above, any one of the plurality of still images acquired in step S230 is selected. A thumbnail. Thereafter, the process proceeds to step S60.

  Note that the processing in step S60 is the same as that in FIG. Further, Step S250, Step S310, Step S260 and Step S30 and Step S35 described above function as video processing means described in each claim.

  In this modification, when a thumbnail is set and output to the host server 20, a still image in which a state where the singer actually sings is recorded can be used as a thumbnail.

(3) Others In the above, the case where photographing is performed using one omnidirectional camera 400 has been described as an example, but the present invention is not limited to this. That is, a plurality of cameras may be used, and a singer can be photographed with a singer photographing camera, and an indoor viewer may be photographed with another viewer photographing camera arranged for photographing the viewer. In any case, it is sufficient to detect the number of viewers heading toward the singer.

  The flowcharts shown in FIG. 9, FIG. 10, FIG. 12, FIG. 13, etc. are not intended to limit the present invention to the procedure shown in the above flow, and additional procedures / additions can be made without departing from the spirit and technical idea of the invention. You may delete or change the order.

  In addition to those already described above, the methods according to the above-described embodiments and modifications may be used in appropriate combination.

  In addition, although not illustrated one by one, the present invention is implemented with various modifications within a range not departing from the gist thereof.

10 Karaoke apparatus 106 Sound source (music playback means)
107 Voice control unit (music playback means)
108 Speaker (music playback means)
109 Display section (display means)
200 remote control 300 microphone (microphone)
304 LED (sign signal generating means)
400 All around camera (video camera)
A user (singer)
B User (viewer)
C User (viewer)

Claims (6)

A karaoke apparatus that provides a reproduction service of karaoke performance music using music data and video data,
A music reproducing means for reproducing the music data for the singer to sing;
Display means capable of displaying the video data as the music data is played back by the music playback means;
At least one video camera that captures a predetermined field of view including a viewer other than the singer and the singer, and generates video data of the predetermined range;
Calculating means for calculating the number of viewers whose face is facing the singer by the predetermined range of video data generated by the video camera;
Determining means for determining a climax timing of a group including the singer and the viewer by temporal transition of the number of persons calculated by the calculating means;
A karaoke apparatus comprising: The karaoke apparatus according to claim 1,
A microphone that is possessed by the singer and for inputting an audio signal of karaoke singing by the singer;
A sign signal generating means for generating a sign signal provided in the microphone;
One moving image shooting camera is provided,
The entire field of view of the predetermined range including the microphone and the singer is imaged all around the device, and the video data of the predetermined range including the marker signal generated from the marker signal generating means is generated,
The calculating means includes
The position of the microphone and the singer is specified based on the marker signal included in the video data of the predetermined range generated by the one video camera, and the position other than the position of the singer within the predetermined range The number of viewers whose faces are facing the singers is calculated by performing a predetermined face recognition process on video data in the range to determine the orientation of the viewer's face. Karaoke device to do. The karaoke apparatus according to claim 2,
At least one partial still image including the singer at the rising timing determined by the determining means is cut out from the video data in the predetermined range into a fan shape having a subarc with respect to the center of the entire circumference, and the cut-out Video processing means for correcting the sector to a square by correcting the fan-shaped subarc to a straight line,
A karaoke apparatus comprising: a still image output means for outputting the still image to a server connected to the karaoke apparatus over a network. The karaoke apparatus according to claim 3,
A determination unit that performs a predetermined face recognition process on the at least one still image corrected by the video processing unit and determines whether the singer's face can be recognized;
The still image output means includes
A karaoke apparatus that outputs, to the server, a still image that is determined by the determining unit to be able to recognize the face of the singer among the at least one still image that has been corrected by the video processing unit. . The karaoke apparatus according to claim 4,
It has a singer ID input means for inputting a singer ID from the outside,
The video processing means further includes
From the video data of the predetermined range, cut out partial video data including the position of the specified singer, perform a predetermined correction process,
The still image output means includes
The singer ID input from the singer ID input means, the still image determined to be able to recognize the singer's face by the determination means, the partial video data after the correction processing by the video processing means, Are associated with each other and output to the server connected to the karaoke apparatus via a network. A karaoke singer still image output method for generating and outputting a still image including a singer of the karaoke performance song, which is executed by a computer provided in the karaoke apparatus for reproducing the karaoke performance song,
An acquisition procedure for acquiring video data of a predetermined range including a viewer other than the singer and the singer, which is captured and generated by at least one video camera;
A calculation procedure for calculating the number of viewers whose face is facing the singer by the video data in the predetermined range acquired in the acquisition procedure;
A determination procedure for determining a climax timing of a group including the singer and the viewers by temporal transition of the number of persons calculated in the calculation procedure,
An extraction procedure for extracting at least one still image including the singer at the excitement timing determined in the determination procedure from the video data in the predetermined range acquired in the acquisition procedure;
An output procedure for outputting the still image extracted in the extraction procedure or a still image after correcting the still image to a server connected to the karaoke device;
A method for outputting a still image of a karaoke singer characterized by comprising:

Images