JP5964722B2 - Karaoke system - Google Patents

Description

  The present invention relates to a karaoke system using a karaoke device that outputs a tune selected from a large number of tunes, and in particular, the convenience of imaging data obtained by photographing a user performing a karaoke performance with a portable terminal device. It is related with the improvement for improving.

  A video information distribution system that distributes video information to a plurality of communication terminal devices via a predetermined communication line is known, and further development is expected as the core of the content business in recent years as communication technology improves. . In such a video information distribution system, usually, a plurality of video information posted (uploaded) from a predetermined communication terminal device at a so-called video posting site is stored in a predetermined video database, and the video is distributed by an arbitrary communication terminal device. Access via the posting site is possible. Thereby, the video information posted by the predetermined communication terminal device can be browsed by any communication terminal device.

  By the way, the present applicant incorporates the karaoke device into the video information distribution system as a communication terminal device, and the video information captured by the imaging device and the voice input by the voice input device when the karaoke device outputs a predetermined piece of music. A technology that accepts and allows browsing of information has been devised and put into practical use. In addition, regarding this technology, a co-star video (collaboration video) that can be viewed by not only the karaoke device but also a communication terminal device such as a personal computer by editing a plurality of video information associated with a common music piece. ) Service. For example, the video information distribution system described in Patent Document 1 is an example. According to this technique, the state of the karaoke performance image | photographed with the imaging device with which the said karaoke apparatus was equipped can be delivered so that browsing is possible combining with the other moving image corresponding to the same music.

JP 2012-73379 A

  By the way, not all karaoke apparatuses are equipped with an imaging device, and there are karaoke apparatuses that are not equipped with an imaging device. When it is desired to shoot a karaoke performance by such a karaoke device and upload it to the video information distribution system or the like, for example, shoot the karaoke performance by an imaging device provided in a portable terminal device such as a so-called smartphone. It is possible to do. However, there is an adverse effect that it is not possible to specify which piece of karaoke performance is photographed in the image data captured by the imaging device provided in the portable terminal device. When the music corresponding to the imaging data cannot be specified, it is not possible to perform processing such as combining with another moving image corresponding to the same music as in the conventional technique. Such a problem has been newly found by the inventors in the process of continuing intensive research on the possibility of a karaoke system linked to a video information distribution system.

  The present invention has been made against the background of the above circumstances, and the purpose of the present invention is to improve the convenience of image data obtained by photographing a user performing a karaoke performance with a portable terminal device. Is to provide.

In order to achieve such an object, the gist of the present invention is that a karaoke device that performs performance output of a music selected from a large number of music, a portable terminal device including an imaging device, and a communication line A karaoke system comprising a server device that transmits and receives information to and from the portable terminal device through the karaoke device at the time of imaging data captured by the imaging device of the portable terminal device. Music specifying means for specifying the music for which performance output has been performed, and the image data captured by the imaging device of the portable terminal device are associated with the music specified by the music specifying means in the server device. an imaging storage means to be stored and an image corresponding to the user of the karaoke apparatus, a customer database for storing in association with identification information of the user, the customer data From the stored relationship over scan, on the basis of the image included in the imaging data taken by an imaging device of a portable terminal device, and a singer specifying means for specifying a singer corresponding to the imaging data, And the music specifying means is configured to output, from the plurality of music pieces selected and input to the karaoke device, a music piece whose main selection is the singer specified by the singer specifying means. The music storage means identifies the image data captured by the imaging device of the portable terminal device, the singer identified by the singer identification means, and the music identification. It is characterized by being stored in the server device in association with the music specified by the means .

If it does in this way, the music specific | specification means which specifies the music which the performance output was performed by the said karaoke apparatus at the time of imaging | photography of the imaging data by the imaging device of the said portable terminal device, and the imaging device of the said portable terminal device An image storage unit that stores the image data captured by the music device in association with the music specified by the music specifying unit , the image corresponding to the user of the karaoke device, and the identification information of the user And corresponding to the imaging data based on the image included in the imaging data captured by the imaging device of the portable terminal device from the relationship stored in the customer database and the relationship stored in the customer database a singer specifying means for specifying a singer, wherein the music specifying unit, or a plurality of music pieces music selection input to the karaoke apparatus , The music having the singer specified by the singer specifying means as a music selection input subject is specified as the music for which performance output has been performed by the karaoke device, and the imaging storage means is the portable terminal. The image data captured by the imaging device of the apparatus is stored in the server device in association with the singer specified by the singer specifying means and the music specified by the music specifying means. Therefore, it is possible to specify which song karaoke performance is taken with respect to the imaging data shot by the portable terminal device, and the identified song is associated with the imaging data. By adding and storing, it is possible to perform processing such as combining the captured image data with another moving image corresponding to the same music piece. That is, the karaoke system which improves the convenience of the imaging data which image | photographed the mode that a user performs karaoke performance with a portable terminal device can be provided. In addition, regarding the imaging data captured by the portable terminal device, it is possible to specify in a practical manner whether the karaoke performance of which music is being performed, and the specified music and singers and their imaging data By storing them in association with each other, the convenience of the imaging data can be further improved.

  Preferably, the portable terminal device further includes an imaging list display means for selectively displaying imaging data stored in the server device in association with the same music as the music specified by the music specifying means. It is provided. In this way, the process of combining the image data captured by the portable terminal device with another moving image corresponding to the same music as the music corresponding to the image data is executed in a suitable and practical manner. be able to.

  Preferably, the customer database stores a face image corresponding to a user of the karaoke device and identification information of the user in association with each other, and the singer specifying means includes the customer Based on the relationship stored in the database, the singer corresponding to the image data is specified based on the face image of the person included in the image data captured by the image capturing device of the portable terminal device. If it does in this way, about the imaging data image | photographed with the said portable terminal device, it is a suitable and practical aspect which image | photographed the state of performing the karaoke performance of the singer corresponding to the imaging data and which music Can be specified.

  Preferably, the singer specifying means responds to the imaging data based on an image corresponding to a microphone included in the imaging data when the imaging data includes face images of a plurality of persons. It identifies the singer. If it does in this way, about the imaging data image | photographed with the said portable terminal device, the singer corresponding to the imaging data can be specified in a suitable and practical aspect.

  Preferably, when the imaging data includes a plurality of person's face images, the imaging storage means determines the person's face image included in the imaging data from the relationship stored in the customer database. The identification information of all persons included in the imaging data is acquired based on the image data, and the imaging data is associated with the acquired identification information of all persons and the music specified by the music specifying means in the server device. It will be memorized. In this way, the image data captured by the portable terminal device is stored in association with all the users who were performing karaoke performances corresponding to the image data, thereby further improving the convenience of the image data. Can be improved.

It is a figure explaining an example of composition of a karaoke system to which the present invention is applied suitably. It is a block diagram which illustrates the structure of the karaoke apparatus with which the karaoke system of FIG. 1 was equipped. It is a block diagram which illustrates the composition of the server apparatus with which the karaoke system of Drawing 1 was equipped. It is a block diagram which illustrates the structure of the mobile telephone with which the karaoke system of FIG. 1 was equipped. FIG. 5 is a functional block diagram illustrating a main part of control functions provided in the CPU of the server device of FIG. 3 and the CPU of the mobile phone of FIG. 4. FIG. 5 is a diagram illustrating an example of image data captured by a digital camera provided in the mobile phone of FIG. 4. It is a figure which illustrates the image containing the music confirmation button which specifies the music corresponding to the imaging data of FIG. It is a figure which illustrates the image containing the music selection button which specifies the music corresponding to the imaging data of FIG. FIG. 5 is a diagram illustrating an example of imaging data including face images of a plurality of persons photographed by a digital camera provided in the mobile phone of FIG. 4. FIG. 5 is a diagram illustrating a state in which thumbnails of moving image data associated with the same music as the image data captured by the digital camera provided in the mobile phone of FIG. 4 are retrieved and displayed in parallel with the image data. . It is a figure which shows an example of the co-starring animation delivered by the server apparatus of FIG. 5 is a flowchart for explaining a main part of an example of imaging data storage control by a CPU of the mobile phone of FIG. 4. 5 is a flowchart for explaining a main part of an example of co-star video setting control by a CPU of the mobile phone of FIG.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration of a karaoke system 10 to which the present invention is preferably applied. In the karaoke system 10 shown in FIG. 1, a plurality of private rooms 14a, 14b, 14c,... In a store 12 such as a karaoke box, snack, inn, etc. (hereinafter simply referred to as a private room 14 unless otherwise distinguished). Karaoke devices 16a, 16b, 16c,... (Hereinafter simply referred to as karaoke devices 16 unless otherwise distinguished) are installed. The plurality of karaoke devices 16 are connected to a communication line 20 via a LAN (local area network) 24 and a router 22, and are similarly connected to the server device 18 and the like connected to the communication line 20. Information can be communicated via the communication line 20.

  The karaoke system 10 includes a plurality of electronic quick sample devices 28a, 28b, 28c,... (Hereinafter simply referred to as the electronic quick sample device 28 unless otherwise distinguished). 1 to several of the electronic sample devices 28 are lent to each user (group), and an association process is performed between each individual room 14 and the karaoke device 16 as will be described later. Therefore, it can be used as a remote control device for the karaoke device 16. Further, as shown in FIG. 1, a LAN 24 for connecting the plurality of karaoke devices 16 to each other is laid in the store 12, and an input to the karaoke device 16 from the electronic quick sample device 28 is predetermined. The access point 26 and the LAN 24 are used for LAN communication.

  The communication line 20 is a wide area information communication network connected to the Internet such as a WWW (World Wide Web) configured by, for example, a public telephone line, an ADSL line, or an optical fiber line. The server device 18 is a server operated by an information distribution service provider that manages the karaoke system 10, for example, music data (karaoke data), background video information, information between songs, etc. in the karaoke system 10. It functions as a center device that performs basic control of storage and input / output management of digital contents. Further, for each user who uses the karaoke system 10, an SNS database 108 (FIG. 5) stores information related to karaoke performance using the karaoke device 16 in association with identification information (user ID) of the user. Function as a database server that manages In the present embodiment, the configuration including the single server device 18 having both the function as the center device of the karaoke system 10 and the function as the database server will be described. It may be configured as an individual server device. In addition, a mode in which various controls by the server device 18 described later are executed in a distributed manner by a plurality of server devices is also conceivable.

  As shown in FIG. 1, the karaoke system 10 according to the present embodiment can transmit and receive information to and from a plurality of systems or communication terminal devices via the communication line 20. That is, a mobile phone 30 and a tablet 32 as portable terminal devices are connected to the communication line 20 via a relay base station 34, and between the mobile phone 30 and the tablet 32 and the server device 18 and the like. It is possible to send and receive information to each other. The communication line 20 is connected to a server device 36 as a so-called proxy server that relays information such as music selection input to the karaoke device 16 from the mobile phone 30 and the tablet 32. The input of information from the mobile phone 30 and the tablet 32 to the karaoke device 16 is preferably performed via the server device 36. Here, the communication line between the mobile phone 30, the tablet 32, and the like and the server device 36 may be different from the communication line between the server device 36 and the karaoke device 16. . For example, the cellular phone 30, the tablet 32, etc., and the server device 36 may be connected by a general line, and the server device 36 and the karaoke device 16 may be connected by a dedicated line. .

  FIG. 2 is a block diagram illustrating the configuration of the karaoke apparatus 16. As shown in FIG. 1, the karaoke device 16 includes a CPU 42 as a central processing unit, a ROM 44 as a read-only memory, a RAM 46 as a write / read memory as needed, a hard disk 48 as a storage device, and a graphic. Connected to the CPU 42 through a video processing unit 50 such as a chip (graphics board), an audio processing unit 52 such as a sound chip (sound board), an operation panel 54, a display control unit 56 and an input control unit 58. The touch panel display 60, the LAN interface 66, the wireless communication unit 68, the display 72 which is a video display device connected to the video processing unit 50 via the video output terminal 70, and the audio input terminal 74. A microphone 76 that is an audio input device connected to the audio processing unit 52; An amplifier 80 an audio amplifier connected to the audio processing unit 52 via an audio output terminal 78, a speaker 82 is an audio output device provided to the amplifier 80 is configured to include.

  The CPU 42 is a so-called microcomputer that processes and controls electronic information based on a predetermined program stored in the ROM 44 using the temporary storage function of the RAM 46, and executes various controls in the karaoke device 16. . That is, when a predetermined music piece (karaoke performance music piece) is selected and input by the operation panel 54, the touch panel display 60, the electronic quick sample device 28, or the like, the selected music piece is reserved music provided in the RAM 46 or the like. Music selection reservation control to be registered in the table, music data read control for reading performance information and lyric information (music data) of the karaoke performance music selected from the hard disk 48 in accordance with the performance order of the reserved music table, music performance progress In response to the performance output control for transmitting performance information from the RAM 46 to the sound processing unit 52, the lyrics character image is generated based on the lyric information developed in the RAM 46 during the performance output control, and sent to the video processing unit 50. The lyric character video output control to be transmitted, the video processing in the performance output control Background video output control for controlling 50 to reproduce a predetermined background video, and inter-song information for outputting inter-song information such as new music information, music selection ranking, store advertisement, etc. while karaoke performance is not performed, that is, between songs Basic control such as output control is executed.

  The video processing unit 50 performs various controls related to drawing of a screen (video) displayed on the display 72. For example, based on the data supplied from the CPU 42, the data is written into the graphics memory, and the data is read out, thereby controlling the display 72 to display a predetermined screen via the video output terminal 70. Specifically, at the time of musical performance output (karaoke performance) by the karaoke device 16, a text image (telop) such as a lyric character image generated by the CPU 42 is output, or MPEG (stored in the hard disk 48). Based on the background video information such as Moving Picture Experts Group) data, a predetermined background video is reproduced (decoded), the lyrics character video is synthesized on the front side of the background video and displayed on the display 72, Various display controls are performed such as displaying lyric text images in a color-changed manner as the music progresses. In the present embodiment, a mode in which display control of the display 72 is performed by the video processing unit 50 and display control of the touch panel display 60 (display device 62) is performed by the display control unit 56 described later will be described. A mode in which display control of the touch panel display 60 is also performed by the video processing unit 50 is also conceivable. In this aspect, the display control unit 56 is not necessarily provided.

  The sound processing unit 52 includes various sound sources such as an FM sound source and a PCM sound source, and performs various controls related to sound output by the karaoke apparatus 16. Preferably, a synthesizer that synthesizes sound by an electronic circuit and generates various timbres is provided. This synthesizer generates a music signal such as a musical instrument performance signal based on performance information of a karaoke performance music read from the hard disk 48 and sent. The synthesizer preferably includes a MIDI (Musical Instrument Digital Interface) terminal, and the performance information is, for example, data in the MIDI format. The music signal generated by the synthesizer based on the MIDI data is mixed with the singing voice of the user (performer) input from the microphone 76 via the audio input terminal 74, and via the audio output terminal 78. Is supplied to the amplifier 80, amplified by the amplifier 80, and output from the speaker 82.

  The operation panel 54 selects a song that the user of the karaoke apparatus 16 wants to sing, adjusts the pitch related to the performance output of the song, adjusts the volume balance between the performance and the song, The input device includes operation buttons (switches) or knobs for performing various adjustments such as volume and tone. The touch panel display 60 is a device that displays an image (video) and inputs an operation to the karaoke device 16 according to a user's operation. The touch panel display 60 displays a predetermined image (video). And a touch panel 64 for performing input by touching the touch panel display 60 with a user's finger, a pen (not shown), or the like. The display control unit 56 is a video processing unit that controls drawing of a screen (video) displayed on the display device 62 based on information supplied from the CPU 42. The input control unit 58 is an input processing unit that supplies operation input information input from the touch panel 64 to the CPU 42 and the like. With the above configuration, the touch panel display 60 functions as a second video display device separately from the display 72, and the user of the karaoke device 16 selects a song that the user wants to sing, It functions as an input device for adjusting the pitch related to the performance output, adjusting the volume balance between the performance and the song, and performing various adjustments such as echo, volume and tone.

  The LAN interface 66 is a connector for connecting the karaoke device 16 to the LAN 24 via a LAN connection terminal 84. By being connected to the LAN 24 in this way, the karaoke device 16 is similarly connected to the LAN 24. Information can be transmitted to and received from other devices such as the electronic sample device 28 connected to the LAN 24. In addition, when a plurality of karaoke devices are provided in a store or the like where the karaoke device 16 is installed, information can be transmitted and received between karaoke devices connected to the LAN 24 in the same manner. For example, the music selection input from the electronic sample device 28 received via the access point 26 connected to the LAN 24 is accepted and stored in the reserved music table provided in the RAM 46, or via the access point 26. Mutual information exchange is performed between the karaoke device 16 and the electronic quick sample device 28 via radio waves, such as transmitting predetermined information from the karaoke device 16 to the electronic quick sample device 28.

  As shown in FIG. 1, the karaoke apparatus 16 is connected to the communication line 20 via the LAN 24, the router 22, and the like. Similarly, between the karaoke apparatus 16 and other devices connected to the communication line 20 Information can be communicated via the communication line 20. Preferably, it is connected to the server device 18 via the communication line 20, and digital contents (Digital Contents) such as music data (karaoke data), background video information, and information between songs are transmitted from the server device 18. Accept delivery. That is, the karaoke device 16 is preferably a communication karaoke device that is connected to a predetermined communication line and transmits / receives various information to / from the server, but is not connected to such a communication line. The present invention is also suitably applied to devices and the like.

  The wireless communication unit 68 performs wireless communication between the karaoke device 16 and an input device such as the electronic quick sample device 28. For example, it functions as a remote control receiving unit that receives a remote control signal transmitted from an input device such as the electronic sample device 28. The association (sticking) processing between the karaoke device 16 and the electronic quick sample device 28 is preferably performed via the wireless communication unit 68 by such a remote control signal (infrared signal). In other words, the electronic sample device 28 has an individual serial number, and in the association process, for example, a signal including the serial number (for example, the last four digits) and a predetermined connection code is notified of the connection. The electronic quick sample device 28 is associated with the karaoke device 16 that is transmitted to the karaoke device 16 and receives the connection signal by the wireless communication unit 68. The electronic quick sample device 28 thus associated with the karaoke device 16 functions as an input device (remote control device) of the karaoke device 16, and a signal transmitted from the electronic quick sample device 28 is the CPU 42. The user of the karaoke device 16 selects a song that the user wants to sing, adjusts the pitch related to the performance output of the song, adjusts the volume balance between the performance and the song, and so on. Input for making various adjustments such as volume, tone, etc. is accepted. After the association process is performed, communication between the electronic sample device 28 and the karaoke device 16 is performed by LAN communication via the LAN 24, the access point 26, and the like. Further, in the present embodiment, the following description will be given on the assumption that the input device such as the electronic quick sample device 28 subjected to the association processing with the karaoke device 16 also constitutes a part of the karaoke device 16.

  FIG. 3 is a diagram illustrating the configuration of the server device 18. As shown in FIG. 3, the server device 18 includes a CPU 90 that is a central processing unit, a ROM 92 that is a read-only memory, and a RAM 94 that is a write / read memory as needed. The CPU 90 provides a temporary storage function of the RAM 94. This is a so-called Neumann computer that performs signal processing in accordance with a program stored in the ROM 92 in advance. Also, a video display device 96 such as TFT or PDP, a video processing unit 98 such as a video board (graphics chip) for controlling display of video by the video display device 96, an input device 100 such as a keyboard, An interface 102 for processing input by the input device 100 and a modem 104 for connecting the CPU 90 and the like to the communication line 20 are provided. The server device 36 is a so-called Neumann computer having a configuration equivalent to that of the server device 18 shown in FIG. 3 with respect to the CPU 90, ROM 92, RAM 94, modem 104, and the like, for example. Description is omitted.

  As shown in FIG. 3, the server device 18 includes various databases including a music database 106 and an SNS database 108. The music database 106 stores a large number of music data (karaoke data) for distribution to the karaoke device 16 or the like in the karaoke system 10, and newly created music data is stored in the music database 106. The Then, new music data is distributed from the music database 106 of the server device 18 via the communication line 20 periodically by a predetermined distribution control program or in response to a distribution request from the karaoke device 16 or the like. , And delivered to the karaoke device 16 or the like.

  The music database 106 stores a large number (for example, tens of thousands of songs) of music data (karaoke data) corresponding to the music that can be output by the karaoke device 16. The music data is generated based on performance information for generating a performance sound of a predetermined instrument, lyrics information for generating a lyric character image (lyric telop), and the lyrics information. Lyrics color change information for sequentially changing the color of the lyric character video in accordance with the progress of the performance, and is identified by a music selection number unique to each music piece as the content ID. The song data includes the song name, artist name (singer name), date of announcement, song length (performance time), genre, song tone, birthplace of the artist (singer), concert information (close) Information related to the venue where the music concert will be held in the future), tie-up information (for example, the day of the week, time zone, CM song on which the drama was used as the theme song) It is remembered.

  The SNS database 108 stores various types of information related to a social network service for users of the karaoke system 10. This social network service refers to, for example, a member-based community-type website that provides services such as information browsing only between members who are registered in advance. In the following description, the social network service is abbreviated as SNS. The SNS database 108 stores information on karaoke performance using the karaoke device 16 for each user who uses the karaoke system 10 in association with identification information (user ID) of the user. It is. This SNS database 108 corresponds to each user identified by the user ID, for example, the user's name (nickname), date of birth, actual age, gender, e-mail address, user address or residence. Area, blood type, constellation, password used for login authentication to SNS, questions and answers for forgotten passwords, information on avatars (humanoid images that symbolize users on the net), and users Is stored in association with the user ID of the user. This singing age is virtual age information indicating what age (how many years) the user's favorite tendency of the music corresponds to, and the target user uses the past in the karaoke apparatus 16. This value is determined based on the music selected (performed).

  Further, in the SNS database 108, for example, the store 12 corresponding to the karaoke device 16 used by the user in the past (the karaoke device 16 is installed) as information related to the karaoke performance of each user using the karaoke device 16. The store visit history, which is information related to the store 12), and the music registered by the user as the eighteenth song or favorite in the karaoke performance by the karaoke device 16 (the song set so that the music selection input can be performed by a simple operation) Information, information related to the music selection history that the user has selected in the past in the karaoke performance by the karaoke device 16 (history of the music selected in the past in the karaoke device 16), and the user in the karaoke performance by the karaoke device 16 in the past Information on evaluation results of performance evaluations And the user information concerning other users registered as friends are stored in association with the user ID of the user for each user.

  The SNS database 108 stores image data corresponding to each user of the karaoke apparatus 16 in association with the user ID of the user. This image data preferably corresponds to a face image of each user, that is, an image necessary and sufficient for grasping the relative positional relationship of parts such as eyes, nose and mouth with respect to the contour of the face. To do. Preferably, a face image of each user photographed in advance is stored in, for example, the SNS database 108 at the time of the SNS registration. As described later, the moving image data is extracted from the moving image data posted by each user. May be stored as a face image corresponding to the user, for example, by capturing a face image of a person included in. Moreover, the image of accessories etc. which each user owns may be associated with the user and stored in the SNS database 108. That is, in the present embodiment, the SNS database 108 corresponds to a customer database that stores an image (face image) corresponding to the user of the karaoke apparatus 16 and the identification information of the user in association with each other.

  The SNS database 108 stores moving image data posted from each user of the karaoke apparatus 16 in association with the user ID of the user. For example, moving image data (imaging data) taken by an imaging device (not shown) provided in the karaoke device 16 or a digital camera 144 provided in the mobile phone 30 which is a portable terminal device is included in the moving image data. Corresponding music identification information (preferably music selection number) and user identification information (preferably user ID) are stored in association with each other. That is, in the present embodiment, a description will be given of a configuration including a single server device 18 that has both a function as a database server related to the SNS and a function as a moving image information distribution server. A mode of being executed in a distributed manner by the server device is also conceivable.

  FIG. 4 is a diagram illustrating the configuration of the mobile phone 30. As described above, the karaoke system 10 of the present embodiment includes the mobile phone 30 and the tablet 32 as portable terminal devices that perform information communication with the server device 18 and the like. In the description, an example in which the mobile phone 30 is used as a portable terminal device will be described, and the description of the other portable terminal devices will be omitted. That is, the various controls in the embodiments described below are also suitably executed in portable terminal devices other than the mobile phone 30 including the tablet 32.

  As shown in FIG. 4, the mobile phone 30 includes a CPU 110 as a central processing unit, a ROM 112 as a read-only memory, a RAM 114 as a write / read memory as needed, a storage unit 116 such as a flash ROM, an antenna, and the like. 120, a transmission / reception unit 122, a modem unit 124, an audio codec unit 126, an analog front end 128, a speaker 130, a microphone 132, a touch panel display 134, and a digital camera 144 that is an imaging device. It is configured.

  The CPU 110 is a so-called microcomputer that performs signal processing in accordance with a program stored in advance in a ROM 112 that is a read-only memory while using a temporary storage function of the RAM 114 that is a read / write memory as needed, and is displayed on the touch panel display 134. So that character input is executed according to the input operation to the character input button, or communication or a call channel is established according to the telephone number input by such input operation, and mutual communication or communication is enabled. In addition to basic control such as controlling, accessing a predetermined website according to the URL input by the input operation, controlling creation and transmission / reception of e-mails, imaging by the digital camera 144, etc. Imaging data storage of this embodiment to be described later To perform various control of your like. For example, in a call state of the mobile phone 30, a call signal received by the transmission / reception unit 122 via the antenna 120 is converted from a digital code signal to a voice signal by the voice codec unit 126 via the modem unit 124. The speaker 130 is driven by the front end 128 to output sound. At the same time, an audio signal converted from audio by the microphone 132 is sent to the audio codec unit 126 via the analog front end 128 and converted into a digital signal there, and the modem unit 124, the transmission / reception unit 122, and the antenna 120. Sent through.

  The mobile phone 30 includes a display device 136 that displays a predetermined image on the touch panel display 134, a display control unit 138 that controls display by the display device 136, a touch panel using a user's finger, a pen that is not shown, or the like. A touch panel 140 that performs input by touching the display 134 and an input control unit 142 that controls input by the touch panel 140 are provided. Further, it is connected to the communication line 20 via the transmission / reception unit 122 and the like, and information can be communicated with various communication devices including the server device 18 connected to the communication line 20. In the communication state via the communication line 20, the CPU 110 transmits a signal input from the touch panel display 134 or the like to the communication destination and simultaneously displays the signal received from the communication destination on the touch panel display 134. To control. That is, the mobile phone 30 is a so-called smartphone having the touch panel display 134 and various functions including an access function to the Internet according to an operation through the touch panel display 134.

  The digital camera 144 includes, for example, an image sensor and a lens such as a charge coupled device (CCD) and a complementary metal-oxide semiconductor (CMOS), detects an image incident from the lens by the image sensor, and converts the image into electronic information. It is a so-called digital video camera acquired as (imaging data) and can shoot at least moving images (videos that move with time), but still images (still photos) can be taken as needed. It may be configured as described above. The imaging data photographed by the digital camera 144 is supplied to the CPU 110 or the like via an interface such as a video terminal (not shown), and together with audio information input from the microphone 132, for example, AVI (Audio-Video Interleaved) format, for example. The video data is stored in the storage unit 116 or the like as a video file (moving image data) in MPEG (Moving Picture Experts Group) format, FLV (Flash Video) format, or the like.

  FIG. 5 is a functional block diagram illustrating a main part of control functions provided in the CPU 90 of the server device 18, the CPU 110 of the mobile phone 30, and the CPU 90 of the server device 36. As shown in FIG. 5, preferably, the SNS management control means 150, the video distribution control means 152, and the music identification means 170 are functionally provided in the CPU 90 of the server device 18, and the music selection input is performed. The control means 154, user identification means 158, login authentication control means 162, music identification means 164, imaging storage means 166, and imaging list display means 168 are functionally provided in the CPU 110 of the mobile phone 30. The music selection input control means 156 is functionally provided in the CPU 90 of the server device 36. That is, in this embodiment, the specific control of the singer corresponding to the image data captured by the mobile phone 30 (digital camera 144) as the portable terminal device, the specific control of the music corresponding to the image data, and A description will be given of an example in which the CPU 110 of the mobile phone 30 performs imaging storage control in which the imaging data is stored in the server device 18 (SNS database 108) in association with corresponding music or the like. For example, the music specifying unit 170 will be described later. As described above, part or all of these controls may be executed by the server device 18 or the karaoke device 16. That is, a part or all of the user specifying unit 158, the music specifying unit 164, and the imaging storage unit 166 may be functionally provided in the CPU 90 of the server device 18 or the CPU 42 of the karaoke device 16. Good.

  The SNS management control means 150 performs information management control related to SNS in the karaoke system 10. For example, information registration control regarding the SNS is performed. Specifically, a new user (user) is registered in the SNS database 108 or stored in the SNS database 108 in response to an input operation by an external device such as the electronic sample device 28 or the mobile phone 30. The registered content is changed (updated), a plurality of users stored in the SNS database 108 are registered in association with each other, and the karaoke device of each user is based on information related to the friends. The management management control of SNS in the karaoke system 10 is performed such as setting the browsing level of information related to karaoke performance using the karaoke system 16.

  The moving image distribution control unit 152 distributes moving image data stored in the SNS database 108 via the communication line 20. For example, when there is a request for distribution of predetermined moving image data from the communication terminal device connected to the communication line 20, such as the mobile phone 30 which is a portable terminal device or a personal computer (not shown), via the communication line 20. Distributes the video data related to the distribution request stored in the SNS database 108 to the communication terminal device that is the request source via the communication line 20 in response to the distribution request. The moving image data distributed to each communication terminal device by the moving image distribution control means 152 is displayed on a display unit (video display device) of the communication terminal device by a browser (Web browser) provided in the communication terminal device. The

  Further, the moving image distribution control means 152 distributes a plurality of moving image data stored in the SNS database 108 through the communication line 20 as a co-starring moving image (collaboration moving image). This co-star video is a plurality (preferably one set) of video data corresponding to the same music piece, which is output in synchronization (parallel) within a predetermined display screen. That is, when there is a distribution request for a predetermined co-starring moving image via the communication line 20 from the communication terminal device connected to the communication line 20, the moving image distribution control means 152 responds to the distribution request and Data for displaying the co-starring moving image related to the distribution request stored in the SNS database 108 is distributed to the communication terminal device that is the request source via the communication line 20. The data for displaying the co-star video is preferably video data to be combined and stored as independent video data in the SNS database 108, and for synchronously outputting the video data as the co-star video. Although it is co-starring moving image information as combination information, a plurality of moving image data related to the combination may be synthesized in advance and stored as a single moving image data (moving image file).

  The login authentication control means 162 authenticates the SNS user in the karaoke system 10 according to the identification information (user ID) input from the touch panel display 134, and if the authentication is normally performed, The user is permitted to log in to the SNS with the identification information. Specifically, when a user ID and password, which are identification information, are input from the touch panel display 134, does the password match the user ID and the password stored in the SNS database 108? The server device 18 is inquired of whether or not, and if they match, normal authentication is performed, whereas if they do not match, an error is determined. The login authentication control unit 162 may transmit a user ID and a password to the server device 18 by the touch panel display 134, and substantial authentication is performed on the server device 18 side. It doesn't matter.

  Further, the login authentication control means 162 may be functionally provided in the control device of the electronic quick sample device 28, or may be executed by the electronic quick sample device 28. That is, when a user ID and password, which are identification information, are input from the touch panel display provided in the electronic quick sample device 28, the password stored in the SNS database 108 in association with the user ID is input. The server device 18 is inquired as to whether or not they match, and if they match, normal authentication is performed, whereas if they do not match, an error is assumed. When normal authentication is performed by the processing in the electronic quick sample device 28, preferably, the serial number of the electronic quick sample device 28 that has been authenticated and the karaoke associated with the electronic quick sample device 28 Information such as the device ID of the device 16 is stored in the SNS database 108 for each user.

  The music selection input control means 154 performs an association process for associating (attaching) the mobile phone 30 to the target karaoke device 16 when using the mobile phone 30 as a portable terminal device in the karaoke system 10. That is, when the mobile phone 30 is used as an input device of the karaoke device 16, processing for setting the karaoke device 16 that is an object of input (for example, music selection input) by the mobile phone 30 is performed. Preferably, such an association process is performed using an authentication result by the login authentication control means 162. For example, when normal authentication is performed by the login authentication control unit 162, the SNS database 108 is referred to, and the device ID of the karaoke device 16 used by the user corresponding to the received user ID is acquired. . If the password associated with the user ID and stored in the SNS database 108 matches the password received from the mobile phone 30, the karaoke corresponding to the mobile phone 30 and the acquired device ID Processing for associating the device 16 is performed, and the association information is transmitted to the server device 36. That is, the substantial processing in the association between the mobile phone 30 and the karaoke device 16 may be performed on the server device 18 side. The association processing may be executed at the start of use of the music selection input function by the mobile phone 30, but a music selection input of a predetermined music from the mobile phone 30 is input to the karaoke device 16. It may be executed when it is performed.

  The music selection input control means 154 inputs music selection of a predetermined music piece to the karaoke apparatus 16. That is, music selection input information of a predetermined music is transmitted to the karaoke apparatus 16 associated with the mobile phone 30 by the association processing as described above in response to an input operation by the touch panel display 134. Specifically, at least the music selection number of the music to be input for music selection is transmitted. Preferably, the music selection number of the music to be input for music selection and the user ID of the user who is the music selection subject are transmitted. In this embodiment, transmission of music selection input information from the mobile phone 30 to the karaoke device 16 is performed via the server device 36 for security protection, for example. In other words, the music selection input information transmitted from the music selection input control means 154 to the karaoke device 16 is once received by the server device 36 via the communication line 20. Then, the music selection input control means 156 in the server device 36 transmits the music selection input information to the target karaoke device 16, that is, the karaoke device 16 associated with the mobile phone 30 by the association processing. Registration (music selection reservation) of the music related to the music selection input information is performed in the reserved music table in the target karaoke apparatus 16.

  Based on the image stored in the image data captured by the digital camera 144 of the mobile phone 30, which is a portable terminal device, the user specifying means 158 is based on the image stored in the SNS database 108. The user corresponding to is identified. Preferably, a user corresponding to the image data is specified based on a face image included in the image data captured by the digital camera 144 of the mobile phone 30. Specifically, with respect to the image data captured by the digital camera 144, a face image of a person included in the image data is recognized by a well-known face recognition technique. Then, the face image is compared with each user's face image stored in the SNS database 108, and if a matching face image exists, the user corresponding to the face image corresponds to the imaging data. Identified as a user. Further, in an aspect in which images of accessories or the like related to the possession of each user are stored in the SNS database 108 in association with the user, the user specifying means 158 is stored in the SNS database 108. From the relationship, the user corresponding to the image data is specified based on the image of the accessory or the like included in the image data captured by the digital camera 144.

  The login authentication control unit 162 may preferably perform the login authentication control using the identification result by the user identification unit 158. For example, when the user specifying unit 158 inquires the stored contents of the SNS database 108 and specifies a user corresponding to the image data captured by the digital camera 144, Allow login. In addition, when such login authentication is normally performed, various information stored in the SNS database 108 corresponding to the user related to the login authentication (for example, the user's name, gender, song selection history) Or the like may be acquired and displayed on the touch panel display 134.

  As shown in FIG. 5, the user specifying means 158 includes a singer specifying means 160. This singer specifying means 160 is based on the image data included in the image data captured by the digital camera 144 of the mobile phone 30 that is a portable terminal device, based on the relationship stored in the SNS database 108. The singer corresponding to is identified. Preferably, a singer corresponding to the image data is specified based on a face image included in the image data captured by the digital camera 144 of the mobile phone 30. Specifically, with respect to the image data captured by the digital camera 144, a face image of a person included in the image data is recognized by a well-known face recognition technique. Then, the face image is compared with each user's face image stored in the SNS database 108, and if a matching face image exists, the user corresponding to the face image corresponds to the imaging data. Identified as a singer. Further, in an aspect in which images of accessories or the like related to the possession of each user are stored in the SNS database 108 in association with the user, the user specifying means 158 is stored in the SNS database 108. From the relationship, the singer corresponding to the imaging data is specified based on the image of the accessory or the like included in the imaging data captured by the digital camera 144.

  The singer identification unit 160 preferably uses an image corresponding to the microphone 76 included in the image data when the image data captured by the digital camera 144 includes face images of a plurality of persons. The singer corresponding to the imaging data is specified based on the image data. Specifically, regarding the image data captured by the digital camera 144, an image corresponding to the microphone 76 included in the image data is recognized by a well-known image recognition technique. Then, the face image closest to the recognized image corresponding to the microphone 76 among the plurality of face images included in the captured image data is compared with the face image of each user stored in the SNS database 108 and matched. If there is a face image to be played, the user corresponding to the face image is specified as the singer corresponding to the image data.

  The music specifying means 164 and 170 specify the music for which performance output has been performed by the karaoke device 16 at the time of capturing the image data by the digital camera 144 of the mobile phone 30 which is a portable terminal device. In other words, control is performed to identify which image data captured by the digital camera 144 is an image of the karaoke performance of a tune that can be output by the karaoke device 16. . Here, the music specifying means 164 operates in such a manner that the CPU 110 of the mobile phone 30 specifies the music when uploading the imaged data to the SNS database 108 by the mobile phone 30. In such an embodiment, the music specifying means 170 is not necessarily provided. The music specifying means 170 operates in such a manner that the music is specified by the CPU 90 of the server device 18 after the imaging data is uploaded to the SNS database 108 by the mobile phone 30. In the aspect, the music specifying means 164 is not necessarily provided.

  The music specifying means 164 is preferably connected to the karaoke apparatus 16 by the mobile phone 30 (music selection input control means 154) before photographing the target image data by the digital camera 144 of the mobile phone 30. The music piece selected and input is specified as the music piece for which performance output has been performed by the karaoke device 16 at the time of shooting the target imaging data. That is, the server device 36 (music selection input control means 156) is moved from the mobile phone 30 before the shooting time of the image data taken by the digital camera 144 of the mobile phone 30 (for example, within a predetermined time after that time). If the music selected and input to the karaoke device 16 is inquired via the mobile phone 30 and there is a music selected and input from the mobile phone 30, the karaoke device is captured at the time of capturing the image data for the song. 16 identifies the music piece for which performance output has been performed. In addition, when there are a plurality of music pieces selected and input from the mobile phone 30 to the karaoke device 16, it is preferable that the music pieces can be selectively input to the touch panel display 134 as shown in FIG. And the music selected by the user's input operation is specified as the music for which performance output has been performed by the karaoke device 16 at the time of capturing the target imaging data.

  In addition, the music specifying unit 164 preferably selects a song having a singer specified by the singer specifying unit 160 as a music selection input subject from a plurality of songs selected and input to the karaoke apparatus 16. The karaoke device 16 is specified as a musical piece for which performance output has been performed. That is, the singer specified by the singer specifying means 160 is selected as a music selection input subject from a plurality of music pieces selected and input to the karaoke device 16, that is, a plurality of reserved music stored in the reserved music table of the RAM 46. When there is a song that searches for the song to be performed (reserved song) and the singer is the music selection input subject, the karaoke device 16 performs the performance output at the time of shooting the imaging data for the song. Specified as a song. In addition, when a plurality of songs having the singer specified by the singer specifying means 160 as a music selection input subject are searched, preferably, the songs are displayed on the touch panel display 134 as shown in FIG. The music piece selected by the user's input operation is specified as the music piece whose performance is being output by the karaoke device 16 at the time of shooting the target imaging data.

  FIG. 6 is a diagram illustrating an example of imaging data 172 (one frame of a moving image corresponding to the imaging data 172) captured by the digital camera 144 of the mobile phone 30 which is a portable terminal device. Is shown on the touch panel display 134. When the music specifying unit 164 performs music specifying control with respect to the image data 172 and the candidate music is determined as one music, as shown in FIG. 7, the music title (and preferably the singer name) of the music is shown. ) And a music confirmation button 174 and a music rejection button 176 are displayed on the touch panel display 134. In the screen shown in FIG. 7, a contact operation (hereinafter referred to as a contact input operation) is performed with a user's finger or a pen provided on the music confirmation button 174 (relative position corresponding to the music confirmation button 174 on the touch panel 140). In the case where the music is displayed, the music displayed on the screen is specified as the music for which performance output has been performed by the karaoke apparatus 16. Further, in the screen shown in FIG. 7, when a contact input operation is performed on the music denial button 176, the music displayed on the screen is not specified as the music on which the performance output has been performed by the karaoke device 16, The music specifying control by the music specifying means 164 is resumed. In addition, when a candidate song is not fixed to one song and a plurality of songs exist as candidates for a song for which performance output has been performed by the karaoke apparatus 16, a plurality of candidates are obtained as shown in FIG. A song selection button 178a, 178b (hereinafter simply referred to as a song selection button 178 unless otherwise specified) is provided with the song name (and preferably the singer's name) of each song (in the example shown in FIG. 8, two songs). 134. In the screen shown in FIG. 8, when a contact input operation is performed on any of the music selection buttons 178, the music corresponding to the music selection button 178 related to the contact input operation is played by the karaoke device 16. It is identified as the song that was playing.

  The music specifying means 170 specifies music corresponding to moving image data (imaging data) posted (uploaded) to the SNS database 108 from the mobile phone 30 or the like which is a portable terminal device. Preferably, when video data is posted from the mobile phone 30 or the like to the SNS database 108, the music selection history of the karaoke device 16 stored in association with the user who is the poster of the video data is inquired. The music corresponding to the moving image data is specified based on the music selection history. For example, the shooting date / time (data creation date / time) of the target video data is compared with the music selection date / time of each song in the music selection history. For example, the song selected immediately before the shooting date / time is recorded by the karaoke device 16. It is specified as a song for which performance output has been performed. When the music selection history stored in the SNS database 108 stores the date and time when each piece of music was actually played in the karaoke device 16, the music selection history is played at the shooting date and time of the target moving image data. The tune that has been played is identified as the tune for which performance output was performed by the karaoke device 16.

  The imaging storage unit 166 associates the imaging data captured by the digital camera 144 of the mobile phone 30 that is a portable terminal device with the music specified by the music specifying unit 164, and the SNS of the server device 18. Store in database 108. Preferably, the imaging data photographed by the digital camera 144 is stored in the SNS database 108 in association with the singer identified by the singer identifying unit 160 and the song identified by the song identifying unit 164. Let Specifically, after the imaging data is captured by the digital camera 144, when an operation for posting the imaging data to the SNS database 108 is performed by the touch panel display 134 or the like, the communication line 20 is used. The imaging data is uploaded to the server device 18 and stored in the SNS database 108 in association with the user ID of the singer and the music selection number of the specified music.

  When moving image data is posted to the SNS database 108 from the mobile phone 30 or the like, the music specifying unit 170 specifies the music corresponding to the moving image data, and the function corresponding to the imaging storage unit 166 has a function. The CPU 90 of the server device 18 is provided. In such an aspect, the imaging storage unit 166 associates the video data uploaded from the mobile phone 30 or the like with the user ID of the poster and the music specified by the music specifying unit 170 in the SNS database 108. Remember me. That is, in such an aspect, the imaging storage unit 166 performs a process of associating the moving image data stored in the SNS database 108 with the user ID of the singer and the music selection number of the specified music piece.

  The imaging storage unit 166 preferably specifies the user identification when the imaging data captured by the digital camera 144 of the mobile phone 30 which is a portable terminal device includes a plurality of human face images. The user corresponding to all the face images included in the imaged data is identified by means 158 and its identification information is acquired, and the imaged data is acquired from all the acquired user identification information and the music specifying means 164. Is stored in the server device 18 (SNS database 108) in association with the music specified by. In other words, the image data captured by the digital camera 144 is associated with the identification information of all users corresponding to the face images included in the image data and the music specified by the music specifying means 164. It is stored in the SNS database 108. For example, as shown in FIG. 9, a plurality of person's face images 182a, 182b, 182c (hereinafter, not particularly distinguished) in the image data 180 (one frame of a moving image corresponding to the image data 180) captured by the digital camera 144. If the user corresponding to all the face images 182 included in the image data 180 is specified by the user specifying unit 158, the image data 180 is specified. The SNS database 108 stores the identification information of all users and the music specified by the music specifying means 164 in association with each other. By such processing, the posters of the imaging data 180 posted from the mobile phone 30 are all users corresponding to the face images 182 included in the imaging data 180, respectively. In other words, the imaging data 180 posted from the mobile phone 30 is associated with all the users corresponding to the face images 182 included in the imaging data 180, respectively, and is posted as a posted video of each user. Stored in the SNS database 108.

  The imaging list display means 168 displays the moving image data (imaging data) stored in the server device 18 in association with the same music as the music specified by the music specifying means 164 so as to be selectable on the touch panel display 134. Let Preferably, the moving image data associated with the same music as the music specified by the music specifying means 164 and stored in the server device 18 is displayed on the touch panel display 134 as a combination candidate for the co-starring moving image. Let In other words, when the music specifying means 164 specifies the music for which performance output is being performed in the karaoke apparatus 16 during the shooting of the image data by the digital camera 144, the same music is recorded in the SNS database 108. A search for moving image data stored in association with each other is performed. With regard to this search, preferably, the priority of search is determined according to the number of times each video data is browsed and the presence / absence of friend settings between the poster of each video data and the user who is the subject of imaging data. Then, when the moving image data stored in association with the same music is searched, as shown in FIG. 10, the thumbnails 184a of each of the searched plural (three in the example shown in FIG. 10) moving image data. 184b, 184c (hereinafter simply referred to as thumbnail 184 unless otherwise distinguished) are displayed on the touch panel display 134 of the mobile phone 30 along with the image data 172 being photographed. When a contact input operation is performed on any one of the thumbnails 184 displayed on the touch panel display 134 as described above, moving image data corresponding to the thumbnail 184 related to the contact input operation and the digital camera 144 are captured. Co-starring moving image setting is performed in which the captured image data 172 is combined as the co-starring moving image. That is, when image data captured by the digital camera 144 is posted to the SNS database 108, the co-star video is set in combination with video data corresponding to the thumbnail 184 related to the touch input operation.

  FIG. 11 is a diagram illustrating an example of the co-starring moving image 188 distributed by the moving image distribution control unit 152. The co-star video 188 shown in FIG. 11 is video data stored in the SNS database 108 in association with the image data 172 captured by the digital camera 144 and the same music as the music corresponding to the image data 172. This is a co-star movie 188 combined with 186. As shown in FIG. 11, the moving image distribution control means 152 converts a plurality of moving image data stored in the SNS database 108 and moving image data 172 and 186 set as co-starring moving images into a co-starring moving image 188. Is distributed via the communication line 20. That is, in the communication terminal device that is the request source of the distribution request, the plurality of moving image data 172 and 186 are output as the co-starring moving image 188 in synchronization (parallel). With such control, the image data 172 photographed by the digital camera 144 of the mobile phone 30 can be targeted for the combination of the co-starring moving image and the moving image data stored in the SNS database 108.

  FIG. 12 is a flowchart for explaining a main part of an example of the imaging data storage control by the CPU 110 of the mobile phone 30 and is repeatedly executed at a predetermined cycle.

  First, in step (hereinafter, step is omitted) S1, it is determined whether or not imaging data is being captured by the digital camera 144. If the determination in S1 is negative, the routine is terminated accordingly. If the determination in S1 is affirmative, in S2, the image is taken by the digital camera 144 using a well-known face recognition technique. The face image of the person included in the captured image data is recognized. Next, in S3, the face image recognized in S2 is compared with the face image of each user stored in the SNS database 108, and it is determined whether or not a matching face image exists. If the determination in S3 is negative, the routine is terminated accordingly, but if the determination in S3 is affirmative, the processing in S4 and subsequent steps is executed.

  In S4, a user whose face image stored in the SNS database 108 matches the face image recognized in S2 is specified as a singer corresponding to the image data captured by the digital camera 144. In addition, when the imaging data includes a plurality of face images, a user corresponding to each face image is specified. In addition, when login authentication of the user in the SNS is not performed, login authentication of the user corresponding to the image data captured by the digital camera 144 is performed. Next, in S5, among the plurality of music pieces selected and input to the karaoke device 16, that is, among the plurality of reserved music stored in the reserved music table in the RAM 46, the singer specified in S4 is selected as the main music selection subject. It is determined whether there is a reserved song to be performed. If the determination in S5 is negative, the routine is terminated accordingly. If the determination in S5 is affirmative, in S6, the reservation is performed with the singer identified in S4 as the subject of music selection. It is determined whether or not the song is one song. If the determination in S6 is affirmative, the processing from S8 onward is executed. If the determination in S6 is negative, in S7, the reservation is performed with the singer identified in S4 as the subject of music selection. For example, a screen including a music selection button 178 as shown in FIG. 8 described above is displayed on the touch panel display 134 and a touch input operation to any music selection button 178 is made. It is determined whether or not has been performed. If the determination in S7 is negative, the routine is terminated accordingly, but if the determination in S7 is affirmative, the processing in S8 and subsequent steps is executed.

  In S8, the reserved music mainly selected by the singer specified in S4 or the music selected in S7 among the reserved music is specified as music corresponding to the image data captured by the digital camera 144. Is done. Next, in S9, it is determined whether or not the shooting of the image data is ended by the digital camera 144 (or whether or not the music being shot has ended). While the determination of S9 is denied, the process waits by repeating the determination of S9. However, when the determination of S9 is affirmed, shooting by the digital camera 144 is controlled in S10, and the digital Image data captured by the camera 144 is uploaded to the server device 18 via the communication line 20, the singer identified in S4, the song identified in S8, and the plurality of users in S4. Is identified and stored in the SNS database 108 in association with all users, the routine is terminated.

  FIG. 13 is a flowchart for explaining a main part of an example of the co-star video setting control by the CPU 110 of the mobile phone 30, and is repeatedly executed at a predetermined cycle. In the control shown in FIG. 13, the steps common to the control shown in FIG. 12 described above are denoted by the same reference numerals, and the description thereof is omitted.

  In the control shown in FIG. 13, in S11 following the process of S8 described above, the video data stored in the SNS database 108 in association with the same music as the music specified in S8 is searched. A thumbnail 184 of each of the moving image data is displayed on the touch panel display 134 of the mobile phone 30. Next, in S12, it is determined whether or not a contact input operation has been performed on any of the thumbnails 184 displayed on the touch panel display 134. While the determination of S12 is denied, the process waits by repeating the determination of S12. However, when the determination of S12 is affirmative, in S13, the moving image data corresponding to the thumbnail 184 related to the touch input operation Then, after performing the co-star movie setting in which the image data captured by the digital camera 144 is combined as the co-star movie, this routine is terminated.

  In the above control, S2 to S4 are the operations of the user specifying means 158, the singer specifying means 160 and the login authentication control means 162, S5 to S8 are the operations of the music specifying means 164, and S10 is the imaging storage. S11 corresponds to the operation of the means 166 and the operation of the imaging list display means 168, respectively.

  As described above, according to this embodiment, the karaoke device 16 outputs a performance at the time when the digital camera 144 as the imaging device of the mobile phone 30 which is a portable terminal device is captured. The music specifying means 164 (S5 to S8) to 170 for specifying the music and the image data captured by the digital camera 144 of the mobile phone 30 are associated with the music specified by the music specifying means 164 and 170. Since the image pickup storage means 166 (S10) to be stored in the server device 16 is provided, the karaoke performance of which piece of music is being played with respect to the image pickup data picked up by the mobile phone 30 was photographed. It is possible to specify whether or not the recorded music is associated with the captured image data and stored. Data it is possible to perform processing such as combination with other moving picture corresponding to the same musical composition. That is, it is possible to provide the karaoke system 10 that improves the convenience of the imaging data obtained by photographing the user performing the karaoke performance with the mobile phone 30.

  Further, from the relationship stored in the SNS database 108 as the customer database for storing the image corresponding to the user of the karaoke device 16 and the identification information of the user in association with each other, the mobile phone Singer specifying means 160 (S2 to S4) for specifying a singer corresponding to the image data based on an image included in the image data captured by the digital camera 144 of the telephone 30; The means 164 performs a musical performance output by the karaoke device 16 on a song selected by the singer specifying means 160 as a music selection input subject from a plurality of music pieces selected and input to the karaoke device 16. The imaging storage means 166 is a digital camera of the mobile phone 30. The image data captured by the camera 144 is stored in the server device 18 in association with the song specified by the song specifying unit 160 and the song specified by the song specifying unit 164. With respect to the image data captured by the mobile phone 30, it is possible to specify in a practical manner which karaoke performance of which music is being performed, and associate the identified music and singers with the image data. The convenience of the captured image data can be further improved by storing them.

  In addition, the mobile phone 30 is an imaging list display unit that displays the imaging data associated with the same music as the music specified by the music specifying unit 164 and stored in the SNS database 108 of the server device 18 in a selectable manner. 168 (S11) is provided, and therefore, it is preferable and practical to perform processing such as combining image data captured by the mobile phone 30 with another moving image corresponding to the same music as the music corresponding to the image data. Can be implemented in various ways.

  The SNS database 108 stores a face image corresponding to the user of the karaoke device 16 and the identification information of the user in association with each other. From the relationship stored in 108, based on the face image of the person included in the image data captured by the digital camera 144 of the mobile phone 30, the singer corresponding to the image data is identified. With respect to the image data captured by the mobile phone 30, it is possible to specify in a suitable and practical manner whether the singer corresponding to the image data and the karaoke performance of which music is being captured.

  In addition, when the image data includes face images of a plurality of persons, the singer specifying unit 160 selects a singer corresponding to the image data based on an image corresponding to a microphone included in the image data. Therefore, the singer corresponding to the image data can be specified in a suitable and practical manner with respect to the image data captured by the mobile phone 30.

  In addition, when the imaging data includes a plurality of human face images, the imaging storage unit 166 uses the relationship stored in the SNS database 108 based on the human face images included in the imaging data. The identification information of all persons included in the imaging data is acquired, and the imaging data is associated with the acquired identification information of all persons and the music specified by the music specifying means 164 to the server device 16. Since it is stored, the image data captured by the mobile phone 30 is stored in association with all the users who have performed karaoke performance corresponding to the image data, thereby further improving the convenience of the image data. Can be improved.

  The preferred embodiments of the present invention have been described in detail with reference to the drawings. However, the present invention is not limited to these embodiments, and may be implemented in other modes.

  For example, in the above-described embodiment, the touch panel display 134 uses the image data stored in the SNS database 108 in association with the same music as the music specified by the music specifying unit 164 as the combination candidate of the co-starring video. However, the present invention is not limited to this, and can be implemented in another embodiment. For example, a user whose video data corresponding to the same music as the music specified by the music specifying means 164 is stored in the SNS database 108 is searched, and the searched user is used as a friend registration candidate related to the SNS. The music specifying means 164 may be displayed on the touch panel display 134 so as to be selectable, or the singer's music corresponding to the music specified by the music specifying means 164 may be displayed on the touch panel display 134 so that the music can be selected. It is conceivable to perform various controls based on the music specified by.

  In the above-described embodiment, the control for performing the setting for combining the image data captured by the digital camera 144 of the mobile phone 30 and the moving image data stored in the SNS database 108 as the co-starring moving image has been described. Video data (same music) transmitted in real time from a mobile phone that shoots corresponding to the same music by matching the corresponding music with image data taken by another mobile phone And the image data captured by the digital camera 144 of the mobile phone 30 may be combined and displayed as shown in FIG. 11 described above. Further, the moving image data captured in real time by the mobile phone 30 and the moving image data recorded (recorded) in the SNS database 108 in advance may be combined as a co-starring moving image. Since it is necessary to synchronize the timing of the video data related to the co-starring, the playback elapsed time and speed control information (music speed) specified by the music specifying means 164, the key height, etc. It is desirable to set the moving image data recorded in advance. Alternatively, the recorded music data having the same music speed and key height as the music being played is searched, the searched video data is displayed as a thumbnail as a co-star video candidate, and the elapsed playback time of the music is recorded. A mode of setting to already-moved video data is also conceivable. In addition, the setting of the co-star video does not necessarily have to be a predetermined combination of video data from the start of performance to the end of performance through the corresponding music. For example, in a mode in which division information (for example, information for dividing moving image data into a plurality of divisions such as every predetermined time) is stored in advance in each moving image data, only a predetermined division may be used as the co-starring moving image based on the division information. It is also possible to perform control such as setting to be output in synchronization with the moving image data, or performing co-starring moving image setting with different moving image data for each section.

  In the above-described embodiment, the music selection input control means 154 provided in the mobile phone 30 performs music selection input to the karaoke device 16 via the communication line 20 via the server device 36. However, for example, the music selection input to the karaoke device 16 may be performed by infrared communication, short-range wireless communication, or the like without passing through the communication line 20 (that is, without passing through the server device 36).

  In the above-described embodiment, the mobile terminal device in the karaoke system 10 is exemplified by the configuration including the mobile phone 30 and the tablet 32 as a so-called smartphone, but other devices may be considered as the mobile terminal device. For example, a digital video camera having a communication function, a consumer game machine, or a portable music player having a communication function and a touch panel display may be provided as a portable terminal device.

  In addition, although not illustrated one by one, the present invention is implemented with various modifications within a range not departing from the gist thereof.

  10: Karaoke system, 16: Karaoke device, 18: Server device, 20: Communication line, 30: Mobile phone (portable terminal device), 32: Tablet (portable terminal device), 108: SNS database (customer database), 144: Digital camera (imaging device), 160: Singer identification means, 164: Music identification means, 166: Imaging storage means, 168: Imaging list display means, 170: Music identification means, 172, 180: Imaging data, 182: Face image

Claims (5)

A karaoke device that performs performance output of music selected from a large number of music, a portable terminal device that includes an imaging device, and a server that transmits and receives information to and from the portable terminal device via a communication line A karaoke system comprising a device,
A music specifying means for specifying a music for which performance output has been performed by the karaoke device at the time of shooting of imaging data by the imaging device of the portable terminal device;
Imaging storage means for causing the server device to store the imaging data captured by the imaging device of the portable terminal device in association with the music specified by the music specifying means;
A customer database that stores the image corresponding to the user of the karaoke device and the identification information of the user in association with each other;
A singer identifying means for identifying a singer corresponding to the imaging data based on an image included in the imaging data captured by the imaging device of the portable terminal device based on the relationship stored in the customer database ; , Prepare ,
The music specifying unit is configured to perform a performance output by the karaoke device, with a singer specified by the singer specifying unit as a music selection input subject from a plurality of songs selected and input to the karaoke device. To identify it as a song
The imaging storage means associates the imaging data taken by the imaging device of the portable terminal device with the singer specified by the singer specifying means and the music specified by the music specifying means, and the server A karaoke system characterized by being stored in a device . The portable terminal device includes imaging list display means for selectively displaying imaging data stored in the server device in association with the same music as the music specified by the music specifying means. Item 3. The karaoke system according to item 1 . The customer database stores the face image corresponding to the user of the karaoke device and the identification information of the user in association with each other,
The singer specifying means, based on the relationship stored in the customer database, singing corresponding to the imaging data based on the face image of the person included in the imaging data captured by the imaging device of the portable terminal device The karaoke system according to claim 1 or 2 , wherein a karaoke system is specified. The singer specifying means specifies a singer corresponding to the imaging data based on an image corresponding to a microphone included in the imaging data when the imaging data includes face images of a plurality of persons. The karaoke system according to claim 3 . When the imaging data includes a plurality of human face images, the imaging storage means stores the imaging data based on the human facial images included in the imaging data based on the relationship stored in the customer database. The identification information of all persons included is acquired, and the imaging data is stored in the server device in association with the acquired identification information of all persons and the music specified by the music specifying means. Item 5. The karaoke system according to item 3 or 4 .