KR101802906B1 - Augmented reality system interacting with exterior contents player - Google Patents

Abstract

The present invention relates to an augmented reality system interacting with external content, which enables natural singing without a necessity of looking at a media player continuously and enables more convenient and enjoyable singing. For the above, the augmented reality system in a karaoke provided by the present invention includes: a recognition camera which is installed to face a karaoke player; a text recognition unit which recognizes text of lyrics displayed on the karaoke player based on image information received through the recognition camera; a recognizer which has a recognition communication unit of transmitting the recognized text to player-connected glasses and a glasses communication unit which receives information transmitted from the recognition communication unit; and the player-connected glasses which have an augmented reality display unit of displaying the text received from the glasses communication unit as an augmented reality image in front of eyes. Therefore, the present invention enables convenient singing and natural gesture without a necessity of looking at the karaoke player screen continuously during singing.

Description

Augmented reality system interacting with external contents player {

 BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an augmented reality system for interfacing with an external content player, and more particularly, to an augmented reality system capable of singing songs naturally without looking at a karaoke player continuously and singing songs more conveniently and happily will be.

People are enjoying audio, video and games through a variety of media players. DVD, TV, MP3 player and Karaoke media player. Among these media players, a karaoke room is installed in each room, and a karaoke player is installed in each room so that a user can sing and play a desired karaoke.

1, the karaoke player includes a microphone 2 and a speaker 4, a video information section 6 in which lyrics and music are displayed, a music information section 8, a control section 9 And an earphone 10 connected to the microphone 2 by wires or wires, and a play unit 20 for playing a song through the earphone.

The lyrics and background screen images stored in the image information section are synchronized when the song is reproduced and displayed for the user to view through the screen 3.

The karaoke player will sing while watching the accompaniment sent through the speaker and the lyrics displayed on the screen. In addition, according to the above-described configuration, separate lyrics and music driven by the play section are simultaneously transmitted to the earphone, so that the user can listen to the song through the earphone while watching the lyrics displayed on the screen and sing the song, There are advantages to be able to.

However, according to the conventional karaoke player, if the user can not memorize the lyrics perfectly, he should sing while watching the karaoke player screen, and even if he / she listens to the song through the earphone, have.

SUMMARY OF THE INVENTION The present invention has been made to solve the above-mentioned problems, and it is an object of the present invention to allow a user to sing a song naturally without interruption of housework without having to constantly look at a player in a karaoke room.

It is another object of the present invention to enable easy selection without searching for a song list book.

According to an aspect of the present invention, there is provided a karaoke system comprising: a recognition camera installed to face a karaoke player; A character recognition unit for recognizing characters displayed on the karaoke player based on image information received through the recognition camera; A recognizer having a recognizing communication unit for transmitting recognized characters to the player interlocking glass; and a glass communication unit for receiving information transmitted from the recognizing communication unit; And a player interlocking glass having an augmented reality display unit for displaying characters received from the glass communication unit as augmented reality images in front of the eyes.

The player interlocking glass stores a song name and lyrics in a storage unit, and the recognizer recognizes the song name from the karaoke player and transmits the song name to the player interlocking glass. The player interlocking glass compares the song name transmitted from the recognizer with the stored song name So that a tune currently being played can be detected.

Wherein the player interlocking glass stores the sound source file in a storage unit, the waveform of the sound source file is matched with the stored lyrics, and the control unit of the player interlocking glass controls the sound wave of the music to be played through the karaoke player, It is preferable to detect the position of the tune currently being played.

The character recognition unit recognizes the timing at which the image received through the recognition camera changes and generates a start preparation signal, and the recognition communication unit can transmit the start preparation signal to the player interlocking glass.

Here, the character recognizing unit recognizes a lyric part in which the lyric is located, and recognizes that the color of the lyric is changed, generates a progress position signal, and the recognition communication unit transmits the progress position signal to the player interlocked glass .

According to another aspect of the present invention, there is provided a karaoke system comprising: a karaoke player provided with a sound source data wireless communication unit; And a glass camera for receiving the karaoke player image; A character recognition control unit for recognizing a character displayed on the karaoke player based on image information received through the glass camera; A glass communication unit for receiving information transmitted from the sound source data wireless communication unit; A storage unit for storing information through the glass communication unit; And a player interlocking glass having an augmented reality display unit for displaying a lyrics of a sound source played on the basis of data received from the sound source data wireless communication unit as augmented reality images before the eyes.

Preferably, the karaoke player displays the player specific code on an image, and the character recognition controller recognizes and pairs the player specific code received through the glass camera.

The character recognition control unit recognizes a song name or an artist displayed by printing or video, displays the artist name or the artist number on the augmented reality display unit, and transmits the selected song name or artist to the karaoke player.

The printing or image is provided with a music code including at least music name information, and the character recognition control unit analyzes the music code to extract the music title, reads the music title corresponding to the music title or the music title stored in the storage unit, It is preferable to display it on the augmented reality display unit.

The storage unit stores a virtual stage image in which a virtual stage facility is displayed, and the augmented reality display unit displays the virtual stage image.

The storage unit stores the name of the played music and the date, and the controller displays the recommendation list on the augmented reality display unit based on the name and date of the played music.

Preferably, the controller displays a recommendation list through the augmented reality display unit, and when the user selects one of the recommendation lists, the controller displays the words corresponding to the song name through the augmented reality display unit.

According to the present invention, it is not necessary to keep watching the karaoke player screen continuously while singing, so it is possible to sing very comfortably and natural gestures are possible.

In addition, there is an advantage that song selection can be made very easy.

1 is a block diagram showing a karaoke system according to the prior art;
FIG. 2 is a diagram showing an augmented reality system in a karaoke system according to the present invention; FIG.
3 is a diagram showing the configuration of a player interlocking glass (only half of the player interlocking glass is shown and the other has a symmetrical appearance);
4A to 4D illustrate examples of start signals displayed on a karaoke player;
5 is an explanatory diagram for explaining a tempo change recognition process (A is a recognized tempo and B is a basic tempo);
FIG. 6 is a diagram illustrating an augmented reality system in a karaoke system according to another embodiment of the present invention; FIG.
7 is an explanatory view for explaining a method of selecting a music piece using a player interlocking glass;
8 is an exemplary diagram for explaining another method of selecting a music piece using a player interlocking glass;

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A detailed embodiment of the present invention will be described with reference to the drawings.

Referring to FIG. 2, an augmented reality system in cooperation with an external content player according to the present invention includes a recognizer 200 and a player interlocking glass 300.

3, the player interlocking glass 300 includes a glass camera 310, a glass character recognition unit 320, an augmented reality display unit 330, a touch panel switch 340, a glass communication unit 350, (Not shown), and a glass microphone (not shown).

The glass camera 310 is installed toward the front of the player interlocking glass and shoots a forward image with a resolution enough to recognize the text displayed on the display screen or the poster at a suitable distance. The glass camera 310 may be installed not only on the left and right but also on the center of the player interlocking glass 300. It is preferable that the glass camera has a resolution of 5,000,000 pixels or more and takes an image of the front side.

The glass character recognition unit 320 may be formed of a chip serving as a control unit and is configured to be able to recognize characters, figures and the like by analyzing an image photographed by the glass camera.

The augmented reality display unit 330 displays the necessary information together with the actual view in front of the user, and can be configured on the same principle as the HMD display.

The touch panel switch 340 has upper, lower, left, and right buttons in the periphery, and a select button is located at the center. Using the touch panel switch, the user can select a desired portion of the screen displayed on the augmented reality display unit.

The glass communication unit 350 supports wired or wireless communication, and wireless communication can support Bluetooth or infrared (IR) communication.

In the storage unit, a song name, a singer, a lyric data, a sound source file, a waveform of a sound source file, and the like stored in a media player such as a karaoke player are stored in correspondence with each other and classified according to genres such as ballads, dances and hip-hop.

The recognizer 200 includes a recognition camera 210, a character recognition unit 220, and a recognition communication unit 230, which are installed facing the karaoke machine.

The karaoke player reads out the accompaniment data when the selected song is started, generates an accompaniment signal, converts the corresponding lyric data into a video signal having a bitmap font, and synthesizes it with a background screen and outputs it to a monitor.

At this time, the accompaniment signal may include time information and beat information, and a synchronization signal may be inserted to facilitate synchronization of the melody and the lyrics.

In the karaoke player, when a performance is started, an accompaniment signal is generated through a speaker, and a song name, a singer, a background screen, and the like are displayed through the monitor. In addition, as shown in FIGS. 4A to 4D, a start signal such as a number or a finger signal is generated through the monitor immediately before the start of the song (about 5 seconds before) to allow the user to know the timing to start the song.

The recognition camera 210 provided in the recognizer always photographs the monitor screen, and the character recognition unit 220 analyzes the captured image through the recognition camera to recognize characters or graphics.

The song information, such as the name of the song, the song number, etc. recognized by the character recognition unit is transmitted to the player interlocking glass 300 through the recognition and communication unit 230. Communication between the recognition and communication unit 230 and the player interlocking glass is preferably performed by wireless communication, and Bluetooth, Zigbee, or infrared communication can be used.

In the player interlocking glass, song information such as a song name, a song number, and the like is received from the recognizer through the glass communication unit 350, and the currently played song is compared with the song name and the song number stored in the storage unit. Then, the lyric data corresponding to the song name and the music number is converted into a video signal having a bitmap font, and displayed on the augmented reality display unit 330.

On the other hand, the start signal displayed on the monitor before the start of the song is changed in units of 1 second. The character recognition unit 220 recognizes the timing of the change of the image displayed on the monitor, generates a start preparation signal, To the player interlocking glass. That is, the character recognition unit detects an object changed every predetermined time before the start of a song, recognizes the object as a start signal, and generates a start preparation signal at a changed timing to transmit the start preparation signal to the player interlocking glass.

When receiving the start preparation signal, the player interlocking glass displays a start preparation signal through the augmented reality display unit. Then, the user can prepare for starting a song even if he is heading towards other colleagues without watching the karaoke player screen.

When the song is started, the karaoke player changes the color of the lyrics in progress according to the beat on the monitor, and displays the progress position. Then, the character recognition unit 220 of the recognizer senses a point where the color of the monitor head portion is changed and transmits a progress position signal to the player interlocking glass.

The control unit of the player interlocking glass displays the progress position of the lyrics on the augmented reality display unit using the progress position signal transmitted from the recognizer. In addition, the controller of the player interlocking glass detects the progress position received from the recognizer at predetermined time intervals and calculates the tempo of the song.

The player interlocked glass stores a basic tempo for each song. When the calculated tempo and the stored basic tempo do not match, the lyrics progress speed is adjusted by the calculated tempo.

5, the learner data is stored in the storing unit of the player interlocking glass in correspondence with the progress time of the basic tempo, and the progress position B is displayed on the augmented reality display unit at the speed of the basic tempo . Then, at the predetermined time interval, the progress speed is calculated using the progress position A transmitted from the recognizer, and if it differs from the basic tempo, the calculated progress speed is stored as the current tempo, and then the progress speed of the lyrics is advanced to the current tempo . During the song, this process is repeated to change the tempo.

Meanwhile, the player interlocking glass receives the sound waves of the music piece played through the karaoke music player and compares the sound waves with the waveform stored in the storage unit. Specifically, it is possible to detect the position of the music currently being played by comparing the peak point of a specific musical instrument (e.g. drum) with the sound of glass micro-received.

Since the musical instruments such as the drums produce notes in accordance with a certain beat, it is possible to calculate the progress speed as well as the progress position of the song through the waveform comparison.

According to this embodiment, since the karaoke player and the player interlocking glass are synchronized to display the lyrics, an environment in which the user can enjoy the song naturally without having to constantly look at the screen of the karaoke player is created.

Next, the following embodiment will be described with reference to FIG. However, detailed description of the same parts as those of the previous embodiment will be omitted.

In the present embodiment, the karaoke player is connected to the player interlocking glass 300 through the sound source data wireless communication unit 150. [ The player interlocking glass 300 includes a glass camera 310, a character recognition control unit, an augmented reality display unit 330, a touch panel switch 340, a glass communication unit 350, a glass storage unit (not shown) (Not shown).

Specifically, the karaoke player displays a player specific code on a monitor through a physical button or a selection screen on a monitor. The player specific code is different for each karaoke player, and may be formed in various ways such as a bar code, a cue code or a serial number.

At this time, when the user looks at the karaoke player, the glass camera 310 photographs the player unique code, and the player unique code is recognized by the character recognition control unit of the player interlocking glass. Then, the character recognition control unit adds a glass code, which is a unique code of the player-associated glass, to the recognized player specific code, and retransmits it to the player-associated glass through the glass communication unit 350 in a predetermined data format. The glass code is also set differently for each player-associated glass.

Through this process, the karaoke player and the player interlocking glass are paired, and the player interlocking glass is prevented from being mixed with the karaoke player of the next room.

In the karaoke player, song names, singers, lyrics data, sound source files, waveforms of sound source files, and the like are stored. When the songs are paired and the songs are selected, these data are transmitted to the player interlocking glass through the sound source data wireless communication unit. In addition, a synchronization signal is inserted between the lyrics, so that the player can control the lyrics progression speed in accordance with the synchronization signal in the player interlocking glass.

Specifically, the lyrics are stored corresponding to the progress time, and when the user changes the tempo quickly or slowly, the karaoke player transmits information on the changed progress speed to the player interlocking glass. Then, in the player interlocking glass, the progress speed is adjusted by the rate of the changed progress speed to change the progress speed of the lyrics.

On the other hand, by using the player interlocking glass, a song can be selected by itself without directly manipulating the karaoke player or operating a separate remote controller.

Specifically, when the selection button of the touch panel switch 340 provided in the player interlocking glass is pressed, the augmented reality display unit 330 displays the installed selection menu, and when the music selection menu is selected, And displays the song name and the number of songs stored in the memory. At this time, when the direction button is pressed, the other songs are highlighted and can be selected first, and when the select button is pressed in the first selected song, the second selected is selected and the second selected song and the artist data are transmitted to the karaoke player through the glass communication unit. Then, the karaoke player detects the song transmitted from the player interlocking glass and starts playing.

The controller of the player interlocking glass activates a timer when a primary selection is made in a specific music and, when the primary selection is maintained for a certain period of time or longer, the lyric data corresponding to the primary selected music is read out from the storage unit, Display. At this time, the lyrics are moved and the following lyrics are displayed sequentially so that the user can surely know which song is the first selected song before the second selection.

In addition, the player interlocking glass recognizes a song name or artist displayed as a print or image through a character recognition control unit. For example, a list of songs attached to a wall or a list of songs displayed on a monitor can be recognized. Then, the character recognition control unit displays a selection screen on the augmented reality display unit so that the recognized music title or artist can be selected. When the song is selected as described above, information on the selected music title or artist is transmitted to the karaoke player through the glass communication unit Respectively.

The image displayed by the print or image may not be a character such as Korean or English but may be provided with a promised music code such as a bar code or a cue code as shown in FIG. The music code may include information such as a song name, a song number, and the like.

Specifically, when the user looks at the music code of the poster attached to the wall, the music code is photographed by the glass camera, and the character recognition control unit analyzes the music code and extracts information such as a song name, a singer, and the like. Then, the character recognition control unit reads the lyrics corresponding to the recognized music title from the storage unit and displays it on the augmented reality display unit or displays a selection screen on the augmented reality display unit so that the corresponding music can be selected.

A virtual stage image in which a virtual stage facility is displayed around a user can be stored in the storage unit of the player interlocking glass as well as information about a song such as a song name, lyrics, and the like. When a song starts, the augmented reality display unit displays the virtual stage image The user can display various scenes such as a background screen, a moving picture, and an advertisement image.

On the other hand, the name of the played music, the number of musicians, and the date are stored in the storage unit of the player interlocking glass. In addition, the character recognition control unit compares the name of the song and the song number with the song list stored for each genre, and stores the song name and genre for each genre.

Then, when the user wears the player's interlocking glass, the character recognition control unit detects the number of plays of the music title played for a predetermined period of time, arranges them in a great number of order, displays the sorted music names and artists as a recommendation list to the user through the augmented reality display unit I will. At this time, since the name of the played music is stored according to the genre, it is also possible to select a preferred genre to view the recommendation list. Therefore, a user can easily select a song without viewing a karaoke book, so that the user can sing more freely and more fun.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention as defined in the appended claims. It will be understood that the present invention can be changed.

100: karaoke player 150: sound source data wireless communication unit
200: recognizer 210: recognition camera
300: player interlocking glass 310: glass camera
330: Augmented Reality Display

Claims (12)

A recognition camera installed facing the media player;
A character recognition unit for recognizing characters displayed on the media player based on the image information received through the recognition camera;
An recognizer having a recognizing communication unit for transmitting recognized characters to a player-associated glass; and
A glass communication unit for receiving information transmitted from the recognition / communication unit;
And a player interlocking glass having an augmented reality display unit for displaying characters received from the glass communication unit in front of the eyes as augmented reality images,
The player interlocking glass stores the music title and the lyrics in a storage unit,
The recognizer recognizes the music title from the media player and transmits the music title to the player interlocking glass,
Wherein the player interlocking glass detects a currently played music by comparing the music name transmitted from the recognizer and the stored music name, and displays the lyrics of the music in the augmented reality system. delete The method according to claim 1,
Wherein the player interlocking glass stores the sound source file in a storage unit, the waveform of the sound source file is stored by being matched with the stored lyrics,
Wherein the controller of the player interlocking glass detects the position of the music currently being played by comparing the sound wave of the music played through the media player with the waveform of the sound source file. The method according to claim 1,
Wherein the character recognition unit recognizes a timing at which the image received through the recognition camera changes and generates a start preparation signal, and the recognition communication unit transmits the start preparation signal to the player interlocking glass. Augmented reality system. 5. The method of claim 4,
The recognition unit recognizes the lyric unit in which the lyrics are located and recognizes that the color of the lyrics is changed, thereby generating a progress position signal. The recognition and communication unit transmits the progress position signal to the player interlocking glass Wherein the augmented reality system is interlocked with an external content player. A media player having a sound source data wireless communication unit; And
A glass camera for receiving the media player image;
A character recognition control unit for recognizing characters displayed on the media player based on image information received through the glass camera;
A glass communication unit for receiving information transmitted from the sound source data wireless communication unit;
A storage unit for storing information through the glass communication unit;
And a player interlocking glass having an augmented reality display unit for displaying the lyrics of the sound source played on the basis of the data received from the sound source data wireless communication unit in the form of augmented reality image in front of the eyes,
Wherein the media player displays a player specific code on an image and the character recognition controller recognizes and pairs the player specific code received through the glass camera. delete The method according to claim 6,
Wherein the character recognition control unit recognizes a song name or an artist displayed in print or video and displays the selected song name or artist on the augmented reality display unit so that the selected song name or artist is transmitted to the media player An augmented reality system linked to an external content player. 9. The method of claim 8,
The printing or image is provided with a music code including at least music name information, and the character recognition control unit analyzes the music code to extract the music title, reads the music title corresponding to the music title or the music title stored in the storage unit, And displays it on the augmented reality display unit. 9. The method of claim 8,
Wherein the storage unit stores a virtual stage image in which a virtual stage facility is displayed, and the augmented reality display unit displays the virtual stage image. The method according to claim 6,
Wherein the storage unit stores the name of the played music and the date, and the character recognition control unit displays a recommendation list on the augmented reality display unit based on the name and date of the played music. system. The method according to claim 6,
Wherein the character recognition control unit displays a recommendation list through the augmented reality display unit and displays the lyrics corresponding to the music title through the augmented reality display unit when the user selects one of the recommendation lists. An augmented reality system linked to an external content player.

Images