KR101295862B1 - Karaoke apparatus and method thereof for providing augmented reality images - Google Patents

Abstract

According to the present invention, a user photographs a predetermined image corresponding to a song selected by a camera, plays back accompaniment music of a song selected by a karaoke device using the photographed image, and in synchronization with this, sings a song photographed by the camera. A karaoke device that can synthesize the user's actual video and the virtual image of the singer or character corresponding to the selected song and show an augmented video as if the user and the singer are singing together. It's about how.

Description

Karaoke device and method for providing augmented reality video {KARAOKE APPARATUS AND METHOD THEREOF FOR PROVIDING AUGMENTED REALITY IMAGES}

The present invention relates to a karaoke apparatus and a method for providing an augmented reality image, and more particularly, after the user selects a song, the actual image of the user singing a song taken by the camera, the singer corresponding to the selected song or The present invention relates to a karaoke apparatus and a method for providing an augmented reality image that shows an augmented image as if a user and a singer are singing together by synthesizing a virtual image of a character.

The karaoke apparatus used in the conventional karaoke stores a plurality of accompaniment music data, reproduces the music data of the selected song according to the user's selection, and provides lyrics and image data to the image device.

To play the accompaniment of a song selected by a user on a karaoke device, find a song to sing using a song guide book provided in karaoke or a song selection list attached to the karaoke wall, and then select the number of songs selected on the karaoke device. You must enter the number directly or use the remote control to enter the number of the selected song. However, the letters of the song book provided in the karaoke is small, the lighting of the karaoke is dark, it was inconvenient to find the number of the number of the desired song and enter the number directly on the karaoke device.

In addition, when a user selects a song in karaoke and plays back the accompaniment music data of the selected song, the image of the karaoke device is always displayed on the displayer regardless of the song that is recorded on the videotape or compact disc and is displayed. It is used only as a screen, and lyrics are displayed on its video. However, the conventional video output method as described above has a problem in that an image irrelevant to the selected song is output to the display device so that the entertainment of the song or the atmosphere suitable for the song cannot be produced.

As another method of outputting an image of a karaoke apparatus, one song and one image corresponding thereto are recorded on the same medium or file, and sound, lyrics, and image are simultaneously output on one frame. This method does not need to load the lyrics on the video as described above, but the video that is suitable for the song has the advantage that the output, but the video for each song to be recorded in each file or media, so a large number of songs must be stored Since it requires memory, it is not suitable for all songs.

Therefore, the conventional karaoke device merely stays in the function of providing music and lyrics for accompaniment of the song and thus cannot play the role of multimedia. Therefore, the function of the karaoke device is fixed without being differentiated, and users get tired of the number of karaoke times. There is a problem that it is easy to feel.

On the other hand, the prior art has a computer graphics method, that is, augmented reality technology that synthesizes virtual objects in a real environment and looks like objects existing in the original environment.

Augmented reality technology, unlike the existing virtual reality that targets only virtual space and objects, synthesizes virtual objects based on the real world and provides additional information that is difficult to obtain only in the real world. However, in order to reinforce additional information in the real world through augmented reality technology, augmented reality data should be authored. In general, authoring of augmented reality data is performed by recognizing a predefined marker and matching and displaying a person's object defined in the marker.

The augmented reality data authored in this general manner is produced to display only one predefined image. Therefore, there is a problem that does not reflect the various needs of the user.

The present invention has been made to solve the above problems, by using augmented reality technology, synthesized video of the image of the singer or character corresponding to the song selected by the user and the image of the actual image of the user singing The present invention provides a karaoke apparatus and a method for providing song selection and augmented reality images through an image, which enables the user to obtain a visual effect as if they are singing a song with the singer instead of singing. The purpose is.

To this end, the karaoke apparatus according to the present invention, a marker recognition unit for recognizing the marker image obtained by photographing the marker attached to the karaoke wall or floor; A search unit for searching accompaniment information and image data corresponding to a song selected by a user; A rendering unit for rendering the image data retrieved by the searching unit and loading the virtual image at the position of the marker recognized by the marker recognizing unit; And a synthesizer for composing an augmented reality image by synthesizing a virtual image rendered by the rendering unit and a real image of a user photographed by a camera in real time.

The karaoke apparatus may include a memory unit for storing a predetermined image, a marker image, accompaniment information, and image data; An operation unit for inputting a song selected by a user into a karaoke device; A sound reproducing unit reproducing sound data of the retrieved accompaniment information; The apparatus may further include an image display device displaying the augmented reality image created by the synthesis unit.

The marker recognizing unit may further recognize a predetermined image corresponding to the selected song, and the predetermined image may be an image stored in advance in the memory unit by determining that the provider corresponds to the accompaniment information.

The marker image is characterized in that it has location information on which a virtual image is loaded.

The accompaniment information includes sound data and lyrics, which are karaoke accompaniment music, and the image data includes a corresponding singer image, a character image, or a video.

According to the present invention, a method for providing a song selection and augmented reality image, the marker recognition step of allowing the karaoke device to recognize the marker image in the captured image by photographing the marker attached to the floor or wall of the karaoke with a camera; A search step of searching accompaniment information and image data corresponding to the song selected by the user; A rendering step of checking whether there is image data corresponding to the selected song, and if the corresponding image data exists, rendering the corresponding image data and loading the virtual image on the recognized marker position; And synthesizing the augmented reality image by synthesizing the real image photographed with the rendered virtual image by photographing a user singing by a camera in real time.

The method includes: reproducing sound by transmitting sound data of the retrieved accompaniment information to a sound reproducing unit; The method may further include transmitting an augmented image created in the synthesizing step to the image display device and displaying the image in synchronization with the sound reproducing step.

The method may further include an input step of inputting a song selected by the user to the karaoke device through the operation unit of the remote controller or the karaoke device before the searching step.

The method may include: an acquiring step of acquiring an image obtained by photographing a predetermined image corresponding to a song selected by a user before the searching step; The method may further include a recognition step of sending the acquired image to the marker recognition unit to recognize the karaoke apparatus through matching with the image previously stored in the memory unit.

If there is no image data corresponding to the selected song, photographing a user singing by a camera in real time, and inserting lyrics of accompaniment information into the actual image to be photographed, and displaying the same on a video display device; It features.

In the recognition step, it is possible to add a marker by the user's selection.

The added marker is characterized in that it corresponds to the genre or singer of the song selected.

A computer-readable recording medium having recorded thereon a program according to the present invention stores a program for executing the processes of the above methods of providing song selection and augmented reality images.

According to the present invention, when a user inputs a song selected to a karaoke device, the karaoke recognizes a predetermined image corresponding to the song selected through the camera to input the song selected into the karaoke device without a separate manual input process. By doing so, there is an effect of providing convenience to the user.

In addition, the camera recognizes a marker such as a karaoke floor or a wall, and changes the singer or character according to the selected song with respect to one recognized marker, and shoots the singer or character (virtual image) corresponding to the selected song. By displaying the composite screen (augmented reality video) of the singer (real video), the user can bring a visual effect as if the user is actually singing with the singer.

In addition, by recognizing the marker attached to the floor or the wall of the karaoke by the user's selection through the camera, by adding a variety of effects to the recognized marker, there is an effect that can stimulate the excitement of the song or create an atmosphere suitable for the song. .

1 is a block diagram of a karaoke apparatus for providing a song selection and augmented reality image according to a preferred embodiment of the present invention.
2 is a flow chart of a method for providing song selection and augmented reality video according to a preferred embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The configuration of the present invention and the operation and effect thereof will be clearly understood through the following detailed description. Prior to the detailed description of the present invention, the same components will be denoted by the same reference numerals even if they are displayed on different drawings, and the detailed description will be omitted when it is determined that the well-known configuration may obscure the gist of the present invention. do.

Referring to FIG. 1, a karaoke apparatus providing song selection and augmented reality images according to an exemplary embodiment of the present invention will be described.

Prior to this description, the markers used in the augmented reality system will be briefly described.

A marker is any object that can be easily recognized by computer vision technology. For example, a marker is a flat pattern written on a black background or a geometric object with an unusual color. A marker is a logo or a photo ( concept images, characters, symbols, etc., including photographs or pictures or portions of images.

On the other hand, the augmented reality system renders a virtual image by using a three-dimensional perspective projection method such as a real camera projecting the data modeled in three dimensions in order to match the virtual image to the real image, and then the real image and virtual Shows a composite of the graphics. In this case, the augmented reality system has a registration problem of finding out the exact position and orientation of the virtual objects in the 2D screen in order to synthesize the virtual graphic object at the exact position of the live image. To solve this problem, three-dimensional coordinates of a point in the real world (for example, a position where a virtual object is to be drawn) are required, and this coordinate should be a coordinate value based on a camera.

Therefore, the augmented reality system needs to obtain the relative three-dimensional coordinates of the camera with respect to a certain point or object in the real world in the image of the camera, and uses a marker to obtain the three-dimensional coordinates.

Karaoke device that provides a song selection and augmented reality image through marker recognition according to a preferred embodiment of the present invention is a microphone (not shown) to which a predetermined image 10, a marker 20, the first camera 100 is attached; , A second camera 200, a karaoke device 300, an image display device 400, and a speaker (not shown) are required.

The microphone to which the first camera 100 is attached serves to collect voices of a singer and a user, and output the collected voices through a speaker.

The first camera 100 is attached to a microphone, which is free to move, and plays a role of photographing a predetermined image displayed on a song selection list provided by karaoke. Each of the predetermined images is an image set to correspond to a specific song by the provider, and may be, for example, in the form of a song number of a song, a title of a song, a barcode, or a marker.

The second camera 200 is attached to a wall located on the image display device 400 to capture a marker and a user who sings on the floor of a karaoke room in real time.

Although two cameras are used in the present embodiment, a predetermined image displayed on a song selection list provided by karaoke, a marker attached to the karaoke floor or a wall, and a user singing a song are photographed to transmit shooting information to the karaoke device 300. If possible, there is no limit to the type, type or number of cameras.

The karaoke device 300 may include a marker recognition unit 310, a memory unit 320, a search unit 330, a sound reproduction unit 340, a rendering unit 350, a synthesis unit 360, and an operation unit (not shown). And so on.

The marker recognition unit 310 is connected to the first camera and the second camera by wire or wirelessly, recognizes the predetermined image 10 taken by the first camera, and finds a marker in the image taken by the second camera. It has a role to recognize.

The database of the memory unit 320 stores marker images, accompaniment information, image data, and a program for controlling the operation of the karaoke apparatus by the CPU. The accompaniment information corresponds to the predetermined image as sound data and lyric of the karaoke accompaniment music, and the image data includes a corresponding singer image, a character image, or a video.

The searcher 330 searches for accompaniment information corresponding to a predetermined image recognized by the marker recognizer from a database stored in the memory 320 to extract matching accompaniment information.

The sound reproducing unit 340 plays a role of reproducing sound data among the extracted accompaniment information.

The rendering unit 350 renders the image data corresponding to the extracted accompaniment information and loads the virtual image at the position of the marker recognized by the marker recognition unit. Here, rendering refers to a process of creating a 3D model using a 3D computer graphics tool and converting it into a 2D picture for use in a movie image.

The synthesizing unit 360 synthesizes the virtual image and the actual image photographed by the second camera and authors an augmented reality image that seems to be sung by a user and a corresponding singer.

The video display device 400 may be configured of a liquid crystal display panel or a CRT. The image display device 400 serves to provide an augmented reality image or an actual image to a user in synchronization with the reproduction of accompaniment information of the sound reproducing unit 340.

The operation unit (not shown) includes a keypad, a jog, a dial, and the like, and can be operated by a user.

Referring to FIG. 2, a method of selecting a song through a user operation and providing an augmented reality image through a karaoke device according to a preferred embodiment of the present invention will be described.

The user selects from a list of song selections provided by karaoke (step S10).

Recognizing the selected song to the karaoke device, there is a method for the user to use the operation unit of the remote control or karaoke device, and a method using a predetermined image of the selected song.

The method of using the operation unit of the remote control or karaoke apparatus is a method of directly inputting a song number or song title selected by the user by directly operating the operation unit of the remote control or karaoke apparatus.

The method of using a predetermined image is a method of recognizing an image by capturing a predetermined image displayed on a song selection list of a song selected by a user with a camera mounted on a microphone and sending the photographed predetermined image to a marker recognition unit. Recognition of the image is performed by determining whether a predetermined image photographed matches with an image previously stored in the memory unit 320.

The search unit searches for accompaniment information corresponding to the recognized song from the memory (step S20). When a matching song is found, sound data of accompaniment information of the found song is transmitted to the sound reproducing unit.

In addition, the search unit checks whether there is video data corresponding to the accompaniment information of the found song (step S30).

When the image data corresponding to the retrieved song is stored in the memory, the augmented reality image using the image data is authored by the karaoke device (step S40).

In order to author an augmented reality image in a karaoke device, first, a user who sings with a camera attached to the monitor wall and a marker attached to the floor are photographed. This shooting acquires the actual image and the marker image of the user singing.

The obtained marker image is sent to the marker recognition unit, and the marker is recognized through a process of determining whether any of the objects included in the captured image match the marker image previously stored in the memory 320. The marker recognition unit detects position information of the recognized marker according to the marker position information previously stored in the memory.

When image data corresponding to the retrieved song is stored in the memory, corresponding image data, that is, an image or a video of a singer or a character, is selected and rendered by the rendering unit, and a virtual image is loaded at the position of the marker. The synthesized unit synthesizes the acquired virtual image and the actual image photographed (step S40), inserts lyrics of accompaniment information into the synthesized image, authors an augmented reality image, and synchronizes the reproduction of the acoustic data sent to the sound reproducing unit. And display on the video display device (step S50).

If the image data corresponding to the retrieved song is not stored in the memory, the song lyrics of accompaniment information is inserted into the actual image image photographed by the synthesis unit, and the image is displayed in synchronization with the reproduction of the sound data sent to the sound reproduction unit. It is displayed on the device (step S60).

The user can watch an image as if singing with a singer or a character while singing. When a song is changed, the corresponding character or singer is also changed, and thus, a single marker has a conversion effect to various characters.

In addition to the above markers, a second marker, a third marker, or the like may be attached to produce various effects such as various backgrounds, firecracker images, and soap bubble images in the augmented reality image. The second marker and the third marker may be matched to each song, may be matched to a song genre, or may be matched for each singer. Alternatively, additional markers may be made detachable to the karaoke wall or floor to allow the user to select. At this time, the marker provider should store in advance a memory image with each marker image and an effect image corresponding to the marker image.

Hereinafter, the case where the marker which can be attached and detached by a user is further used is demonstrated as an example.

First, the user selects markers corresponding to an image to be added to the augmented reality image, for example, an organ and a firecracker popping image, to place additionally selected markers within a shooting range of the second camera.

Next, the user selects a song and inputs it directly into the karaoke apparatus, or recognizes the karaoke apparatus using a predetermined image to play back the accompaniment of the song. The accompaniment of the selected song is reproduced, and in synchronization with this, an image of a popping firecracker is displayed on the video display screen along with the singer of the selected song and the user playing the organ and singing.

The present invention can be applied to various apparatuses by implementing a method for providing song selection and augmented reality images using the markers described above by using a software program and recording them on a computer-readable recording medium.

For example, the recording medium may be a hard disk, a flash memory, a RAM, a ROM, or the like embedded in each reproduction apparatus, or an external optical disk such as a CD-R or a CD-RW, a compact flash card, a smart media, a memory stick, have.

In this case, the program recorded on the computer-readable recording medium may include a marker recognition step of allowing the karaoke apparatus to recognize the marker image in the captured image by photographing the marker attached to the floor or wall of the karaoke using a camera; A search step of searching accompaniment information and image data corresponding to a song selected by a user from a memory; A rendering step of checking whether there is image data corresponding to the selected song, and if the corresponding image data exists, rendering the corresponding image data and loading the virtual image on the recognized marker position; Can be programmed to perform a synthesis step of real-time shooting of a user singing by a camera, synthesizing the actual image being photographed with the rendered virtual image, and inserting lyrics of the accompaniment information into the synthesized image to author the augmented reality image. have.

The foregoing description is merely illustrative of the present invention, and various modifications may be made by those skilled in the art without departing from the spirit of the present invention. Accordingly, the embodiments disclosed in the specification of the present invention are not intended to limit the present invention. The scope of the present invention should be construed according to the following claims, and all the techniques within the scope of equivalents should be construed as being included in the scope of the present invention.

When the karaoke user selects and plays back the accompaniment using the present invention, the user does not have to manually enter a song number or title of the song to play the accompaniment by photographing and playing the accompaniment. An augmented reality image that provides an effect may be provided to a user to induce interest of the user.

300: Karaoke device
310: marker recognition unit
320: memory unit
330: search unit
340: sound playback unit
350: renderer
360: synthesis unit
400: video display device

Claims (13)

In the karaoke apparatus for providing an augmented reality image corresponding to the song selected in the karaoke,
A marker recognition unit for recognizing a marker image obtained by photographing a marker attached to the bottom of the karaoke;
A search unit for searching accompaniment information and image data corresponding to a song selected by a user;
A rendering unit for generating a virtual image by rendering the image data and loading the virtual image at a position of the recognized marker;
Synthesis unit for authoring an augmented reality image by synthesizing the virtual image and the real image of the user taken by the camera in real time
Including;
When the additional marker selected by the user and detachable by the marker recognition unit is recognized, the synthesis unit provides an augmented reality image, characterized in that for synthesizing the effect image corresponding to the additional marker to the augmented reality image Karaoke device. The method of claim 1,
A memory unit for storing the marker image, accompaniment information, and image data corresponding to each song;
An operation unit for inputting a song selected by a user into a karaoke device;
A sound reproducing unit reproducing sound data of the retrieved accompaniment information;
An image display device for displaying an augmented reality image authored by the synthesis unit
Karaoke device providing an augmented reality image further comprising. The apparatus of claim 2, wherein the marker recognizing unit further recognizes a song image corresponding to the selected song,
The song image is an image stored in advance in the memory unit by determining that the song image corresponds to accompaniment information, and includes a song number of a song, a title of a song, a barcode or a marker. . The karaoke apparatus of claim 1, wherein the marker image has location information on which a virtual image is loaded. The karaoke apparatus according to any one of claims 1 to 3, wherein the accompaniment information includes sound data and lyrics that are karaoke accompaniment music, and the image data includes a singer image, a character image, or a video. . In the method for providing an augmented reality image corresponding to the song selected in the karaoke,
A marker recognition step of confirming a marker position by recognizing a marker image in a captured image by photographing a marker attached to the bottom of the karaoke with a camera;
A search step of searching accompaniment information and image data corresponding to the song selected by the user;
Checking whether there is image data corresponding to the selected song, and if there is corresponding image data, rendering the corresponding image data to generate a virtual image and loading the generated virtual image on the recognized marker position. A rendering step;
A synthetic step of authoring an augmented reality image by real-time shooting a user singing by a camera, synthesizing the actual image being photographed and the rendered virtual image
Including;
In the marker recognition step, if an additional marker selected and detachable by the user is recognized, an augmented reality image providing method comprising synthesizing an effect image corresponding to the additional marker with the augmented reality image in the synthesis step. . The method according to claim 6,
A sound reproduction step of sending sound data of the retrieved accompaniment information to a sound reproduction unit for reproduction;
The image reproducing step of sending the augmented image authored in the synthesizing step to the image display device and displaying in synchronization with the sound reproducing step
Augmented reality image providing method comprising a. The method according to claim 6,
Input step of inputting the song selected by the user to the karaoke device through the operation unit of the remote control or karaoke device before the search step
Augmented reality image providing method further comprising. The method according to claim 6,
An acquiring step of acquiring an image obtained by photographing a song image corresponding to a song selected by a user before the searching step;
Recognition step of sending the acquired image to the marker recognition unit to recognize the karaoke device by matching the image previously stored in the memory unit
Augmented reality image providing method that further comprises. The method according to claim 6,
If the image data corresponding to the selected song does not exist, real-time recording of the user singing by the camera, inserting the lyrics of the accompaniment information into the actual image to be displayed on the video display device
Augmented reality image providing method further comprising a. delete The method of claim 6, wherein the additional marker corresponds to a genre or a singer of the selected song. A computer-readable recording medium having recorded thereon a program for executing the process according to any one of claims 6 to 10 and 12.

Images