KR20160004914A - Method and device for playing multimedia - Google Patents

Abstract

Disclosed are a device and a method which include: acquiring music information on music played in an electronic device; acquiring story line on the music based on the music information; and displaying an image by acquiring one or more images based on the story line.

Description

[0001] METHOD AND DEVICE FOR PLAYING MULTIMEDIA [0002]

The present disclosure relates to a method and device for multimedia playback, and more particularly to a method and device for displaying an image associated with a music while the music is playing in an electronic device.

As devices for multimedia playback, electronic devices such as digital audio players, smart phones, tablet PCs, and smart TVs are widely used by users. Such electronic devices can reproduce various multimedia files such as music, image, and video.

Also, even if the multimedia files are not stored in the electronic device, streaming media may be reproduced in the electronic device via the Internet or the like. With advances in network technology, the types and amounts of digital multimedia that individuals can access through electronic devices are becoming increasingly vast.

When the music is played back on the electronic device, the electronic device sends an image or text associated with the music such as album cover, lyrics, and tag information of the music to the user so that the user can enjoy a more audiovisual experience and better understand the music Display.

However, there is a problem that the album cover of the music is not enough to enhance the user's understanding of the music.

In addition, since the lyrics and tag information of the music are displayed as text to the user, it is very cumbersome and time consuming to recognize and understand text on the part of the user.

The user can enjoy a video such as a music video of the music through an electronic device for a rich audiovisual experience and a deep understanding of the music. However, when a video file such as a music video of a corresponding music is played back in an electronic device, the video files generally have a large file size, which causes a problem of wasting disk capacity of the corresponding electronic device.

When these video files are played back streaming from an electronic device via the Internet or the like, the problem of wasting disk capacity can be solved. However, when the electronic device is connected to the Internet through a mobile communication network, a problem of increasing the burden of data usage on the user may arise. There is a problem that the video file may not be reproduced smoothly depending on the connection speed of the Internet and the connection state even if the load of the data usage is small to the user. If the video file does not play smoothly, the music that the user originally intended to listen to may not play smoothly, which may cause inconvenience to the user.

Accordingly, there is a need for a technique that can enhance the user's understanding of the music while providing a rich audiovisual experience to the user while the music is being played back in the electronic device.

Various embodiments may provide a method and device for displaying an image associated with the music while the music is being played on the electronic device.

As a technical means for achieving the above-mentioned technical object, a first aspect of the present disclosure provides a method for reproducing music, comprising: acquiring music information about music reproduced in an electronic device; acquiring a story line for the music based on the music information The method comprising the steps of: acquiring one or more images based on the storyline; and displaying the one or more images.

In addition, the music information may include meta information of the music, and the story line may be obtained based on the meta information.

In addition, the music information may include audio fingerprint information of the music, and the story line may be obtained based on the audio fingerprint information.

In addition, the step of acquiring one or more images based on the story line

And selecting one or more keywords based on the storyline, wherein the one or more images are obtained based on the selected one or more keywords.

In addition, the step of acquiring one or more images based on the story line

The method comprising: obtaining a plurality of candidate images based on the selected one or more keywords; and determining a priority between the plurality of candidate images based on a predetermined criterion, wherein the one or more images are selected from the plurality of candidate images And the multimedia data is obtained based on the priority.

In addition, the step of determining the priority among the plurality of candidate images based on a predetermined criterion

Wherein the priority order is determined by the similarity between the semantic information of the plurality of candidate images and the keyword selected based on the story line. Can be provided.

In addition, the story line may include one or more sentences composed of text, and the one or more images may be obtained based on the one or more sentences.

The one or more sentences constitute one or more paragraphs based on similarity, and the music is divided into one or more music segments corresponding to the one or more paragraphs, and the one or more images are displayed on the display The multimedia playback method according to the present invention can provide a multimedia playback method.

In addition, the story line may be a lyric of the music, the music information may include lyric information about the lyric, and the at least one image may be displayed based on the lyric information. have.

The method may further include acquiring emotion information based on the storyline, wherein the one or more images are acquired based on the emotion information.

In addition, the at least one image may be rendered based on the emotion information.

Further, the story line is divided into one or more paragraphs based on at least one of time information, spatial information, and person information, the music is divided into one or more music segments corresponding to the one or more paragraphs, May be displayed based on the one or more music segments.

The second aspect of the present disclosure also relates to a method for obtaining music information about music to be played back in an electronic device, acquiring a story line for the music based on the music information, And a display unit for displaying the at least one image.

Further, the music information may include meta information of the music, and the story line may be obtained based on the meta information.

Further, the music information may include audio fingerprint information of the music, and the story line may be obtained based on the audio fingerprint information.

Further, the control unit may select one or more keywords based on the story line, and the one or more images may be obtained based on the selected one or more keywords.

Further, the storyline may include one or more sentences composed of a series of texts, and the one or more images may be obtained based on the one or more sentences.

The controller may further acquire emotion information related to the music based on the storyline, and the one or more images may be obtained based on the emotion information.

Further, the story line is divided into one or more paragraphs based on at least one of time information, spatial information, and person information, the music is divided into one or more music segments corresponding to the one or more paragraphs, Is displayed on the basis of the at least one music segment.

In addition, the third aspect of the present disclosure can provide a computer-readable medium having recorded thereon a program for causing a computer to execute the method of the first aspect.

1 is a diagram illustrating an example in which an image related to music is displayed while music is being reproduced in an electronic device, according to an embodiment.
Figures 2 and 3 are block diagrams of an electronic device according to one embodiment.
4 is a flow diagram of a method for displaying an image associated with the music while the music is playing in the electronic device, according to one embodiment.
FIG. 5 is a flowchart illustrating a method of displaying an image related to a music using a keyword included in a story line of the music while the music is being reproduced in the electronic device according to an exemplary embodiment.
FIG. 6 is a diagram illustrating an example of displaying an image related to a corresponding music using a description of the music while the music is being reproduced in the electronic device according to an embodiment.
7 is a diagram illustrating an example in which an image related to a corresponding music is displayed using lyrics of the music while the music is being reproduced in the electronic device according to an embodiment.
FIG. 8 is a flowchart of a method of displaying an image related to music using a plurality of candidate images related to the music while the music is being reproduced in the electronic device, according to an embodiment.
9 is a flow diagram of a method for rendering and displaying an image associated with a music piece during playback of music in an electronic device, according to an embodiment.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily carry out the present invention. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. In order to clearly illustrate the present invention, parts not related to the description are omitted, and like parts are denoted by similar reference numerals throughout the specification.

As used herein, the terminology used herein is intended to encompass all commonly used generic terms that are possible in light of the functionality of the various embodiments, which may vary depending upon the intent or circumstance of the skilled artisan, the emergence of new technology, and the like. In addition, in certain cases, there may be a term arbitrarily selected by the applicant, in which case the meaning thereof will be described in detail in the description of the corresponding embodiment. Accordingly, the terms used in the present specification should be defined based on the meaning of the term, not on the name of a simple term, and on the contents throughout the specification.

The singular expressions include plural expressions unless the context clearly indicates otherwise. It is to be understood that the terms "comprises", "having", and the like in the specification are intended to specify the presence of stated features, integers, steps, operations, elements, parts or combinations thereof, But do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, or combinations thereof. In particular, the numbers set forth in the specification are intended to be illustrative, and the invention should not be limited by the numbers set forth in the specification.

Also, the terms " part, "" module," etc. in the specification mean units for processing at least one function or operation, which may be implemented in hardware or software or a combination of hardware and software.

Although "first "," second "and the like are used throughout the specification to describe various components, it goes without saying that these components are not limited by these terms. These terms are used only to distinguish one component from another. It goes without saying that the "first component" mentioned below may be a "second component" within the technical concept of the embodiment.

The terms used in this specification will be briefly described and the present invention will be described in detail.

In the entire specification, the electronic device is a device capable of playing music, a smart phone, a tablet, a mobile phone, a personal digital assistant (PDA), a media player, an audio player, a digital audio player, a portable multimedia player (PMP) A mobile terminal, a personal computer (PC), a laptop, a micro server, a global positioning system (GPS) device, a navigation device, a kiosk, an MP3 player, a smart TV, a digital camera and other mobile or non-mobile computing devices It does not.

Music throughout the specification may mean an audio file in which the music is recorded. The electronic device can reproduce the audio file by driving the multimedia reproduction application, and the music recorded in the audio file can be reproduced by reproducing the audio file. Such a multimedia playback application may be installed basically in an electronic device, or may be installed separately by a user.

The format of the audio file includes all the formats that can be reproduced via the electronic device. The format of the audio file may include an uncompressed audio format, a lossy compressed audio format, a lossless compressed audio format, and the like.

For example, the uncompressed audio format may be stored as an extension such as WAV or AIFF (Audio Interchange File Format), and the lossy compressed audio format may be stored as an extension such as MP3 or AAC (Advanced Audio Codec) The lossy compressed audio format can be stored as extensions such as Free Lossless Audio Codec (FLAC) or APE of Monkey's Audio. However, the format of the audio file is not limited to this and may have various extensions.

Throughout the specification, the multimedia playback device is an electronic device capable of playing music and / or displaying images, and all of the electronic devices capable of playing music and / or displaying images of the above-described electronic devices can be referred to as multimedia playback devices have.

Throughout the specification, the story line of the music means a narrative description of the music, and the story line of the music can consist of one or more images, one or more videos, and / or text for narrative explanation.

1 is a diagram illustrating an example in which an image related to music is displayed while music is played back in an electronic device, according to an embodiment.

Although music is reproduced in one and the same electronic device in Fig. 1 and an image related to the music is displayed, it is possible to realize that music is reproduced in one electronic device and images related to the music are displayed in another electronic device It is possible.

The multimedia playback application 100 may be driven in the electronic device, and the music may be played back. Here, the reproduction of the music means the reproduction of the audio file of the music.

1, the multimedia playback application includes an image window 110 for displaying an image related to the music to be played back, a controller 120 for controlling the multimedia playback application, and an information window for displaying music information about the music being played (130).

Referring to FIG. 1, when music is reproduced in an electronic device, images 110A, 110B, 110C, and 110D related to the music can be displayed through the image window 110. FIG. The images 110A, 110B, 110C, and 110D related to the music may be acquired based on the story line of the music.

The electronic device can acquire a story line of music from a server outside the electronic device.

By displaying the acquired image based on the storyline of the music, the user who enjoys the music can understand the music more easily. In addition, a rich audiovisual experience may be given to the user.

Since the electronic device only reproduces music and displays an image related to the music, it has an effect of consuming less resources of the electronic device than playing the video related to the music. In addition, when the electronic device is connected to the Internet through the mobile communication network and streaming music or images, there is an effect of reducing the burden of data usage by the user.

Figures 2 and 3 are block diagrams of an electronic device according to one embodiment.

As shown in Fig. 2, the electronic device 1000 includes a display portion 1210, and a control portion 1300. Fig. However, not all of the components shown in Fig. 2 are essential components of the electronic device 1000. The electronic device 1000 may be implemented by more components than the components shown in Fig. 2, and the electronic device 1000 may be implemented by fewer components than those shown in Fig.

The control unit 1300 typically performs the functions of the electronic device 1000 by controlling the overall operation of the electronic device 1000.

Specifically, the control unit 1300 acquires music information about music reproduced in the electronic device 1000 or the electronic device separate from the electronic device 1000.

When music is reproduced in the electronic device 1000, the music may be output through the sound output unit 1220 of the electronic device 1000. At this time, the audio file of the music may be stored in advance in the storage unit 1700 of the electronic device 1000 or may be received from the external server through the communication unit 1500 of the electronic device 1000 and reproduced.

The music information may be stored in advance in the storage unit 1700 of the electronic device 1000 or may be obtained by being received from an external server through the communication unit 1500 of the electronic device 1000. [

The music information may be a file name of an audio file in which music is recorded. If the meta information of the audio file is not recorded, the file name of the audio file itself is used as music information, and the story line of the music can be acquired.

The music information may be meta information of an audio file in which music is recorded. The meta information may include information about at least one of a title, an artist, a composer, a lyricist, a player, a genre, a playing time, an album, a composer, a music segment, lyrics, have. If the meta information of the audio file is not completely recorded, the meta information of the audio file may be obtained by searching the search engine or the like based on the file name of the audio file and / or the meta information already recorded. Here, the search engine may provide meta information of the audio file in response to the query.

The music information may be the lyric information corresponding to the audio file in which the music is recorded. When the audio file is reproduced, the electronic device 1000 can load the lyrics information corresponding to the audio file and display the lyrics contained in the lyrics information.

The lyric information can be obtained from the lyric file of the LRC extension. The lyric information may include information related to the starting point of each sentence constituting the lyrics. When the audio file is played back and the lyric information corresponding to the audio file is loaded, the electronic device 1000 loads the lyric information corresponding to the audio file, and based on the start time of each sentence included in the lyric information Each sentence can be scrolled and displayed. Here, the lyrics of the music can be displayed according to the reproduction time of the audio file, so that the user can more easily understand the lyrics of the music being reproduced.

The music information may be audio fingerprint information of the music. The audio fingerprint information of the music means information on the characteristics of the music, and can be used for identification and / or searching of music.

When the music is reproduced in the electronic device 1000, the electronic device 1000 can analyze the audio file in which the music is recorded to obtain audio fingerprint information.

When the music is reproduced in the electronic device separate from the electronic device 1000, the electronic device 1000 records the music through the microphone 1620 to generate audio data, analyzes the audio signal of the generated audio data Audio fingerprint information can be obtained.

Specifically, the audio fingerprint information may be acquired using a power spectrum generated after converting an audio signal into a frequency domain using an FFT (Fast Fourier Transform) method, ≪ / RTI >

The electronic device 1000 may transmit the acquired audio fingerprint information to the audio DB and acquire meta information of the audio file identified by the corresponding audio fingerprint information through DB matching in the audio DB. Here, the audio DB may be stored in the storage unit 1700 of the electronic device 1000 or an external server, and may include audio fingerprint information and meta information of various audio files. The meta information of the audio file includes at least one of a title, an artist, a composer, a lyricist, a player, a genre, a playing time, an album, a composer, a music segment, lyrics, Lt; / RTI >

The control unit 1300 acquires a story line for the music to be reproduced based on the acquired music information.

The story line of the music means a narrative description of the music, and the story line of the music may consist of one or more images, one or more videos, and / or text for narrative explanation.

The story line for music may be stored in advance in the storage unit 1700 of the electronic device 1000 or may be obtained by being received from an external server through the communication unit 1500 of the electronic device 1000. [

The story line for the music may be obtained by searching the search engine or the like based on the meta information of the audio file in which the music is recorded. Here, the search engine may provide a story line for the music in response to the query.

The control unit 1300 acquires one or more images based on the acquired story lines.

One or more images may be stored in advance in the storage unit 1700 of the electronic device 1000 or may be obtained by being received from an external server via the communication unit 1500 of the electronic device 1000. [

If the story line of the music is composed of more than one image, the image associated with the music can be obtained from one or more images constituting the story line of the music.

If the story line of music is composed of more than one video, the image associated with the music may be obtained from one or more frames of video that constitute the storyline of the music. A method of acquiring an image related to the music in the electronic device 1000 from one or more frames of video that constitute a story line of music will be described later with reference to FIG.

When the story line of the music is composed of text, the image related to the music can be obtained based on the text constituting the story line of the music. A method of acquiring an image related to the music in the electronic device 1000 based on a text constituting a story line of music will be described later with reference to Fig.

The display unit 1210 can display the acquired one or more images.

An image obtained based on the storyline of music is displayed through the display unit 1210 of the electronic device 1000, so that a user who enjoys the music can understand the music more deeply. In addition, a rich audiovisual experience may be given to the user.

All of the components shown in Fig. 2 are not essential components of the electronic device 1000. The electronic device 1000 may be implemented by more components than the components shown in FIG. 2, for example, as shown in FIG. 3, an electronic device 1000, according to one embodiment, The input unit 1100, the output unit 1200, the sensing unit 1400, the communication unit 1500, the A / V input unit 1600, and the storage unit 1700 in addition to the control unit 1210 and the control unit 1300 .

Hereinafter, each component of the electronic device 1000 will be described in detail.

The user input receiving unit 1100 means a means for the user to input data in order to control the electronic device 1000. For example, the user input receiving unit 1100 may include a key pad, a dome switch, a touch pad (contact type capacitance type, pressure type resistive type, infrared ray detection type, surface ultrasonic wave type, A fatigue strength measuring method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto. In particular, when the touch pad constitutes a layer structure with the display unit 1210 and is configured as a touch screen or a touch screen display, the user input receiving unit 1100 may be used as an output device in addition to the input device.

The touch pad can be configured not only to detect a real-touch but also to detect a proximity touch. For convenience of description, the touch-pad can be configured to detect both a real-touch and a proximity touch Can be referred to as "touch ".

In the present specification, the term "real-touch" refers to an input generated when a pointer is physically touched on the screen, and "proximity-touch" Refers to an input that is generated in a state in which it is approached by a predetermined distance from the screen even if it is not physically touched.

As used herein, the term " pointer "refers to a tool for directly touching or touching a specific portion of a displayed screen. Examples include stylus pens and fingers.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210, an acoustic output unit 1220, and a vibration motor 1230 .

The display unit 1210 displays information processed in the electronic device 1000. [ For example, the display unit 1210 may display an execution screen of an operating system running on the electronic device 1000 and an execution screen of an application running on the operating system.

Meanwhile, when the display unit 1210 and the touch pad constitute a layer structure to constitute a touch screen, the display unit 1210 can be used as an input device in addition to the output device. The display unit 1210 may be a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional display A 3D display, and an electrophoretic display. According to the embodiment of the electronic device 1000, the electronic device 1000 may include two or more display units 1210. At this time, the two or more display units 1210 may be arranged to face each other using a hinge.

The audio output unit 1220 outputs audio data received from the communication unit 1500 or stored in the storage unit 1700. Further, the sound output unit 1220 outputs an acoustic signal related to the function performed in the electronic device 1000. [ The sound output unit 1220 may include a speaker, a buzzer, and the like.

The vibration motor 1230 can output a vibration signal. For example, the vibration motor 1230 can output a vibration signal corresponding to an output of audio data or video data. In addition, the vibration motor 1230 may output a vibration signal when a touch is input to the touch screen.

The control unit 1300 can perform the functions of the electronic device 1000 by controlling the overall operation of the electronic device 1000. [ For example, the control unit 1300 may include a user input receiving unit 1100, an output unit 1200, a sensing unit 1400, a communication unit 1500, and an A / V input unit 1100, by executing programs stored in the storage unit 1700. [ (1600), and the like.

The sensing unit 1400 may sense a state of the electronic device 1000 or a state around the electronic device 1000 and may transmit sensed information to the control unit 1300. [

The sensing unit 1400 includes a magnetic sensor 1410, an acceleration sensor 1420, an on / humidity sensor 1430, an infrared sensor 1440, a gyroscope sensor 1450, a position sensor (GPS) 1460, an air pressure sensor 1470, a proximity sensor 1480, and an RGB sensor (illuminance sensor) 1490. However, the present invention is not limited thereto. The function of each sensor can be intuitively deduced from the name by those skilled in the art, so a detailed description will be omitted.

The communication unit 1500 may include one or more components that allow communication between the electronic device 1000 and an external device. For example, the communication unit 1500 may include a local communication unit 1510, a mobile communication unit 1520, and a broadcast receiving unit 1530.

The short-range wireless communication unit 1510 includes a Bluetooth communication unit, a Bluetooth low energy communication unit, a near field communication unit, a WLAN communication unit, a Zigbee communication unit, an IrDA , an infrared data association communication unit, a WFD (Wi-Fi Direct) communication unit, an UWB (ultra wideband) communication unit, an Ant + communication unit, and the like.

The mobile communication unit 1520 transmits and receives a radio signal to at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data depending on a voice signal, a video call signal, or a text / multimedia message transmission / reception.

The broadcast receiver 1530 receives broadcast signals and / or broadcast-related information from outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. The electronic device 1000 may not include the broadcast receiver 1530 according to the embodiment.

The A / V (Audio / Video) input unit 1600 is for inputting an audio signal or a video signal, and may include a camera 1610, a microphone 1620, and the like. The camera 1610 can obtain an image frame such as a still image or a video through the image sensor in a video communication mode or a photographing mode. The image captured through the image sensor can be processed through the control unit 1300 or a separate image processing unit.

The image frame processed by the camera 1610 may be stored in advance in the storage unit 1700 or may be transmitted to the outside through the communication unit 1500. More than one camera 1610 may be provided according to the configuration of the terminal.

The microphone 1620 receives an external acoustic signal and processes it as electrical voice data. For example, the microphone 1620 may receive acoustic signals from an external device or speaker. The microphone 1620 may use various noise reduction algorithms to remove noise generated in receiving an external sound signal. The microphone 1620 may receive a user ' s voice response input corresponding to the problem content provided by the control unit 1300 described above.

The storage unit 1700 may store a program for processing and controlling the control unit 1300 and may store data input to or output from the electronic device 1000. [

The storage unit 1700 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory) (Random Access Memory) SRAM (Static Random Access Memory), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory) A disk, and / or an optical disk. The storage unit 1700 may be referred to as a memory.

Programs stored in the storage unit 1700 can be classified into a plurality of modules according to their functions. For example, the programs can be classified into a UI module 1710, a touch screen module 1720, a notification module 1730, have.

The UI module 1710 can provide a specialized UI, a GUI, and the like that are interlocked with the electronic device 1000 for each application. The touch screen module 1720 senses a touch gesture on the user's touch screen and can transmit information on the touch gesture to the control unit 1300. [ The touch screen module 1720 according to one embodiment can recognize and analyze the touch code. The touch screen module 1720 may be configured as separate hardware including a controller.

Various sensors may be provided inside or around the touch pad to detect a user's touch. An example of such various sensors is a tactile sensor. A tactile sensor is a sensor that detects the contact of a specific object with a degree or more that a person feels. The tactile sensor can detect various information such as the roughness of the contact surface, the rigidity of the contact object, and the temperature of the contact point.

In addition, proximity sensors are examples of such various sensors. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or a nearby object without mechanical contact using the force of an electromagnetic field or infrared rays. Examples of proximity sensors include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor.

An example of such various sensors is a force touch sensor. Different functions can be performed in the electronic device 1000 according to the magnitude of the pressure of touching the touch pad, so that gesture input that can be applied through the touch pad can be further diversified.

The gesture input can be implemented in a variety of ways, among which "tap" means the input method in which the pointer is touched off the touchpad once the pointer is touched on the touchpad, and the "double quot; tap "means that an input method in which three or more touches are applied within a predetermined time is referred to as " multiple tap" Means an input method applied while maintaining a state, that is, a long tapping method. Also, "drag" means an operation of moving the pointer from the touch pad to another position while the touch pad and the pointer remain touched after the user touches the touch pad using the pointer. Swipe "is basically the same as" drag ", in which the user touches the touch pad using the pointer and then moves the pointer to another position on the touch pad while keeping the touch pad and the pointer touched Meaning, however, means that the movement speed is relatively faster than "drag".

The notification module 1730 may generate a signal for notifying the occurrence of an event of the electronic device 1000. [ Examples of events generated in the electronic device 1000 include signal reception, message reception, key signal input, schedule notification, and the like. The notification module 1730 may output a notification signal in the form of a video signal through the display unit 1210 or may output a notification signal in the form of an audio signal through the sound output unit 1220, It is possible to output a notification signal in the form of a vibration signal.

In addition, some or all of the respective configurations of the electronic device 1000 shown in Figs. 2 and 3 may be implemented by at least one hardware processor. For example, some or all of the configurations of the electronic device 1000 may be implemented through a separate processor other than the main processor of the electronic device 1000. [

In addition, some of the respective configurations of the electronic device 1000 shown in Figs. 2 and 3 may be implemented by at least one software program. For example, some functions of the electronic device 1000 may be implemented by an Operating System program, and some functions may be implemented by an application program. Accordingly, the functions of the electronic device 1000 may be implemented by at least one piece of hardware and at least one piece of software, and the functions of the electronic device 1000 implemented by the software may be implemented by operating system And an application.

The electronic device 1000 according to one embodiment includes a processor, a memory for storing and executing program data, a permanent storage such as a disk drive, a communication port for communicating with an external device, a touch panel, a key, A user interface device such as a button, and the like. Methods implemented with software modules or algorithms may be stored on a computer readable recording medium as computer readable codes or program instructions executable on the processor. Here, the computer-readable recording medium may be a magnetic storage medium such as a read-only memory (ROM), a random-access memory (RAM), a floppy disk, a hard disk, ), And a DVD (Digital Versatile Disc). The computer-readable recording medium may be distributed over networked computer systems so that computer readable code can be stored and executed in a distributed manner. The medium is readable by a computer, stored in a memory, and executable on a processor.

4 is a flow diagram of a method for displaying an image associated with the music while the music is playing in the electronic device, according to one embodiment.

Will be described with reference to Fig. 1 for convenience of explanation. Referring to FIG. 1, a multimedia playback application 100 may be driven in an electronic device, and music may be played back. Here, the reproduction of the music means the reproduction of the audio file of the music.

In step S100, the electronic device acquires music information about music reproduced in the electronic device.

The music information may be pre-stored in the electronic device or may be obtained by being received from an external server via the Internet or the like.

The music information may be meta information of an audio file in which music is recorded. The meta information may include information about at least one of a title, an artist, a composer, a lyricist, a player, a genre, a playing time, an album, a composer, a music segment, lyrics, have.

For example, as shown in the information window 130 of FIG. 1, the electronic device can obtain the music's number and title (MOBB-Chenzao). The electronic device may acquire various meta information not shown in the information window 130 in addition to the song's title and music title.

On the other hand, the electronic device may be implemented to acquire music information about music reproduced in the electronic device and other electronic devices.

At this time, the music information may be audio fingerprint information, the electronic device may record the music to generate audio data, and analyze audio signals of the generated audio data to obtain audio fingerprint information.

In step S110, the electronic device acquires a story line for the music based on the music information obtained in step S100.

A story line for music can consist of one or more images, one or more videos, and / or text.

The story line for music can be pre-stored in the electronic device or can be obtained by being received from an external server via the Internet or the like.

The story line for the music may be obtained by searching the search engine or the like based on the meta information of the audio file in which the music is recorded. Here, the search engine may provide a story line for the music in response to the query. For example, a search engine may provide a music video for music in response to a query.

Referring to FIG. 1, an electronic device can acquire a video, e.g., a music video, related to the music, based on the music's number and title (MOBB-CHENZAO). That is, the story line for the music is composed of the music video of the music.

In step S120, the electronic device acquires one or more images based on the storyline obtained in step S110.

The image may be pre-stored in the electronic device, or may be obtained by being received from an external server via the Internet or the like.

When the story line of the music is composed of one or more videos, an image related to the music among the frames of one or more videos constituting the story line can be obtained.

For example, referring to FIG. 1, an electronic device may obtain one or more images from frames that make up a music video. At this time, the electronic device may acquire one or more images at predetermined time intervals among the frames constituting the music video. For example, one or more images may be acquired at intervals of 10 seconds among the frames constituting the music video. As shown in the information window 130 of FIG. 1, when the reproduction time of the music reproduced in the electronic device is 3:59 and the reproduction time of the music video of the music is also equal to 3:59, The first frame at 00, the second frame at 0:10, the third frame at 0:20, ... , And a 24th frame at 3:50 can be obtained.

In step S130, the electronic device displays the one or more images obtained in step S120.

For example, referring to FIG. 1, an electronic device may display acquired images 110A, 110B, 110C, and 110D.

The electronic device may be configured to display the acquired images 110A, 110B, 110C, and 110D in accordance with the music being played back. For example, frames obtained at predetermined time intervals from the music video may be displayed in the electronic device at the predetermined time intervals in frame order.

For example, referring to FIG. 1, if the playing time of the music played on the electronic device is 3:59 and ten seconds have elapsed after playing the music, the second frame at 0:10 in the music video Can be displayed. Accordingly, as shown in the information window 130, the second frame 110A of the corresponding music video may be displayed at the 0:12 playback time point. Likewise, the fifth frame 110B, the thirteenth frame 110C, and the eighteenth frame 110D of the music video may be displayed at the time of 0:46, 1:51, and 2:33 playback, respectively.

If the playing time of the music and the playing time of the music video of the music are different from each other, the electronic device can display images based on the ratio of the two playing times. For example, if the playback time of the music video is 7:58 and the music playback time is 2:59, images acquired at intervals of 10 seconds can be displayed at intervals of 5 seconds in the electronic device.

Since the electronic device only reproduces music and displays an image related to the music, it has an effect of consuming less resources of the electronic device than playing the video related to the music. In addition, when the electronic device is connected to the Internet through the mobile communication network and streaming music or images, there is an effect of reducing the burden of data usage by the user.

FIG. 5 is a flowchart illustrating a method of displaying an image related to a music using a keyword included in a story line of the music while the music is being reproduced in the electronic device according to an exemplary embodiment.

6 for convenience of explanation.

FIG. 6 is a diagram illustrating an example of displaying an image related to a corresponding music using a description of the music while the music is being reproduced in the electronic device according to an embodiment.

Steps S200, S210, S230, and S240 are similar to steps S100, S110, S120, and S130 described above in FIG. 4, respectively, and thus redundant description is omitted.

In step S200, the electronic device acquires music information about music reproduced in the electronic device.

The music information may be meta information of an audio file in which music is recorded. The meta information may include at least one of a title, an artist, a composer, a lyricist, a player, a genre, a playing time, an album, a composer, a music segment, lyrics, and a description of the music.

For example, as shown in the information window 230 of FIG. 6, the electronic device may obtain a music composer and title (Beethoven - Symphony No. 5). The electronic device may acquire various meta information not shown in the information window 230 in addition to the song number and title of the music.

In step S210, the electronic device acquires a story line for the music based on the music information obtained in step S200.

A story line for music can consist of one or more images, one or more videos, and / or text.

The story line for music can be pre-stored in the electronic device or can be obtained by being received from an external server via the Internet or the like.

The story line for the music can be stored as meta information of the audio file in which the music is recorded. The story line for the music may include a description of the music and / or lyrics of the music included in the meta information.

For example, the electronic device may acquire various meta information not shown in the information window 230 in addition to the music composer and title (Beethoven - Symphony No. 5) Can be obtained. At this time, the description of the music may include a description of each of the music segments constituting the music.

The music segments may be portions distinguished musically from each other in one piece of music and may be portions distinguished by musical form.

For example, popular music generally includes intro, verse, pre-chorus, chorus, bridge, outro, sabi, hook, ad lib, humming, scat, and the like.

For example, a symphony is generally divided into four movements, each of which has a sonata allegro form with a fast tempo, an ABA or theme and variation with a slow tempo, A mid / fast tempo minuet or scherzo type, and a fast tempo sonata or rondo type. In addition, each movement can be divided into a plurality of paragraphs. The music segment of a symphony music can be each movement of the symphony, or it can be each paragraph.

Beethoven's Symphony No. 5 is divided into four movements, and four movements can correspond to four music segments. The music segment information included in the meta information may include information on the starting point of each music segment. For example, if Beethoven's fifth symphony audio file has a total play time of 34:25, the first move in the audio file starts at 00:00, the second move starts at 8:25, Start at 16:00, and the fourth movement starts at 24:44, the start time of each music segment may be 00:00, 8:25, 16:00, and 24:44.

Each of the four movements has an alias, "Struggle", "Hope", "Doubt", and "Victory". Thus, the story line of Beethoven's Symphony No. 5 may contain a description of the four movements and the alias of the movement.

If the story line for the music includes only the description of the music and / or the lyrics of the music, the story line is composed of text.

On the other hand, the story line for music may be obtained by searching in a search engine or the like based on music information of the music, for example, meta information of the audio file in which the music is recorded. Where the search engine may provide a description of the music and / or lyrics of the music in response to the query. Thus, the electronic device can obtain a description of the music provided from the search engine and / or the lyrics of the music.

If the search engine provides a variety of content, as well as a description of the music and / or lyrics of the music, in response to the query, the electronic device may be configured to select a story line for the music based on predetermined ones of the various content It is possible.

For example, the predetermined condition may be an order between the contents. That is, the highest rated content among the various contents may be selected as a story line for the music.

For example, the predetermined condition may be the number of duplication or redundancy of the content. That is, the content provided most frequently by the search engine among various contents can be selected as a story line for music.

The electronic device may also select story lines for music from a plurality of search engines. Furthermore, the electronic device may be configured to select a story line for music based on predetermined ones of the contents provided from a plurality of search engines. For example, the predetermined condition may be the number of duplication or redundancy of the content. That is, the content provided by the plurality of search engines in duplicate can be selected as a story line for the music.

The electronic device may select a storyline for music from various content based on a combination of the conditions described above.

In step S220, the electronic device obtains one or more keywords based on the storyline obtained in step S210.

If the story line for the music includes only the description of the music and / or the lyrics of the music, the story line is composed of text.

Story lines composed of text can be analyzed to obtain keywords. As a result of the analysis of the story line, the keyword can be selected based on the part of the word in the story line, the number of occurrences of the word, the context, and the like. For example, nouns, verbs, and / or frequently used words may be selected as keywords in the story line, but the present invention is not limited thereto, and keywords can be selected based on various criteria.

Based on the story line composed of text, the electronic device can acquire one or more keywords. For example, when Beethoven's Symphony No. 5 is reproduced on an electronic device, the electronic device is based on the story line of Beethoven's Symphony No. 5, which is called "Struggle," "Hope," "Doubt ) &Quot;, and " Victory ".

In step S230, the electronic device obtains one or more images based on the keywords obtained in step S220.

The one or more images may be stored in advance in the electronic device, or may be obtained by being received from an external server via the Internet or the like.

When one or more images are pre-stored in the electronic device, a plurality of images are stored in the image database of the storage unit of the electronic device, so that one or more images can be obtained based on the keywords from the image database. Here, the image database can store images, titles of images, descriptions of images, and the like. The electronic device can search the image database using the acquired keyword. The electronic device may obtain one or more images having a title or description matched with the search result of the image database, the keyword.

When receiving and acquiring one or more images from an external server via the Internet or the like, the electronic device can acquire one or more images using a search engine. Here, the search engine may provide an image in response to a query based on each keyword.

Referring to FIG. 6, four images 210A corresponding to the keywords 'Struggle', 'Hope', 'Doubt', and 'Victory' , 210B, 210C, 210D can be obtained.

In step S240, the electronic device displays the image obtained in step S230.

For example, referring to FIG. 6, the electronic device may display the acquired four images 210A, 210B, 210C, and 210D.

The electronic device may be implemented to display the four acquired images 210A, 210B, 210C and 210D in accordance with the music being reproduced. For example, four images 210A, 210B, 210C, and 210D may be respectively displayed during each of four time ranges that divide the total playing time of the music being played into quarters.

The electronic device may be implemented to display the acquired four images 210A, 210B, 210C, and 210D in accordance with the music segment of the music being played back. For example, the music segment of the music to be reproduced corresponds to the acquired four images 210A, 210B, 210C and 210D, so that when any music segment is reproduced, an image corresponding to the music segment can be displayed . That is, the image corresponding to the first music segment can be displayed during the time range from the start point of the first music segment to the start point of the second music segment next to the first music segment.

The electronic device can acquire and display an image using a story line including text even if there is no music video corresponding to the music to be reproduced or a separate explanation image. Therefore, the user can understand the music more easily, and the electronic device acquires and displays the image using the text, thereby saving the resources of the electronic device.

7 is a diagram illustrating an example in which an image related to a corresponding music is displayed using lyrics of the music while the music is being reproduced in the electronic device according to an embodiment.

The story line for the music may be obtained by searching in a search engine or the like based on music information of the music, for example, meta information of the audio file in which the music is recorded. Where the search engine may provide a description of the music and / or lyrics of the music in response to the query. Thus, the electronic device can obtain a description of the music provided from the search engine and / or the lyrics of the music.

If the story line for the music includes only the description of the music and / or the lyrics of the music, the story line is composed of text. Here, the story line may include one or more sentences composed of text. One or more sentences can be distinguished by punctuation, such as punctuation marks, question marks, and exclamation points, or by grammatical analysis.

For example, the lyrics of the song "Let's sing together" by the singer "The Earth" is "I'm in times of trouble" She comes to me, Saying looking ahead to, Let's sing together, "the lyrics include four sentences.

Each sentence in the lyric can be distinguished by grammatical analysis, for example, one verse separated by one verb can be a sentence.

If the story line contains punctuation, each sentence can be distinguished by punctuation.

The electronic device may obtain one or more images based on one or more sentences comprising the story line.

The one or more images may be stored in advance in the electronic device, or may be obtained by being received from an external server via the Internet or the like.

When one or more images are pre-stored in the electronic device, a plurality of images are stored in the image database of the storage unit of the electronic device, and one from the image database, based on each sentence constituting the story line or each sentence keyword The above image can be obtained. Here, the image database can store images, titles of images, descriptions of images, and the like. The electronic device can search an image database using each sentence constituting the story line or keyword of each sentence. The electronic device can acquire one or more images having a title or description matching the search result of the image database, each sentence constituting the story line, or the keyword of each sentence.

When receiving and acquiring one or more images from an external server via the Internet or the like, the electronic device can acquire one or more images using a search engine. Here, the search engine may provide an image in response to a query based on each sentence or keyword of each sentence constituting the story line.

Referring to FIG. 7, images 310A, 310B, 310C, and 310D corresponding to each sentence constituting the lyrics of "Let's sing together" of the Earth can be obtained. The electronic device can display the acquired images 310A, 310B, 310C, 310D.

On the other hand, the meta information may include the lyric information. Therefore, when the audio file is played back and the lyric information corresponding to the audio file is loaded, the electronic device loads the lyric information corresponding to the audio file, and based on the start time of each sentence included in the lyric information, Sentences can be scrolled and displayed. Furthermore, the electronic device may display an image corresponding to each sentence at the start of each sentence. Accordingly, the image corresponding to the lyrics and lyrics of the music can be displayed according to the reproduction time of the audio file, so that the user can more easily understand the music being reproduced.

In one embodiment, one or more sentences comprising the story line may constitute one or more paragraphs. The electronic device can distinguish paragraphs based on the similarity between the sentences constituting the story line. The similarity between sentences can be calculated by a well - known similarity calculation method.

At present, the known similarity calculation method can be divided into five categories: a character matching method, a TF-IDF (vector term vector-inverse document frequency) method, a probabilistic method, a sentence structure method and a semantic extension method. The character matching method calculates the similarity of sentences according to the same number of words in two sentences. A typical example is the Jaccard Similarity Coefficient method. The TF-IDF vector method expresses the sentence as a TF-IDF vector and uses the cosine value of the two vectors as the degree of similarity. The probabilistic method introduces a language model framework and obtains the similarity of two sentences using probabilities. In the sentence structure method, the sentence is divided into different components according to the matching approach of the sentence template, and the similarity is calculated based on the structure of the components.

For example, if the similarity value of the first sentence is predetermined, the second sentence can have a similarity value according to the similarity calculated using the first sentence and the second sentence. That is, similarity is calculated by using the own sentence and the previous sentence, so that all sentences can have the similarity value. The similarity value of all sentences can form a similarity value sequence.

A threshold value may be set for the similarity value sequence. In addition, each similarity value in the similarity value sequence can be compared with a threshold value. If a similarity value is larger than the threshold value, the similarity between the sentence having the similarity value and the previous sentence can be judged to be relatively high. If the similarity value is lower than the threshold value, the similarity between the sentence having the similarity value and the previous sentence can be judged to be relatively low. If the similarity value is lower than the threshold value, the sentence having the similarity value belongs to the next paragraph of the paragraph to which the previous sentence belongs, and the sentence having the similarity value is the first sentence of the next paragraph.

The threshold may be set by the user or by the administrator of the back-end server, or may be set to a default value. The default value can be obtained by the following statistical method. First, when a plurality of types of music are selected and a story line for each music is acquired, the number of paragraphs of each story line is determined. The similarity value sequence is obtained for any story line including X paragraphs, and the similarity value values included in the similarity value sequence are listed from a small value to a large value, and the Xth similarity value is set to a critical value Lt; / RTI > An average of the threshold values for each story line can be determined as a default value.

In one embodiment, music may be separated into music segments using the following method. First, samples are extracted from music at predetermined time intervals. For example, the predetermined time interval may be a 10 ms interval. Then, the characteristics of the extracted sample are analyzed. For example, the characteristics of the sample may include time domain and frequency domain characteristics, fundamental frequency characteristics, and Mel Frequency Cepstral Coefficients (MFCC) characteristics. Using the classifier, the extracted sample is determined to which music segment it belongs. For example, the classification device may be a support vector device. The music sample classification technique based on the characteristics of the sample is an area in which research and application is performed, and is not repeated herein.

A sentence and / or paragraph of the lyrics and the music segment of the music may be associated using the lyrics information and the music segment information included in the meta information of the music. The lyric information may include information related to the starting point of each sentence constituting the lyrics, and the music segment information may include information on the starting point of each music segment. Thus, sentences having a start time point between a start point of a music segment and a start point of a next music segment of the music segment may be associated with the music segment. On the basis of the above-described similarity calculation method, if the lyrics are divided into paragraphs, a paragraph and a music segment of the music may be associated.

The paragraph of the story line of the music can be associated with the music segment in other ways. For example, in the case of pop music, the paragraph of the pop music may be associated with the music segment in the following way. For convenience of explanation, it is assumed that the popular music includes an intro, a verse, a chorus, and a bridge music segment. Since the intro and bridge of popular music generally have no lyrics, the chorus music segment and the bridge music segment may not be associated with a paragraph of the story line. And, if the music story line is composed of M paragraphs, the last paragraph of the story line is associated with the chorus music segment of popular music, and M-1 paragraphs excluding the last paragraph in the story line can be associated with the music segment have.

The music segment may be divided into sub-segments. In this example, the clause music segment may be divided into sub-segments. First, samples are extracted at predetermined time intervals from the music segment. For example, the predetermined time interval may be a 10 ms interval. All samples extracted from the music segment form a sample sequence.

For example, if the similarity value of the first sample in the sample sequence is predetermined, the second sample may have a similarity value according to the similarity calculated using the first sample and the second sample. That is, the similarity is calculated using the magnetic sample and the previous sample, so that all the samples can have the similarity value. The similarity values of all samples may form a sequence of sample similarity value values.

The sample similarity value included in the sample similarity value sequence is sorted from a small value to a large value so that the (M-1) th sample similarity value can be set as a threshold value. The threshold value may be set by the user or the administrator of the back-end server, or may be set to a default value. The sample similarity value of each sample included in the sample sequence is compared with a threshold value so that the clause music segment can be divided into M-1 sub-segments. For example, when the sample similarity value of a sample is larger than the threshold value, the similarity between the sample having the corresponding sample similarity value and the previous sample can be determined to be relatively high. If the sample similarity value is lower than the threshold value, the similarity between the sample having the corresponding sample similarity value and the previous sample can be judged to be relatively low. If the sample similarity value is lower than the threshold value, the sample having the corresponding sample similarity value belongs to the next sub-segment of the sub-segment to which the previous sample belongs, and the sample having the corresponding sample similarity value becomes the first sample of the next sub- M-1 sub-segments may be associated with M-1 paragraphs except the last paragraph in the story line.

Music segments of music in an electronic device can be played back continuously. When a music segment is played back, the electronic device can display an image corresponding to each sentence included in a paragraph associated with the music segment. Since the intro and bridge of popular music generally have no lyrics, the chorus music segment and the bridge music segment may not be associated with a paragraph of the story line. As such, when a music segment not associated with a paragraph of a story line is reproduced, a default image can be displayed. The default image may be an album cover of the music being played.

Music is divided into music segments, music segments are divided into sub-segments, and images corresponding to the sub-segments can be displayed while the sub-segments are being reproduced. Therefore, the image is displayed more precisely while the music is being reproduced, and the user's understanding of the music can be further enhanced.

FIG. 8 is a flowchart of a method of displaying an image related to music using a plurality of candidate images related to the music while the music is being reproduced in the electronic device, according to an embodiment.

Steps S300, S310, S320, S330, and S360 are similar to S200, S210, S220, S230, and S240 described above in FIG. 5, and thus redundant description is omitted.

In step S300, the electronic device acquires music information about music reproduced in the electronic device.

In step S310, the electronic device acquires the story line for the music based on the music information obtained in step S300.

In step S320, the electronic device obtains one or more keywords based on the storyline obtained in step S310.

In step S330, the electronic device obtains a plurality of candidate images based on the one or more keywords obtained in step S320.

The electronic device can acquire a plurality of candidate images corresponding to one keyword.

A plurality of candidate images may be previously stored in the electronic device, or may be obtained by being received from an external server via the Internet or the like.

When receiving and acquiring a plurality of candidate images from an external server via the Internet or the like, the electronic device can acquire a plurality of candidate images using a search engine. Here, the search engine may provide a plurality of candidate images in response to a query based on the keyword.

The electronic device may obtain a plurality of candidate images from a plurality of search engines.

In step S340, the electronic device determines a priority among the plurality of candidate images obtained in step S330, based on a predetermined criterion.

For example, the predetermined criteria may be an order among a plurality of candidate images. That is, a candidate image that is highly ranked from a search engine among a plurality of candidate images has a higher priority than a candidate image that is lowly rated from a search engine.

For example, the predetermined criterion may be the number of duplicates or the overlap ratio of the candidate image. That is, the higher the priority of the candidate image provided by the search engine than the plurality of candidate images is, the higher the priority is.

For example, the predetermined criterion may be the similarity between the candidate image and the keyword. Semantic information on the meaning of the candidate image can be obtained through the semantic analysis of the candidate image, and the similarity between the semantic information and the keyword can be determined. Semantic analysis and similarity comparisons are not discussed here because they are well researched and applied.

When an electronic device acquires a plurality of candidate images from a plurality of search engines, the electronic device may be configured to determine a priority among a plurality of candidate images provided from a plurality of search engines, based on a predetermined criterion. For example, the predetermined criterion may be the number of duplicates or the overlap ratio of the candidate image. That is, the candidate images provided by the plurality of search engines in the most overlapped manner have higher priority.

The electronic device may determine a priority among a plurality of candidate images based on a combination of the above-described criteria.

In step S350, the electronic device obtains one or more images from among the plurality of candidate images, based on the priority determined in step S340.

The electronic device can obtain an image having one of the highest priority among a plurality of candidate images corresponding to one keyword. That is, one keyword corresponds to one image.

The electronic device may acquire a plurality of images in order of highest priority among a plurality of candidate images so that a plurality of images correspond to one keyword. The number of images obtained from the plurality of candidate images may be implemented so as to be proportional to the reproduction time of the music segment corresponding to the keyword.

In step S360, the electronic device displays the one or more images obtained in step S350.

An image more suitable for the keyword of the story line of the music can be displayed, so that the user who enjoys the music can understand the music more easily.

9 is a flow diagram of a method for rendering and displaying an image associated with a music piece during playback of music in an electronic device, according to an embodiment.

Steps S400, S410, S420, and S440 are similar to steps S100, S110, S120, and S130 described above in Fig. 4, respectively, and thus redundant description is omitted.

In step S400, the electronic device acquires music information about music reproduced in the electronic device.

In step S410, the electronic device acquires a story line for the music based on the music information obtained in step S400.

In step S420, the electronic device obtains emotion information and one or more images based on the storyline obtained in step S410.

When the story line of music is composed of text, one or more keywords are acquired based on the story line, and emotion information corresponding to each keyword can be acquired through the word emotion database. The word emotion database can store the words and the emotion information of the corresponding words in a pair and the electronic device can acquire emotion information corresponding to each keyword through DB matching between the words stored in the word emotion database . For example, since the word "trouble" is negative, the word emotion database may store a pair of emotional information representing the word " trouble "and the word" negative ". Emotion information can be composed of text, code, symbol, number, etc., and various emotions can be represented by the combination.

When the story line of the music is composed of one or more images, the semantic information about the meaning of the one or more images can be obtained through the semantic analysis of the one or more images, and the emotion information corresponding to the semantic information Can be obtained through the emotion database. Through the DB matching between the respective semantic information and the words stored in the word emotion database, the electronic device can acquire emotion information corresponding to each image.

The word emotion database may be pre-stored in the electronic device, or may be pre-stored in an external server.

In step S430, the electronic device renders the one or more images obtained in step S420 based on the emotion information obtained in step S420.

Rendering an image means adjusting the hue, saturation, brightness, contrast, and / or transparency of the image or giving a graphic effect to the image in order to generate a certain emotion for the user who viewed the image.

For example, if the acquired emotion information represents a negative emotion, the electronic device may lower the brightness of the acquired image or lower the saturation to cause a negative emotion to the user who viewed the image.

For example, if the acquired emotion information represents a positive emotion, the electronic device may increase the brightness of the acquired image, or increase the saturation, in order to generate a positive emotion for the user who viewed the image.

In step S440, the electronic device displays one or more images rendered in step S430.

Since the image is rendered and displayed, the user can enjoy the story line of the music to be played more sensitively.

According to one embodiment, a story line is divided into one or more paragraphs based on at least one of time information, spatial information, and person information, and the music may be divided into one or more music segments corresponding to one or more paragraphs.

Time information refers to time related words in a story line composed of text. For example, the words related to time may be morning, noon, night, spring, summer, autumn, winter, and the like. If the story line contains the time information "fall", the sentence containing "fall" belongs to the next paragraph of the paragraph to which the previous sentence of that sentence belongs.

Spatial information is a space-related word in a story line composed of text. For example, words related to space can be mountain, building, sea, room, house, and so on. If the story line contains spatial information called "mountain", the sentence containing "mountain" belongs to the next paragraph of the paragraph to which the previous sentence of that sentence belongs.

The character information means a word related to a person in a story line composed of text. For example, a word related to a person can be the name of you, you, me, me, him, her, and so on. If the story line contains person information "her", the sentence containing "her" belongs to the next paragraph of the paragraph to which the previous sentence of the sentence belongs.

The separated paragraphs of the storyline can be related to the 'period', 'win', 'former', and 'final' of the storyline. If the story line is divided into four paragraphs, then each of the four paragraphs may be associated with each of the 'period', 'win', 'former', and 'outcome' of the story line, respectively. If the story line is divided into three paragraphs, the first of the three paragraphs is associated with a period, the second paragraph is associated with a 'w' and 'before', the third paragraph is associated with a ' . If the storyline is divided into two paragraphs, the first paragraph may be associated with 'period' and 'w', and the second paragraph may be associated with 'former' and 'final'. If the story line is divided into five or more paragraphs, the first paragraph is associated with the period, the last paragraph is associated with the word, and the preceding and following paragraphs of the remaining paragraphs are equally divided into & Can be associated with 'before'.

The paragraph of the story line of the music can be associated with the music segment in other ways. For example, in the case of pop music, the paragraph of the pop music may be associated with the music segment in the following way. For convenience of explanation, it is assumed that the popular music includes an intro, a verse, a chorus, and a bridge music segment. The 'period', 'win', 'previous', and 'final' of the storyline can be associated with the intro, verse, chorus, and bridge music segments.

Music segments of music in an electronic device can be played back continuously. The corresponding music segment corresponds to the associated 'period', 'win', 'previous', or' result ', and' period ',' win ',' previous', or ' , When a music segment is reproduced, the electronic device can display an image corresponding to each sentence included in the paragraph corresponding to the music segment.

All publications, including publications, patent applications, patents, and the like, cited herein are incorporated herein by reference in their entirety as if each cited document were individually and specifically incorporated or represented in its entirety by incorporation herein .

In order to facilitate understanding of the present invention, reference will be made to the preferred embodiments shown in the drawings, and specific terminology is used to describe the embodiments of the present invention. However, the present invention is not limited to the specific terminology, Lt; / RTI > may include all elements commonly conceivable by those skilled in the art.

The present invention may be represented by functional block configurations and various processing steps. These functional blocks may be implemented in a wide variety of hardware and / or software configurations that perform particular functions. For example, the present invention may include integrated circuit configurations, such as memory, processing, logic, look-up tables, etc., that may perform various functions by control of one or more microprocessors or other control devices Can be adopted. Similar to the components of the present invention that may be implemented with software programming or software components, the present invention may be implemented as a combination of C, C ++, and C ++, including various algorithms implemented with data structures, processes, routines, , Java (Java), assembler, and the like. Functional aspects may be implemented with algorithms running on one or more processors. Further, the present invention can employ conventional techniques for electronic environment setting, signal processing, and / or data processing. Terms such as "mechanism," "element," "means," and "configuration" may be used broadly and are not limited to mechanical and physical configurations. The term may include the meaning of a series of routines of software in conjunction with a processor or the like.

The specific acts described in the present invention are, by way of example, not intended to limit the scope of the invention in any way. For brevity of description, descriptions of conventional electronic configurations, control systems, software, and other functional aspects of such systems may be omitted. Also, the connections or connecting members of the lines between the components shown in the figures are illustrative of functional connections and / or physical or circuit connections, which may be replaced or additionally provided by a variety of functional connections, physical Connection, or circuit connections. Also, unless explicitly referred to as " essential ", " important ", etc., it may not be a necessary component for application of the present invention.

The use of the terms < RTI ID = 0.0 > and / or < / RTI > like terms used in the specification and claims may refer to both singular and plural. The scope of the present invention is defined by the appended claims. The scope of the present invention is defined by the appended claims rather than by the description preceding them. It is like. Finally, the steps may be performed in any suitable order, unless explicitly stated or contrary to the description of the steps constituting the method according to the invention. The present invention is not necessarily limited to the order of description of the above steps. The use of all examples or exemplary language (e.g., etc.) in this invention is for the purpose of describing the present invention only in detail and is not to be limited by the scope of the claims, It is not. It will also be appreciated by those skilled in the art that various modifications, combinations, and alterations may be made depending on design criteria and factors within the scope of the appended claims or equivalents thereof.

Claims (20)

The method comprising the steps of: acquiring music information about music reproduced in an electronic device;
Obtaining a story line for the music based on the music information;
Obtaining one or more images based on the story line; And
And displaying the at least one image. The method according to claim 1,
Wherein the music information includes meta information of the music,
Wherein the story line is obtained based on the meta information. The method according to claim 1,
Wherein the music information includes audio fingerprint information of the music,
Wherein the story line is obtained based on the audio fingerprint information. The method according to claim 1,
The step of acquiring one or more images based on the story line
Selecting one or more keywords based on the storyline,
Wherein the one or more images are obtained based on the one or more selected keywords. 5. The method of claim 4,
The step of acquiring one or more images based on the story line
Obtaining a plurality of candidate images based on the selected one or more keywords; And
Determining a priority between the plurality of candidate images based on a predetermined criterion,
Wherein the one or more images are obtained based on the priority among the plurality of candidate images. 6. The method of claim 5,
Wherein determining the priority among the plurality of candidate images based on a predetermined criterion comprises:
And obtaining semantic information of the plurality of candidate images,
Wherein the priority is determined by the similarity between the semantic information of the plurality of candidate images and the keyword selected based on the story line. The method according to claim 1,
The story line includes one or more sentences composed of text,
Wherein the one or more images are obtained based on the one or more sentences. 8. The method of claim 7,
Wherein the one or more sentences comprise one or more paragraphs based on similarity,
Wherein the music is divided into one or more music segments corresponding to the one or more paragraphs,
Wherein the one or more images are displayed based on the one or more music segments. 8. The method of claim 7,
The story line is the lyrics of the music,
Wherein the music information includes lyrics information about the lyrics,
Wherein the one or more images are displayed based on the lyric information. The method according to claim 1,
Further comprising the step of acquiring emotion information based on the storyline,
Wherein the at least one image is obtained based on the emotion information. 11. The method of claim 10,
Wherein the one or more images are rendered based on the emotion information. The method according to claim 1,
Wherein the story line is divided into one or more paragraphs based on at least one of time information, spatial information, and person information,
Wherein the music is divided into one or more music segments corresponding to the one or more paragraphs,
Wherein the one or more images are displayed based on the one or more music segments. Acquiring music information about music reproduced in the electronic device,
Acquiring a story line for the music based on the music information,
A control unit for acquiring one or more images based on the story line; And
And a display unit for displaying the one or more images. 14. The method of claim 13,
Wherein the music information includes meta information of the music,
Wherein the story line is obtained based on the meta information. 14. The method of claim 13,
Wherein the music information includes audio fingerprint information of the music,
Wherein the story line is obtained based on the audio fingerprint information. 14. The method of claim 13,
Wherein the control unit selects one or more keywords based on the story line,
Wherein the one or more images are obtained based on the one or more selected keywords. 14. The method of claim 13,
The story line includes one or more sentences composed of a series of texts,
Wherein the one or more images are obtained based on the one or more sentences. 14. The method of claim 13,
Wherein the control unit further obtains emotion information related to the music based on the story line,
Wherein the at least one image is obtained based on the emotion information. 14. The method of claim 13,
Wherein the story line is divided into one or more paragraphs based on at least one of time information, spatial information, and person information,
Wherein the music is divided into one or more music segments corresponding to the one or more paragraphs,
Wherein the one or more images are displayed based on the one or more music segments. 13. A computer-readable recording medium on which a program for implementing the multimedia reproducing method according to any one of claims 1 to 12 is recorded.

Images