KR20140126237A - System, method and recordable medium for providing related contents at low power - Google Patents

Abstract

According to an embodiment, a system for providing low power related content, which can automatically provide relative content of main content operated with low-power while being viewed and listened, is provided. The present invention prevents the immersion of a user on the main content from being interrupted by extracting sound wave ID contained in the main content, which is broadcasted or played through a content player, with low-power; and provides the related content of indirect advertising products contained in the main content.

Description

TECHNICAL FIELD [0001] The present invention relates to a low-power-related content providing system, a method, and a computer-readable storage medium storing a program.

The present invention relates to a low power related content providing system, a method, and a computer-readable recording medium on which a program is recorded by receiving low-power audio included in the content and providing related contents.

Conventional commodity advertisements are not only directly advertising through newspapers, radio, and TV broadcasts, but also indirectly by applying commodities that appear in contents such as movies, dramas, and radio broadcasts, or products that are naturally referred to as part of contents Advertising techniques (hereinafter referred to as product indirect advertising or indirect advertising) are widely used. Particularly, in the case of indirect product advertising, the weight of the product is increasing because it provides a strong advertisement effect to viewers and listeners. However, in the case of the conventional indirect advertisement, the product is simply exposed on the screen or referred to as sound. Therefore, there is a problem that the viewer or the listener must purchase the product based on the retrieved information after separately searching the related information and the selling place for the product even when the desire to purchase the product occurs. . For example, Korean Patent Registration No. 10-1310943 (Mar. 9, 201) (system and method for providing content-related information related to broadcast contents), Korean Patent Registration No. 10-1313293 (March 24, 2013) (System and method for providing additional information of broadcast contents) and Korean Patent Registration No. 10-0893671 (April 9, 2009) (generation and matching of multimedia hashs) Discloses a method of providing related content for a product indirect advertisement contained in a content viewed or listened to by a viewer or a mobile receiver of a listener. For example, Korean Patent Registration No. 10-1363454 (Apr. 2, 2014) (a customized advertisement providing system based on sound signals output from a TV, a method of providing a user-tailored advertisement, and a method of recording an MIM service program A computer-readable recording medium) discloses a method of inserting non-audible sound waves into a broadcast or reproduced content to provide related contents for a product indirect advertisement contained in a content viewed or listened to by a viewer or a mobile receiver of the listener Lt; / RTI >

However, in the above-described techniques, there still exists a problem that a viewer or a listener must positively request an associated content using a mobile receiver. That is, at the time of content reproduction, an application of the mobile receiver must be executed in order to receive the associated content, or an application of the mobile receiver providing the related content should be instructed to recognize the content. In this way, it is necessary to take extra effort in receiving the related contents because it is contrary to the behavior pattern of the viewer or the listener who enjoys the contents passively, or the separate effort is hindering the immersion of the viewer or the listener, The utility of the content may be deteriorated.

On the other hand, such a problem can be solved by continuously extracting acoustic features or non-audible sound waves from a mobile receiver, but this causes a power shortage problem for a mobile receiver operating on a limited battery.

 According to an embodiment of the present invention, a low-power-related content providing system capable of automatically providing an associated content for a main content that is being viewed or listened to without operating a viewer or a listener while actively operating at low power can be provided have.

According to an embodiment of the present invention, a sound wave ID included in a main content broadcasted or reproduced through a content reproduction device is extracted at a low power and transmitted to a mobile receiver of a user who is a viewer or a listener, Power related content providing system that provides the related contents for the indirect advertisement product included in the main content without interfering with the immersion into the main content.

According to an embodiment of the present invention, there is a need for a low-power-related content providing method that can operate automatically at low power without automatically searching for an associated content for a main content that is being watched or listened to by a viewer or a listener.

According to an embodiment of the present invention, a sound wave ID included in a main content broadcasted or reproduced through a content reproduction device is extracted at a low power and transmitted to a mobile receiver of a user who is a viewer or a listener, A method may be provided for providing the related content for the indirect advertisement product included in the content without interfering with the immersion into the content.

According to an embodiment of the present invention, a computer executes a low-power-related content providing method that can automatically provide an associated content of a main content that is being viewed or listened to without operating a viewer or a listener actively while operating at a low- A recording medium on which a program is recorded can be provided.

According to an embodiment of the present invention, a sound wave ID included in a main content broadcasted or reproduced through a content reproduction device is extracted at a low power and transmitted to a mobile receiver of a user who is a viewer or a listener, There is provided a recording medium on which a program for causing a computer to execute a method of providing an associated content for an indirect advertisement product included in a main content without interfering with the flow of the advertisement.

According to an embodiment of the present invention, a computer-readable recording medium on which a low power related content providing system, method, or program is recorded may be provided.

According to one or more embodiments of the present invention, the sound wave ID included in the main content broadcasted or reproduced through the content reproduction device may be extracted at a low power, and may be transmitted to the user through the mobile receiver of the viewer or listener, It is possible to provide the related content for the indirect advertisement product included in the main content without interfering with the main content.

According to one or more embodiments of the present invention, in order not to interfere with program immersion by the viewer or the listener, the related content for the indirect advertisement included in the main content being broadcast or reproduced is temporarily displayed at the bottom of the main content bar) can be provided at low power through the user's mobile receiver according to the point of time when the advertisement appears or the time when the advertisement is displayed on the screen of the main content.

In accordance with one or more embodiments of the present invention, in order not to interfere with program immersion by viewers or listeners, it is desirable that the time period during which broadcast or playback main content ends and the ending credit is provided, The content can be provided at low power through the user's mobile receiver.

1 is a diagram for explaining an associated content providing system for providing related content using a sound wave ID according to an embodiment of the present invention.
2 is a diagram for explaining an embodiment of a mobile receiver.
3 is a diagram for explaining another embodiment of a mobile receiver.
4 is a diagram for explaining a related content providing method according to an embodiment of the present invention.
5 is a diagram for explaining a method of providing related contents according to an embodiment of the present invention.
6 is a diagram for explaining a method of providing related contents according to an embodiment of the present invention.
7 is a diagram for explaining a related content providing method according to an embodiment of the present invention.
8 is a diagram for explaining main contents.
9 is a diagram for explaining that the acoustic recognition module is activated according to a predetermined period for operating at a low power.
10 is a diagram for explaining that the sound recognition module is activated in accordance with the extraction time of the sound wave ID during broadcasting and reproduction time of the main content.
11 is a view for explaining an activation period of the sound recognition module when the existence of a sound wave ID is estimated to exceed a preset probability through a process of extracting a sound wave ID.
FIG. 12 is a diagram for explaining how the period of the sound recognition module is adjusted according to an event in which another application using a microphone, which is a shared resource inside the mobile receiver, is activated.
FIG. 13 is a diagram for explaining how the period of the sound recognition module is adjusted according to an event in which another application using a microphone, which is a shared resource, is executed in the mobile receiver.
14 is a diagram for explaining how the sound recognition module is notified of an event of a mobile receiver to operate at a low power.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other objects, features, and advantages of the present invention will become more readily apparent from the following description of preferred embodiments with reference to the accompanying drawings. However, the present invention is not limited to the embodiments described herein but may be embodied in other forms. Rather, the embodiments disclosed herein are provided so that the disclosure can be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

Also, terms used herein are for the purpose of illustrating embodiments and are not intended to limit the invention. In the present specification, the singular form includes plural forms unless otherwise specified in the specification. The terms "comprises" and / or "comprising" used in the specification do not exclude the presence or addition of one or more other elements.

Hereinafter, the present invention will be described in detail with reference to the drawings. In describing the specific embodiments below, various specific details have been set forth in order to explain the invention in greater detail and to assist in understanding it. However, it will be appreciated by those skilled in the art that the present invention may be understood by those skilled in the art without departing from such specific details. In some instances, it should be noted that portions of the invention that are not commonly known in the description of the invention and are not significantly related to the invention do not describe confusing reasons for explaining the present invention.

■ Definitions of Terms

- Content: Information that provides educational, cultural, informational, and / or entertainment value as structured digital data in the form of video, sound, or a combination of sound and image.

- Content playback device: a device for playing back content, such as a TV, a radio, a desktop computer or a mobile device such as a laptop computer, a tablet computer, a smart phone, a wearable computer, Terminal, and may further include a monitor or video output terminal for displaying an image.

- Content: i) main content or related content, or ii) main content and related content.

- Main content: Content that is output (broadcast or played) through the content playback device.

- related content: content output through a mobile receiver, including information such as sound, video, document, Internet address, etc. associated with the background content for delivering the content or content of the main content output via the content playback device and / Means a digital identifier value. For example, the related content may be a content that is difficult to be included in the main content due to limitation of the content organization or restriction of laws, content that induces the viewer or listener to participate, content about the indirect advertisement, , Or information related to a product that is derived from an indirect advertisement, or content related to purchase of a product.

- Mobile receiver: A device capable of outputting related contents. It includes a microphone and an analog-digital converter (ADC) for analyzing the sound contained in the main content. It analyzes the sound of the main content to identify the main content A mobile device that grasps the relative time position of the main content and then provides the content viewer or listener with the related content related to all or part of the main content in a form that can be recognized visually or audibly. At this time, the viewer or listener may be a person who owns the mobile receiver. For example, the mobile receiver may be a mobile device such as a laptop computer, a tablet computer, a smart phone, or a wearable computer.

Product Indirect Advertising or Indirect Advertisement: An advertisement that is output as part of the main content video or part of the sound as an advertisement, such as a product, service, brand logo, tag line, etc. included in all or part of the main content. Product indirect advertising or indirect advertisement is a concept that includes sponsored advertisement (advertisement which is provided free of charge at the time of producing the content or exposed at a discounted price to present the spot as part of the content), and the related content may be indirect advertisement ≪ / RTI >

- Indirect advertising products: Products or services that are output as part or all of the main content by indirect advertising or indirect advertising. It is used to mean not only products with specific forms but also services, brands, logos, and tag lines. do.

- Identifier: The sound ID is either (1) digital data (non-audible sound ID) represented by the non-audible sound artificially inserted into the sound of the main content, or (2) extracted from the sound contained as part of the main content (Audible fingerprint ID) that represents the characteristics of the sound that can be received. Artificially inserted non-audible sound IDs are characterized by non-audible sound waves that are inaudible to humans using psychoacoustics. Audible fingerprint IDs that characterize the sound do not require artificial insertion, but can be extracted only when there is sound in the main content. (2) identifying relative time locations within the main content; (3) identifying associated content associated with an indirect advertisement in the main content; (4) May be used to identify associated content associated with indirect advertising at a relative time location within the content. The non-audible sound wave ID may include a serial number, a meaningful word or an internet link. The acoustic fingerprint ID of the audible fingerprint includes SVD / EVD (Singular / Eigen Value Decomposition), PCA (Principal Component Analysis), Peak Landmarks Detection, Zero-crossing counting, Mel Frequency Cepstral Coefficient , Perceptual Linear Prediction Coefficient (PLPC), Linear Prediction Coefficient (LPC), and the like can be extracted as a hash function, and the extracted value can be used as a hash key. For the sake of description of the non-audible sound wave ID, the non-audible sound wave ID may indicate one of non-audible sound of an analog form in which digital data is modulated, digital data before modulation, and digital data after demodulation. For the sake of convenience, the acoustic fingerprint sound ID obtained by extracting the characteristics of the sound is extracted from the extracted characteristic digital data, a part of the main content sound which can extract the characteristic digital sound data, A portion of the content sound, and feature digital data extracted by the mobile receiver.

Acoustic Recognition Module: Used to collectively refer to at least one component used for sensing and extracting a sound wave ID, and may include at least one of a microphone, an ADC, a sound wave ID extractor, and an associated content providing application It can be called collectively.

■ Related content providing system

FIG. 1 is a diagram for explaining a system (hereinafter, referred to as an 'associated content providing system') for providing an associated content according to an embodiment of the present invention using a sound ID.

Referring to FIG. 1, an associated content providing system is a system for providing, through a mobile receiver 200, related content related to all or a part of a main content output from a content reproducing device 100. [

The mobile receiver 200 may provide associated content that includes, for example, an advertisement associated with at least one of an image and a sound output as main content. As an example, the mobile receiver 200 may provide associated content for an indirect advertisement product, which is displayed as an image output from the main content or referred to as an audio. According to one embodiment, the mobile receiver 200 may provide associated content associated with each of a plurality of indirect advertisement products, which are displayed as images output from the main content or referred to acoustically.

The associated content providing system according to the present embodiment may include a content reproducing device 100, a mobile receiver 200, and a server 300. [

The content reproduction device 100 may output video and / or sound. The mobile device 200 can extract at least one sound wave ID from the sound output by the content reproduction device 100. [ For example, a sound wave ID may be associated with an associated content DB (Database, DB) - a sound wave ID and associated content are associated with each other via a DB- corresponding to each other.

For example, a sound wave ID may be used to identify a main content, identify a relative time location within the main content, identify associated content associated with an indirect advertisement in the main content, And may be used to identify associated content associated with (associated with).

The content reproduction device 100 includes a speaker or an audio output terminal for reproducing sound such as a TV, a radio, a desktop computer, and a mobile device (e.g., a laptop computer, a tablet computer, a smart phone, a wearable computer, And a monitor or video output terminal for additionally displaying an image.

The mobile receiver 200 can extract the sound wave ID included in the sound of the main content outputted from the content reproducing apparatus 100 and can display the related content corresponding to the sound wave ID in a form that can be recognized visually or audibly A content viewer or a listener. At this time, the viewer or listener may be a person who owns the mobile receiver. For example, the mobile receiver 200 may include mobile devices such as laptop computers, tablet computers, smart phones, wearable computers, and the like. In addition, the mobile receiver 200 may include a microphone (not shown) and an ADC (not shown) for receiving a sound ID.

 In this embodiment, the mobile receiver 200 transmits the sound wave ID to the server 300 when extracting the sound wave ID, receives the associated content corresponding to the sound wave ID from the server 300, and displays it as sound or image have. For example, the mobile receiver 200 may display the associated content corresponding to the sound wave ID to the user in the form of a notification message, an image, and / or a voice.

In this embodiment, the mobile receiver 200 is a portable device such as a mobile device such as a laptop computer, a tablet computer, a wearable computer, a smart phone, etc., and has a microphone (not shown) (Not shown), a computer processor (not shown), a memory (not shown), and a program (not shown). Here, the program is loaded into the memory under control of the computer processor to recognize the sound wave ID, transmit the recognized sound wave ID to the server 300, notify the related content transmitted from the server 300 Message, image, and / or voice).

Next, we explain how sound IDs can be included in the content.

Techniques for inserting and extracting sound wave IDs in main content may utilize techniques disclosed in the following patent application documents.

For example, a technology for inserting an ID into a sound and a technique for extracting a content ID inserted in a sound are disclosed in, for example, Patent Application No. 10, filed by the Korean Intellectual Property Office on Apr. 12, 2012 -2012-0038120 (a position estimation method and position estimation system for estimating the position of a mobile terminal using an acoustic system and an acoustic system used therein).

As another example, a technique disclosed in Patent Application No. 10-2012-0078410 (Proximity Data Collection Method and System) filed by the present inventor on July 18, 2012 in the Korean Intellectual Property Office (including an ID in an acoustic signal, A technique of recognizing and extracting an ID included in a sound signal) can be used.

As another example, the technology disclosed in Patent Application No. 10-2012-0053286 (system for identifying speakers and position estimation system using the same) patented by the present inventor on May 18, 2012 A technique of including an ID in a signal and recognizing and extracting an ID included in the sound signal) may be used.

As another example, a technique disclosed in Patent Application No. 10-2012-0078446 (Method and apparatus for calculating intimacy between users using proximity information) filed by the present inventor on July 18, 2012 in the Korean Intellectual Property Office ID, and recognizing and extracting the ID included in the sound signal).

As another example, a technique disclosed in Patent Application No. 10-2013-0107604 filed on September 6, 2013 by the inventor of the Korean Intellectual Property Office (a method and apparatus for transmitting and receiving sound waves using time-varying frequency-based symbols) (A technique of including digital information in an acoustic signal and recognizing and extracting digital information included in the acoustic signal) can also be used.

On the other hand, the techniques disclosed in the above-mentioned Patent Application Nos. 10-2012-0038120, 10-2012-0078410, 10-2012-0053286, 10-2012-0078446, and 10-2013-0107604, The disclosure of which is incorporated herein by reference.

Techniques for extracting the audible fingerprint sound ID corresponding to the characteristics of the sound included in the main content can use the techniques disclosed in the following patent application documents.

For example, Korean Patent Registration No. 10-1310943 (Mar. 9, 201) (system and method for providing content-related information related to broadcast contents), Korean Patent Registration No. 10-1313293 (March 24, 2013) (A system and method for providing additional information of broadcast contents), Korean Patent Registration No. 10-0893671 (April 9, 2009) (generation and matching of hashs of multimedia contents).

On the other hand, the techniques disclosed in the above-mentioned Korean Patent Registration Nos. 10-1310943, 10-1313293, and 10-0893671 are all hereby incorporated herein by reference.

The server 300 can receive the sound wave ID from the mobile receiver 200 and selects the associated content corresponding to the sound wave ID by referring to the associated content DB (not shown) corresponding to the related content for each sound wave ID, And transmits the associated content to the mobile receiver 200.

In this embodiment, the server 200 may be, for example, an advertisement server that stores and manages advertisements, or may be a multimedia instant messenger (MIM) service server that transmits text and / or multimedia data to friends belonging to the same group have.

When the server 200 is implemented as an MIM service server, the server 200 transmits the associated content to a client program that interacts with the MIM service server.

■ Mobile receiver extracting sound wave ID

2 is a diagram for explaining an embodiment of a mobile receiver.

2, the mobile receiver 200 includes a microphone 201, an ADC 203, a processor 207, a memory 209, a storage device 211, a display unit 213, and a communication unit 215 can do.

According to one embodiment, the microphone 201 can sense the sound output from the content reproduction apparatus 100. [ That is, the received sound can be converted into an electrical signal.

According to one embodiment, the ADC 203 converts the sound, which is an analog sound signal detected by the microphone 101, into digital data. In one embodiment, the ADC 203 may convert the sound wave signals within a predetermined frequency range sensed by the microphone or a maximum frequency range allowed by the hardware to digital data.

According to an alternative embodiment, the ADC 203 may convert to digital data only when the sound has an intensity above a predetermined threshold. In addition, the ADC 203 may convert digital data only when the sound within a predetermined frequency range has a predetermined threshold value or more.

The processor 207 may be a central processing unit (CPU) of a general purpose computer as an apparatus for interpreting and executing computer instructions, and may be, for example, an application processor (AP) for a mobile device such as a smart phone, (209). ≪ / RTI >

The memory 209 may be implemented as a volatile storage device such as a random access memory (RAM) and may include an operating system (OS), application, data, . The storage device 211 may be a non-volatile storage device for storing data, such as a hard disk drive (HDD), a flash memory, or a SD (Secure Digital) memory card.

The display unit 213 can display the contents to the user as audio, video, and / or text. For example, the display unit 213 may be implemented as a device such as a display unit and a sound output terminal or a speaker connected thereto.

And a communication unit 215 for communicating with the outside. For example, the communication unit 215 can communicate with the server 100. [

According to one embodiment, the memory 209 may be loaded with an associated content provisioning application 223 under the control of the processor 207. The related content providing application 223 receives the sound using the microphone 201 and the ADC 203 and extracts the extracted sound wave ID from the sound through the communication unit 215, .

According to one embodiment, the associated content providing application 223 may, as an additional function, determine whether the sound sensed by the ADC 203 has an intensity above a threshold value, and only when the sound has intensity above a predetermined threshold And extracting the sound wave ID from the sound wave ID.

According to another embodiment, the associated content providing application 223 may, as an additional function, determine whether the sound sensed by the ADC 203 has an intensity above a threshold value within a predetermined frequency range, And extracting the sound wave ID from the sensed sound only when the intensity of the sound wave is equal to or greater than a predetermined threshold value within the range.

According to another embodiment, the operation of detecting and extracting the sound wave ID may be implemented in separate hardware rather than in the associated content providing application 223. That is, the sound wave ID extracting unit 205 shown by a dotted line in FIG. 2 is implemented in hardware, and the mobile receiver 200 may be further configured to include a sound wave ID extracting unit 205. [ In this case, the sound-ID extracting unit 205 extracts the sound-wave ID from the digital data converted by the ADC 203, and the associated-content providing application 223 extracts the sound-wave ID extracted by the sound- To the server (300) via the network (215).

3 is a diagram for explaining another embodiment of a mobile receiver.

3, the mobile receiver 400 includes a microphone 401, a low power acoustic intensity measuring unit 402, an ADC 403, a sound wave pattern recognizing unit 404, a processor 407, a memory 409, A storage unit 411, a display unit 413, and a communication unit 415. [

  3 is compared with the embodiment of FIG. 2, it can be seen that the sound wave ID extracting unit 205 of FIG. 2 is divided into a low power acoustic intensity measuring unit 402 and a sound wave pattern recognizing unit 404 have.

 According to one embodiment, the low power acoustic intensity measuring unit 402 consumes low power but is continuously activated, and the sound wave pattern recognizing unit 404 receives relatively more power than the low power acoustic intensity measuring unit 402 Consumed but mostly deactivated and activated intermittently. The low power acoustic intensity measuring unit 402 can sense the sound within a specific frequency range of the sound and can continuously judge whether the intensity of the sensed sound has a predetermined threshold value or more.

In the present embodiment, the low-power acoustic intensity measurement unit 402 determines whether the sound input through the microphone 401 has a predetermined threshold value or more within a predetermined frequency range. If the intensity has an intensity equal to or greater than the threshold value, the low power acoustic intensity measuring unit 402 can activate the ADC 403 and the sound wave pattern recognizing unit 404 which were inactivated for the low power operation. At this time, the activated ADC 403 can convert the sound into digital data and transmit it to the activated sound pattern recognition unit 404.

According to an exemplary embodiment, the sound wave pattern recognition unit 404 receives the sound of the digital data format from the ADC 403 and extracts the sound wave ID if the sound wave ID is included in the received sound. At this time, if the processor 407 is inactivated to operate at a low power, the processor 407 is activated. In the present embodiment, the sound wave pattern recognition section 404 provides the extracted sound wave ID to the associated content providing application 423.

According to another embodiment, the sound wave pattern recognition unit 404 receives sound in the form of digital data from the ADC 403, compares the received sound with at least one previously stored sound wave ID, After determining whether or not the sound wave ID can be extracted from the received sound, if it is determined that the sound wave ID can be extracted, the sound wave ID is directly extracted. According to an alternative embodiment, the sound wave pattern recognition unit 404 determines whether or not a sound wave ID can be extracted from the sound transmitted from the ADC 403. If it is determined that the sound wave ID can be extracted, the ADC 403 To the associated content providing application 423, and the associated content providing application 423 can extract the sound wave ID.

Meanwhile, techniques for detecting and extracting a sound wave ID by using separate hardware can use the techniques disclosed in the following patent application documents.

For example, a technique for extracting a sound wave modulated by a non-audible sound at a low power is disclosed in, for example, Patent Application No. 10-2013-0141504 filed by the Korean Intellectual Property Office on Nov. 20, A method of receiving a power acoustic wave and a mobile device using the same).

 On the other hand, the techniques disclosed in the specification of the above-mentioned Patent Application No. 10-2013-0141504 are all incorporated herein by reference and are treated as part of the specification.

Meanwhile, the memory 409 may be loaded with a DMB application or a moving picture player operating under the control of the processor 407 or a codec 221 required to reproduce contents through it. The DMB application or the video player or the codec 221 may be stored in the storage device 411 and may be loaded into the memory 409 and operated by a user's command or the like. Although not shown, when a message arrives from the server 300, a message processing program (not shown) may be loaded into the memory 409 and the message may be displayed on the display unit 413.

On the other hand, components not described in the embodiment of FIG. 3 are the same as or similar to those of the components having the similar reference numerals in the embodiment of FIG. 2, and therefore, the description of the embodiment of FIG. 2 should be referred to.

■ Low power sound wave ID extraction

In the present specification, for the purpose of the description of the present invention, the term 'acoustic recognition module' includes a microphone, an ADC, a sound wave ID extracting section, an associated content providing application, a low power acoustic intensity measuring section, Which at least one component performs the action of sensing or extracting a sound wave ID or activating a component that senses or extracts a sound wave ID.

According to an embodiment of the present invention, the acoustic recognition module is intermittently activated to operate at a low power, detects sounds when activated, and detects and extracts sound IDs. In this case, the activation module is required to execute a necessary hardware module, execute required software, or input necessary input information in order to enable the sound recognition module to be able to detect and extract the sound ID while minimizing power consumption in order to minimize power consumption. And requests from the operating system.

The intermittent activation for operating at low power is shown in a specific embodiment in Fig. 9 to Fig.

Intermittently activated to operate at low power may be, for example, (a) that the acoustic recognition module is activated in a 10 minute cycle, as shown in FIG. As another example, (B) a sound recognition module may be activated and operated irregularly in consideration of a broadcast time of a main content including a sound ID to be extracted or a time characteristic of extracting a sound ID from the main content And this is shown in FIG. As another example, (c) considering the result of attempting to receive the sound wave ID of the preceding cycle, the sound recognition module may be activated at an irregular period and is shown in FIG. As another example, (d) an acoustic recognition module may be activated and operated at an irregular period according to an event issued within the mobile receiver, which is shown in Figs. 12-14. Figure 14 shows how an event occurring inside a mobile receiver is delivered to an acoustic recognition module.

(A) Periodic activation

9 is a diagram for explaining that the acoustic recognition module is activated according to a predetermined period for operating at a low power.

FIG. 9A shows that the sound recognition module is activated at the same time in preparation for the time at which the main content is output. FIG. 9B shows that the sound recognition module is activated at the time of completion or completion of the main content in preparation for the time at which the main content is output. FIG. 9C shows that the sound recognition module is activated a plurality of times (for example, twice) with a predetermined period set for the time at which the main content is outputted.

(B) Activation of broadcast time and sound wave ID extraction time of contents

10 is a diagram for explaining that the sound recognition module is activated in accordance with the extraction time of the sound wave ID during broadcasting and reproduction time of the main content.

According to one embodiment, when the sound recognition module is periodically activated, the activation period may be determined by taking into account the succession of the temporal position of the sound ID extracted from the main content output by the content reproduction device 100 and the extractable sound ID Can be determined. For example, the sound wave ID in the content can be extracted repeatedly for 2 minutes continuously, and the whole 4 minutes pattern which is difficult to be extracted for 2 minutes thereafter may be continuously repeated throughout the content. At this time, if the sound recognition module determines whether the sound ID can be extracted at a maximum of 2 minutes, it can intermittently attempt to extract the sound ID even while extracting the sound ID extracted from the contents in a short time, Consumption can be reduced.

According to another embodiment, when the broadcast date and time of the content output from the content reproduction device 100 can be known in advance, the sound recognition module can be activated at a shorter time at a corresponding time when the content is broadcasted. For example, assume that the period of the acoustic recognition module is set to 10 minutes. At this time, if the specific contents of the TV are broadcasted every Saturday at 6:00 PM for one hour, and the sound ID can be extracted from the contents, the sound recognition module will be provided every week from 6:00 pm to 7:00 pm every five minutes Can operate. Alternatively, the acoustic recognition module may be continuously inactive and operate on a period of one hour from 6 pm to 7 pm every Saturday.

Now, the above-described two embodiments will be described with reference to FIG. The mobile receiver 200 or 400 knows the time at which the sound wave ID can be extracted, that is, the playing time of the sound waves ID1, ID2, and ID3 in Fig. This time may be received by the sound recognition module of the mobile receiver 200 or 400 via the server. When the reproduction time of the sound wave ID1 is reached, the sound recognition module is activated to extract the sound wave ID1. If the extraction of the sound wave ID1 fails at this time, the sound recognition module may determine that the user of the mobile receiver 200 or 400 is not watching or listening to the main content 1. [ If extraction of the sound wave ID1 is successful, the mobile receiver 200 or 400 can receive and display the associated content corresponding to the sound wave ID1 from the server 300. [ Similarly, when the sound recognition module is activated in accordance with the reproduction time of the sound waves ID2 and ID3, and the extraction succeeds after the extraction of the sound wave ID, the mobile receiver 200 or 400 receives the associated content corresponding to the sound wave ID from the server 300 Can be displayed. In this way, the mobile receiver 200 or 400 can irregularly activate the sound recognition module only at a time predicted to be able to extract the sound ID, and can display the sound ID and display the associated content at low power.

(C) Activation according to the result of extraction of previous sound wave ID

11 is a view for explaining an activation period of the sound recognition module when the existence of a sound wave ID is estimated to exceed a preset probability through a process of extracting a sound wave ID.

Referring to FIG. 11, the mobile receiver 200 or 400 is activated in accordance with the originally planned period. At this time, although the sound recognition module judges that the sound wave ID is extractable (hereinafter, it is described as judging existence), the extraction may fail. That is, for example, when the extracted sound wave ID is equal to or smaller than a preset value of the hamming distance from the specific sound wave ID existing on the associated contents DB, it can be estimated that the sound wave ID has failed to be extracted but exists. Alternatively, CRC (Cyclic Redundant Check) information for identifying the presence or absence of extraction in the sound wave ID itself can be confirmed and confirmed using the CRC (Cyclic Redundant Check) information. Alternatively, it can be estimated that the sound-wave ID exists by checking whether the sound-wave ID is present or not by measuring the SNR (Signal-to-Noise Ratio) and determining whether it is equal to or greater than a preset threshold value. Assume that the presence of the sound ID is confirmed in the above manner, but the extraction fails. At this time, the sound recognition module cancels the activation of the sound recognition module scheduled after the cycle 3, which is the cycle to be originally performed, so as not to miss the extractable time zone of the main content, and activates the cycle 4, which is a cycle shorter than the cycle 3 And the period after period 4 can be adjusted to the original period. Therefore, by adjusting the frequency with the result of the extraction of the sound ID, the extraction probability of the sound ID is increased, and the existing activation frequency is maintained as it is, so that it can operate at low power.

(D) activation according to the internal events of the mobile receiver

According to an embodiment, the acoustic recognition module may be activated and operated at an irregular period according to an event issued inside the mobile receiver 200 or 400. [ This will be described with reference to FIGS. 12 and 13. FIG.

12 is a diagram for explaining how the period of the sound recognition module is adjusted according to an event in which another application using the microphone 201 or 401, which is a shared resource within the mobile receiver 200 or 400, is activated.

Multiple applications may be running concurrently on the mobile receiver 200 or 400. At this time, the microphone 201 or 401, which is a part of the mobile receiver 200 or 400, is a resource shared by a plurality of applications and is used in a first-come-first-served (FCFS) It can be shared in a way that can not be done. In such a case, the intermittently activated acoustic recognition module may cause the use of the microphone 201 or 401 to fail when another application attempts to use the microphone 201 or 401. Therefore, when the application that can use the microphone 201 or 401 is detected, or when the attempt to use the microphone 201 or 401 fails, the sound recognition module delays the activation until the application is terminated, The total number of activations can be reduced without interfering with the operation of the application using the application 201 or 401, thereby enabling operation at low power.

For example, as shown in FIG. 12, the application 1 included in the sound recognition module may be loaded in the memory 209 or 409 of the mobile receiver 200 or 400 at time t1. It is notified that the application 2 having a high probability of using the microphone 201 or 401 before the time t2 has been executed or the application of the sound recognition module Before executing the sound ID extraction, the currently running application information may be obtained from the operating system, and the application 2 may be detected (the method of sensing or notifying is described later in detail with reference to FIG. 14). When the application 2 is running, the sound recognition module can cancel the scheduled activation at time t2 and newly set the cycle for the next activation after time t2. The newly set period may be a period for activating the acoustic recognition module at a time that is twice the period 5 from t1. Also, as shown in FIG. 12, the newly set period can be modified at a time at which the application 2 notifies the time period t3, at which the application 2 is notified, to be a controlled period so as to become the time at which the sound recognition module is activated .

Taking another specific application as an example, when the camera application is activated, the application may be exclusively using a part of the sound recognition module for movie shooting. At this time, the sound recognition module may stop intermittent activation according to the period so that the camera application exclusively uses a part of the sound recognition module, and may have an irregular period activated when the camera application is terminated.

13 is a diagram for explaining how the period of an acoustic recognition module is adjusted according to an event in which another application using a microphone (201 or 401) as a shared resource within a mobile receiver (200 or 400) is activated.

Suppose that the sound recognition module of the mobile receiver 200 or 400 is activated at time t4 to attempt to extract the sound ID. Then, the originally planned period may be the cycle 7, and the acoustic recognition module may be activated again at time t6. However, in this case, the operating system may notify the sound recognition module of the fact that the application 4 using the microphone 201 or 401 is executed at time t5 (the method of detecting or notifying the sound recognition module will be described later with reference to FIG. 14 do). If the operating system provides sharing of the use of the microphone 201 or 401, the sound recognition module that has received the execution notification of the application 4 may be connected to the microphone 201 or 401, the ADC 203 or 403 processor 207 or 407 It is possible to share hardware and software resources used by application 4 and extract sound ID. For example, the operating system simultaneously supplies data, which is converted into digital data through the ADC 203 or 403, to the microphone 201 or 401 simultaneously with the application 4 as well as the sound recognition module, Can receive. Through the support of such an operating system, the acoustic recognition module can be activated separately to extract the sound ID from the power consumed by the mobile receiver 200 or 400, as compared to attempting to extract the sound ID. In the case of such an activation, the acoustic recognition module can operate at low power by canceling the schedule that is activated at the time t6, which was originally planned.

For example, when a telephone call application is activated, the application basically uses the microphone 201 or 401 and the ADC 203 or 403 and activates the processor 207 or 407, , It is possible to extract the sound ID with low power. As another example, when a moving picture player or a DMB application or a codec 221 or 421 operates, it can be assumed that the mobile receiver 200 or 400 also plays a role of a content reproducing device. Therefore, The recognition module can be activated.

In Figures 12 and 13, it has been described that the sound recognition module can estimate whether a particular application will use the microphone 201 or 401, and this estimation is based on the assumption that each application views the hardware usage permission information allowed by the operating system Can be confirmed. At this time, a list of running applications can also be obtained from the operating system.

12 and 13 illustrate that the sound recognition module has an irregular period according to an event inside a mobile receiver in which a specific application is executed or terminated, but the type of the event is not limited thereto. For example, Figures 12 and 13 may be that an application is transitioning from a background mode to a foreground mode or from a foreground mode to a background mode, as well as a particular application being executed or terminated. As another example, FIG. 12 may be an event in which one or more of the internal components of the mobile receiver 200 or 400 other than the specific application is active, and FIG. 13 may be an inactive event. Specifically, the display of the display unit may be an event that is activated or deactivated. As another example, the event may be that the mobile receiver 200 or 400 receives a push message from the push server.

FIG. 14 is a diagram for explaining how the sound recognition module is notified of an event of the mobile receiver 200 or 400 to operate at a low power as shown in FIGS. 12 and 13.

Figure 14a illustrates how an operating system is notified when it notifies applications that it is running that certain events have occurred. 14 (b) shows how a specific application notifies an application corresponding to the sound recognition module that an event has occurred. Figure 14c illustrates how the sound recognition module is informed by the push server that the message is received from the server at low power.

14 (a) will be described as an example.

The mobile receiver 200 or 400 may include the capability to receive a telephone call. In this case, when a call comes in, the operating system activates the application that receives the call, and forwards the call reception event to the filter of each application so that the application responds to the incoming call. For example, application 5 may be an application that handles incoming calls, and application 6 may be an application that is playing music. When a call comes in, the operating system forwards the incoming event to each filter in the application. Since the application 6 must stop music playback when a call is received, the filter of the application 6 may be set to pass the telephone reception event. The filter of application 5 may be set to pass a phone reception event because the user must prepare a user interface and make a telephone call when a telephone call is received. It is assumed that the digital data input through the microphones 201 or 401 and input through the ADC 203 or 403 in the operating system can be simultaneously transmitted to various applications. In this case, the sound recognition module can receive the sound digital data and perform the sound ID extraction. Therefore, an application corresponding to the application of the sound recognition module can set the filter reception event to pass through the filter. That is, an event required to operate at low power among the events sent by the operating system can be set to pass through the filter, and the sound recognition module can extract the sound wave ID at low power.

14 (b) will be described as an example.

The application 2 of FIG. 12 and the application 4 of FIG. 13 respectively notify the sound recognition module of activation of the application and termination of the application, respectively, so that the sound recognition module can operate at low power. Assume that application 7 in FIG. 14 b) is application 2 in FIG. 12 or application 4 in FIG. At this time, the application 7 can transmit an event to the operating system to be delivered to other applications. This event may be the activation event of the application 2 in Fig. 12 or the ending event of the application 4 in Fig. The event is passed to the operating system, and the operating system can forward the event to the filters of all applications. At this time, the filter of the application included in the sound recognition module may be set to pass the activation or termination event. In this way, the sound recognition module can be notified of activation or termination of a specific application. In another example, the sound recognition module may use a method of receiving the use or termination of the hardware or software to be used as an event, rather than activation or termination of the application.

14 (b) will be described as another example.

The broadcast or playback schedule of the main content 1 and the main content 2 in Fig. 10 may be known in advance. At this time, the application 7 in Fig. 14B may be the same application as the sound wave ID extraction application. The sound ID extraction application can receive the broadcast or playback schedule of the main contents 1 and 2 from the server and register the corresponding time as an alarm of the operating system. When the registered alarm reaches the corresponding time, it transmits the registered event to the filter of each application. Since application 5 functions independently of this alarm event, the filter of application 5 does not forward this event to application 5. However, the filter of the application included in the sound recognition module (the filter of the sound ID extracting application in Fig. 14B) can receive the alarm event and start extracting the sound ID. The time at which the sound wave IDs of the main contents 1 and 2 end in the same manner are registered as an alarm of the operating system so that the application included in the sound recognition module can terminate the sound wave ID extraction.

14 (c) will be described as an example.

14C). The mobile receiver 200 or 400 receives the message from the push server and is delivered to the application via the operating system. Suppose that the application receiving the push message is application 8. Specifically, assume that application 8 is a MIM service. At this time, the mobile receiver 200 or 400 receives the push message from the server and is delivered to the operating system, and the operating system delivers the push message to the application filters. The filter of application 8 may be set to pass a push message to perform the MIM function. The application 9 may be set not to pass the push message since the application 9 operates independently of the push message. The mobile receiver 200 or 400 may have received a push message while activating only minimal hardware and software to receive a push message to reduce power consumption and operate. In this case, the MIM service receiving the push message from the operating system takes an action appropriate to the push message. The filter of the application portion of the acoustic recognition module may be set to pass this push message. In such a case, the sound recognition module activates the sound recognition module when various components of the mobile receiver 200 or 400, such as an AP, a display, etc., are activated for push message processing, The hardware and software necessary for the sound recognition module can be shared with the application 8. [ Through the sharing of such hardware and software, the acoustic recognition module can operate at low power from the perspective of the mobile receiver 200 or 400.

■ How to provide associative content that places an associated content database on the server

4 is a diagram for explaining a related content providing method according to an embodiment of the present invention.

Assuming that the related content providing method according to the present embodiment is applied to the system of Fig. 1, the embodiment of Fig. 4 will be described.

The content reproduction apparatus 100 can output the main content (S101). The sound recognition module of the mobile receiver is activated according to the activation period (S103). The activation period in S103 may be various regular and irregular periods.

The sound recognition module performs an operation of extracting a sound wave ID (S105).

When an associated content providing application is being used as a part of the sound recognition module, the associated content providing application operating in the memory 209 of the mobile receiver 200 transmits the extracted sound ID to the server 300. The server 300 then searches for the associated content corresponding to the transmitted sound wave ID (S107). Thereafter, the server 300 transmits the selected related content to the mobile receiver 200, And displays it to the user (S109).

■ How to provide related content that puts an associated content DB in the mobile receiver

5 is a diagram for explaining a method of providing related contents according to an embodiment of the present invention.

Assume that the related content providing method according to the present embodiment is applied to the system of FIG. 1, and the embodiment of FIG. 5 will be described mainly on the difference from the embodiment of FIG.

The content reproduction apparatus 100 may output the main content capable of extracting the sound ID from the sound (S201). The acoustic recognition module of the mobile receiver is activated in accordance with the intermittent activation cycle (S203). The server 300 transmits the associated content corresponding to the sound wave ID to the mobile receiver (S205). The application operating in the memory 209 stores the associated content DB transmitted from the server 300 in the storage device 211 or updates the previously stored DB. The sound recognition module performs an operation of extracting a sound wave ID (S207). The application operating in the memory 209 searches the associated contents DB (S209) and displays the associated contents corresponding to the sound wave ID through the display unit 213 (S211).

5 differs from the embodiment of FIG. 4 in that the mobile receiver 200 directly selects the associated content corresponding to the sound wave ID. To this end, the server 300 may send some or all of the associated content DB to the mobile receiver 200 to update some or all of the associated content stored in the mobile receiver 200.

The time when the server 300 transmits the associated contents DB is not limited to the time shown in Fig. When the server 300 sends an associated content DB to update the associated content DB stored in the mobile receiver 200, it is not limited to a specific time.

■ How mobile receivers act as content playback devices

6 is a diagram for explaining a method of providing related contents according to an embodiment of the present invention.

It is assumed in the system of FIG. 1 that the mobile receiver 200 other than the content reproduction apparatus 300 outputs the main content, and the mobile receiver 200 is implemented to recognize the sound. Assume that the content providing method according to the present embodiment is applied to the system of FIG. 1, and the embodiment of FIG. 6 will be described focusing on differences from the embodiment of FIG.

The mobile receiver 200 may output the main content (S301). The sound recognition module of the mobile receiver is activated according to the intermittent activation period (S305). The sound recognition module extracts the ID from the sound output from the mobile receiver 200 (S307).

In this case, step S307 may be implemented by, for example, the following method.

First, a method for extracting an ID from an acoustic recognition module from a sound sensed by a microphone (201) and sensed by a speaker (not shown) which is a part of a display unit (213) provided in the mobile receiver (200).

Second, a method for extracting a sound wave ID by receiving sound data from an internal operating system, before digital data is output through a speaker (not shown), which is a part of the display unit 213 of the mobile receiver 200.

An input terminal of a microphone 201 of a headset (not shown) connected to the mobile receiver 200 is electrically connected to a small sound output terminal of a headset (not shown) A method for an acoustic recognition module to extract an ID from a signal.

The sound recognition module transmits the sound wave ID extracted in step S307 to the server 300 (S307). The server 300 searches for an associated content corresponding to the sound wave ID by referring to the related content DB-server 300 or storing it in a separate storage device (not shown) (S309). The server 300 transmits the search result to the mobile receiver 200, and the mobile receiver 200 displays the transmitted search result.

7 is a diagram for explaining a related content providing method according to an embodiment of the present invention.

In the system of FIG. 1, it is assumed that the mobile receiver 200 other than the content reproduction apparatus 100 outputs the main content, and the mobile receiver 200 is implemented to recognize sound. In this system of FIG. 1, It is assumed that the related content providing method according to the embodiment of the present invention is applied, and the embodiment of FIG. 7 will be described mainly on the difference from the embodiment of FIG.

The mobile receiver 200 may output the main content (S401). The sound recognition module of the mobile receiver is activated according to the intermittent activation period (S403). The server 300 transmits the associated content DB to the mobile receiver 200 (S405), and the mobile receiver 200 updates the associated content DB that it has stored. The sound recognition module extracts the sound wave ID from the sound output from the mobile receiver 200 (S407). The mobile receiver 200 refers to the associated content DB and searches for associated content corresponding to the sound wave ID (S409). The mobile receiver 200 displays the search result (S411).

The above-described step S407 may be implemented in the same manner as in step S307.

7 differs from the embodiment of FIG. 4 in that the mobile receiver 200 directly selects the associated content corresponding to the sound wave ID. To this end, the server 300 may send some or all of the associated content DB to the mobile receiver 200 to update some or all of the associated content stored in the mobile receiver 200.

The time when the server 300 transmits the associated contents DB is not limited to the time shown in Fig. That is, when the server 300 sends the associated content DB to update the associated content DB stored in the mobile receiver 200, the timing at which the server 300 transmits the associated content DB is not limited to a specific time .

■ Locations of sound wave IDs that can be extracted from the content

8 is a diagram for explaining main contents.

FIG. 8A illustrates an example of main contents. Each of these contents is composed of a combination of an image and sound, and a sound wave ID can be extracted from the main content corresponding time.

For example, assuming that the main content as shown in FIG. 8A is output in the embodiment of FIG. 4, the operation will be described.

The content reproduction apparatus 100 outputs the main contents (main contents 1, 2 and 3) as time elapses, and extracts sound wave IDs from these contents, respectively. The sound wave ID1 can be extracted from the sound when the main content 1 is outputted. The sound wave ID2 can be extracted from the sound when the main content 2 is outputted. The sound wave ID3 can be extracted from the sound when the main content 3 is outputted.

The mobile receiver 200 sequentially extracts each of the sound IDs, transmits the sound IDs to the server 300, and displays the associated contents corresponding to the sound IDs from the server 300.

Meanwhile, the mobile receiver 200 may intermittently extract the sound wave ID. This is the same as the above-described embodiments, and thus a detailed description thereof will be omitted.

FIG. 8B shows another example of the main contents output through the content reproduction device. These main contents are each composed of a combination of an image and sound, and a sound ID can be extracted from the sound.

In FIG. 8 (b), there is a difference from FIG. 8 (a) in that there are contents that can extract two different sound IDs different from each other.

For example, assuming that the main content as shown in FIG. 8B is outputted in the embodiment of FIG. 6, the operation will be described.

The mobile receiver 200 outputs the main content (main content 4, 5, 6) over time and extracts the sound wave ID from each of these main contents. The sound wave ID1 can be extracted from the sound when the main content 4 is outputted. The sound waves ID2 and ID3 can be extracted from the sound when the main content 5 is outputted. The sound wave ID4 can be extracted from the sound when the main content 6 is outputted.

The mobile receiver 200 sequentially extracts each of the sound IDs, transmits the IDs to the server 300, and displays the associated contents corresponding to the sound IDs from the server 300, respectively.

■ How to output related content without interrupting the flow

According to an embodiment of the present invention, the time for displaying the associated content through the mobile receiver may be a time that does not require the immersion of the viewer or the listener in the main content such as advertisement or ending credit in the main content. For example, when the main content is a drama and the related content is indirect advertisement product information derived from a drama, although the sound wave ID from which the related content can be obtained is located in the middle of the drama, Can be postponed to the time when the drama ends and the ending credit is displayed.

Meanwhile, a computer-readable recording medium on which a program for executing each of the steps constituting the above-described method with reference to FIG. 4 to FIG. 7 is recorded.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, And variations are possible.

The scope of the present invention should not be limited to the above-described embodiments, but should be determined by the scope of the appended claims and equivalents thereof.

100: Content playback device
200, 400: mobile receiver
300: server

Claims (1)

The method comprising the steps of: detecting sound output from a content playback device that plays back content (referred to as 'main content');
Converting the sensed sound into digital data;
Extracting a sound wave ID from the digital data;
Selecting an associated content corresponding to the sound wave ID; And
And displaying the associated content as video and / or audio by the mobile receiver.

Images