KR101873793B1 - Contents synchronization apparatus and method for providing synchronized interaction - Google Patents

Abstract

The present invention provides an apparatus and method for providing personalized augmented reality for synchronized content among devices capable of simultaneously sharing content among multiple users and personalized information acquisition and manipulation. The content synchronization apparatus includes a communication unit that communicates with another device capable of content synchronization and a status information acquisition unit that acquires status information on the content and the content from the other device through the communication unit upon receiving a synchronization command for sharing the content being executed by another device A control unit for sharing contents of another device, performing content synchronization with another device using the acquired content and status information, and configuring a screen on which content synchronization has been performed, and a display unit for displaying a configured screen .

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a content synchronization apparatus and method for providing synchronized interaction between devices,

The present invention relates to a technique for sharing and synchronizing content between devices.

Recently, researches have been actively conducted to provide users with more convenient and useful services by wirelessly connecting computers, various sensors, and various electronic devices. Particularly, as the use of electronic devices is increasing, the necessity of sharing contents among electronic devices is also increasing.

On the other hand, augmented reality is being utilized in various electronic products. Augmented reality is a technique of superimposing virtual objects on the real world seen by the user. Augmented reality, a concept that complements the real world with a virtual world, uses a virtual environment created by computer graphics, but the protagonist is a real environment. Computer graphics serve to provide additional information needed for the real world. The augmented reality means that the distinction between the real environment and the virtual screen is blurred by overlaying the virtual image on the real image that the user is viewing. Augmented reality technology, which is a mixture of real environment and virtual objects, allows the user to see the real environment and provides better realism and additional information. For example, when a smartphone camera illuminates the surroundings, information such as a location of a nearby shop, a telephone number, and the like can be displayed as a stereoscopic image.

There is provided an apparatus and method for providing a personalized augmented reality for synchronized content among devices capable of simultaneously sharing content among multiple users and acquiring and manipulating personalized information.

A content synchronization apparatus according to one aspect includes a communication unit that communicates with another device capable of content synchronization and a communication unit that receives a synchronization command for sharing content that is being executed by another device, A control unit that acquires status information to share contents of other devices, performs content synchronization with another device using the acquired content and status information, and configures a screen on which content synchronization is performed, And an output unit.

The content synchronizing apparatus according to another aspect includes a communication unit for communicating with another device capable of content synchronization and a status information generating unit for generating a status information on the content being executed, And a control unit for controlling the communication unit to transmit the enhancement information related to the content to another device.

According to another aspect of the present invention, there is provided a content synchronization method including: receiving a synchronization command for sharing content that is being executed by another device; acquiring status information on the content and the content to share contents of another device; And performing content synchronization with another device using the content and the status information.

In accordance with another aspect of the present invention, there is provided a device for synchronizing content, comprising: a content synchronization device for recognizing content and status information indicating the execution status of content, A synchronization executing unit that performs content synchronization with another device using the acquired content and status information, and an output unit that outputs the synchronized content.

According to another aspect of the present invention, there is provided a content synchronization apparatus comprising: a communication unit that communicates with a web server that provides a web service; and a synchronization unit that receives a synchronization command for sharing a web application as content that another device is executing, A web server for acquiring a web application from the acquired screen; acquiring user information for use in receiving a web service; accessing a web server using user information; Includes a synchronization executing unit for acquiring web application status information indicating the execution status of the running web application and reproducing the execution status of the web application being executed by another device using the identified web application and the acquired web application status information do.

It is possible to synchronize and execute contents executed in another device, and to provide personalized augmentation information to the synchronized contents. In addition, personalized augmented information may be executed to control the reproduction of the content, or another application may be executed simultaneously with the execution of the content to implement a personalized augmented reality.

FIG. 1 is a diagram illustrating a configuration of an augmented reality interaction providing system capable of sharing information among different devices, providing synchronized interaction, and providing personalized augmentation information.
Figure 2 is a flow diagram illustrating an example of an initial operation for the first device of Figure 1 to synchronize with a second device;
Fig. 3 is a diagram showing an example of an operation for allowing the second device of Fig. 1 to share contents of the first device. Fig.
Fig. 4 is a diagram showing an example of an operation for the first device of Fig. 1 to share contents of a second device. Fig.
5 is a diagram showing an example of a screen in which the first device shares and uses the contents of the second device.
6 is a diagram showing an example of the configuration of the first device of FIG.
7 is a diagram showing an example of the configuration of the control unit of Fig.
Figure 8 is a flow diagram illustrating an example of a method by which another device may synchronize a running web application.

Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. In the following description of the present invention, a detailed description of known functions and configurations incorporated herein will be omitted when it may make the subject matter of the present invention rather unclear. In addition, the terms described below are defined in consideration of the functions of the present invention, which may vary depending on the intention of the user, the operator, or the custom. Therefore, the definition should be based on the contents throughout this specification.

FIG. 1 is a diagram illustrating a configuration of an augmented reality interaction providing system capable of sharing information among different devices, providing synchronized interaction, and providing personalized augmentation information.

The augmented reality interaction providing system 100 is configured to provide content-to-user synchronized augmented reality interaction capable of sharing content and synchronizing shared content, and simultaneously acquiring and manipulating personalized augmentation information. The augmented reality interaction providing system 100 includes a first device 110, a second device 120, an augmented reality supporting server 130, a broadcast providing apparatus 140, a content providing server 150, . ≪ / RTI >

The first device 110 and the second device 120 can share each other's contents in synchronization. To this end, the first device 110 and the second device 120 may transmit content, status information, and augmentation information to each other to synchronize and share the content.

Here, the content may include an application running on the device providing the content, various types of data such as photographs, video, audio, and text being played or executed on the application, multimedia data, and meta information about the content. The application may include a general application, such as a media player, for playing the content, and a web application that provides various web services including providing content via communication with an external web server 160. [

Unlike the enhancement information, the meta information about the content may be information about the content itself, and information used to search for the content. For example, when the content is broadcast content, the meta information may include broadcast channel information on which the broadcast content is broadcast, a broadcast title, broadcast time information, broadcast time information indicating how many broadcasts are received, performer information, and the like.

The status information may include status information on the content (e.g., volume, playing time information, etc.). The status information may include various information used for content synchronization by reproducing the content execution status of another device as it is. In addition, the status information may further include status information (e.g., screen brightness, etc.) for the device of the device providing the content. For example, if the running application is a video playback program, the content is a video file being played back by the video playback program, and the status information may be playback volume, playback time information, and the like.

The enhancement information may include additional information that can be provided in addition to the content, playback control information that can control playback of the content, an application that can be executed in conjunction with the content, and the like. For example, when the content is a drama or a movie, the augmentation information may include introduction information about a shooting location, an OST, plot, an actor, and the like. If the content is a sports broadcast program, the augmentation information may include competition statistics information. If the content is a music program, the augmentation information may include a singer, music introduction information, and the like.

The augmentation information may be configured to enable interaction with a user, and may be configured to perform a preset operation according to the selection of the augmentation information of the user or to provide detailed information connected to the selected augmentation information. For example, services such as purchase and approval can be performed according to a user input signal for selecting the augmentation information.

At the same time, the first device 110 may provide personalized augmentation information to the user of the first device 110 along with the content. Likewise, the second device 120 may provide personalized information to the user of the second device 110 with the content. Although only two devices are shown in Fig. 1, there is no limit to the number of devices that can share contents with each other and can provide personalized augmented information.

The first device 110 and the second device 120 may be the same type of device or may be heterogeneous devices. For example, both the first device 110 and the second device 120 may be mobile terminals. Alternatively, the first device 110 may be a mobile terminal and the second device 130 may be an electronic device having a large display such as a TV (Television), a stationary computer.

The first device 110 and the second device 120 may be connected to each other via Bluetooth, Zigbee, Wireless LAN, home RF, UWB ≪ / RTI > Wide Band) and wireless 1394. However, the first device 110 and the second device 120 can perform content synchronization without communicating with each other to exchange information. For example, the first device 110 acquires and analyzes the content execution screen of the second device 120, recognizes the content and status information executed in the second device 120, and stores the recognized content in a separate It can be obtained from the database or the content providing server 150.

The augmented reality support server 130 is connected to the first device 110 and the second device 120 through a communication network and provides the augmentation information related to the content to the first device 110 and the second device 120 Lt; / RTI > The augmented reality supporting server 130 may be connected to one of the first device 110 and the second device 120 to provide the augmentation information.

The broadcast providing apparatus 140 transmits a broadcast signal including information on a broadcast program such as an AV stream and a broadcast program using a broadcast network such as a satellite, a terrestrial wave, the Internet, or a cable. The information on the broadcast program may include EPG (Electronic Program Guide) information and explanatory information on the individual broadcast program. The broadcast providing apparatus 140 is connected to the first device 110 and the second device 120 through a broadcasting network so that the first device 110 and the second device 120 can receive broadcasting contents through the broadcasting network have. The first device 110 and the second device 120 may be configured to receive and play broadcast content.

The content providing server 150 is a server that stores a plurality of contents and searches for and provides the requested contents. The content providing server 150 may be connected to at least one of the first device 110 and the second device 120 through a communication network. The content providing server 150 may be configured to provide the content requested by at least one of the first device 110 and the second device 120 such as the second device 120 to the second device 120 .

The web server 160 is a server for providing web services and the content providing server 140 can be connected to at least one of the first device 110 and the second device 120 through a communication network. When at least one of the first device 110 and the second device 120, for example, the second device 120, logs in using the user information, the web server 160 uses the same user information And provide web application status information indicating the execution status of the web application used by another device such as the first device using the web service to the second device 120. [ Here, the user information may include a user ID, a password, and other authentication information required for each subscriber to use the web service. The web application state information may include the state in which the second device 120 uses the web service, the visited web page history, other user information communicated through the web service, and the like.

Hereinafter, the case where the first device 110 is a mobile terminal and the second device 120 is a digital TV will be mainly described.

The first device 110 may be various types of mobile terminals such as a remote controller equipped with a display, a tablet PC, a smart phone, and a mobile phone. The first device 110 may be configured to provide an interaction function to a user of the mobile terminal, including an interface capable of inputting and outputting various modalities such as voice, text, image, and motion touch.

The first device 110 may synchronize the content of the second device 120 to provide the content to the user. Also, the first device 110 may provide the augmentation information associated with the content of the second device 120 together with the content. Various enhancement information on the content may be provided to the second device 120. However, if the enhancement information on the content is displayed without limitations, it is possible to block the intrusion of the viewers, and a plurality of members using the second device 120 , The augmentation information of interest for each member may be different.

In view of this, the first device 110 may be configured to provide personalized augmentation information to the user of the first device 110 among the augmentation information associated with the content of the second device 120, together with the content. The first device 110 personalizes the enhancement information provided by the second device 120 by using personal information such as profile or preference information of the user of the mobile terminal, . Accordingly, the user of the first device 110 may be provided with personalized augmentation information, or may search for or selectively use the desired augmentation information, while sharing the content of the second device 120. [

For example, the first device 110 may notify the user of the first device 110, along with the TV content, of the first device 110 to the first device 110, And may provide the user of the device 110 with enhancement information related to personalized TV content. The enhancement information associated with the TV content includes interactive interaction information that can be manipulated by a user. The user of the first device 110 selects the interaction information so that the user can operate the augmentation information related to the TV content and the operation on the selected interaction information of the user is performed so that the user performs an operation on the TV content Can be done.

The first device 110 also provides content, state information, and augmentation information of the first device 110 to the second device 120 so that the second device 120 can provide the content of the first device 110 Shared and synchronized. Accordingly, the user of the first device 110 views the second device 120 by allowing the second device 120 having the large display to share and display the content used by the first device 110 The user can use the content and the augmentation information of the first device 110 in synchronization with the first device 110. [ That is, while the user of the first device 110 is using the content being executed by the first device 110, the content is synchronized directly with the second device 120 used by the user who wants to share the content, And can provide synchronized augmented reality interactions that can be shared together.

For example, the user of the first device 110 may share the drama provided by the second device 120, and may provide information about the particular augmentation information, such as the drama, Can be retrieved. The user of the first device 110 may share the retrieved contents together with all the members using the second device 120. [

The enhancement information includes content reproduction control information, and the first device 110 can control reproduction based on the synchronized content itself in accordance with the selection of the reproduction control information. For example, the user of the first device 110 may temporarily stop the shared TV content without disturbing other members, search the broadcast contents of another channel through channel zapping, It is possible to individually operate the broadcast guidance for the content being broadcast.

The first device 110 can also synchronize and share the content of the second device 120 and can search the Internet related to the TV content without interrupting the viewing of the TV by other members or use a social networking such as a Twitter / And may be configured to perform a connected application that can be provided on a TV, such as communicating with an acquaintance who is viewing the same broadcast through the Internet.

Figure 2 is a flow diagram illustrating an example of an initial operation for the first device of Figure 1 to synchronize with a second device;

In the first device 110, types and specifications of devices that can be synchronized can be registered in advance. A plurality of devices capable of synchronization can be registered in the first device 110. In addition, applications used for synchronizing contents with each other can be preset in the first device 110 according to the types of devices that can be synchronized.

The first device 110 tracks (210) another device, for example, a second device 120, which is pre-registered and synchronizable within a range of possible synchronization. The user of the first device 110 may directly register an IP address and device identification information for communication with the second device 120 so that the first device 110 can synchronize with the second device 120 have.

The first device 110 makes a communication connection 220 to and from the second device 120 within a synchronizable distance range so as to send and receive a signal that can be synchronized. In this process, the first device 110 can control the second device 120 to activate the operation mode for performing synchronization with the first device 110. [

Although the first device 110 is described as performing the initial operation to synchronize with the second device 120 in FIG. 2, the second device 120 can be synchronized Track another device, e.g., the first device 110, and perform a communication connection with the first device 110. [

However, if the first device 110 does not need to receive content and status information directly from the second device 120 for content synchronization of the second device 120, for example, the first device 110 Does not perform the initial operation for content synchronization as shown in Fig. 2, when the second device 120 can recognize the content and the status information of the content that the second device 120 is executing. That is, the initial operation for content synchronization as shown in FIG. 2 may be selectively performed when the first device 110 and the second device 120 need to communicate with each other.

Fig. 3 is a diagram showing an example of an operation for allowing the second device of Fig. 1 to share contents of the first device. Fig.

The first device 110 receives a synchronization command to allow the second device 120, which is another device, to share the content being executed by the first device 110 (310).

The first device 110 delivers content and status information to the second device 120 (320). At this time, the first device 110 may additionally transmit the augmentation information related to the content to the second device 120. [

For example, when the first device 110 is a mobile terminal and the second device 120 is a TV, if the contents of the mobile terminal need to be synchronized to the TV, the TV receives the terminal status from the mobile terminal, Information for synchronizing the contents of the mobile terminal, such as the contents being provided, the reinforcement information, and the application for driving the contents, to the TV. The second device 120 can display the provided information at a resolution suitable for TV display.

Fig. 4 is a diagram showing an example of an operation for the first device of Fig. 1 to share contents of a second device. Fig.

The first device 110 receives a synchronization command to share the content being executed by the second device 120 (410).

From the second device 120, status information on the content and augmentation information related to the content to share the content of the other device (420). The augmentation information related to the content can be additionally shared.

The first device 110 configures a screen including content and augmentation information by performing content synchronization with the second device 120 using the acquired information (430). The configured screen is displayed through a display (not shown) of the first device 110.

The first device 110 may perform an operation set to the selected information if information configured to enable interaction among the enhancement information is selected according to a user input signal (440).

5 is a diagram showing an example of a screen in which the first device shares and uses the contents of the second device.

In FIG. 5, the device 520 is currently providing broadcast content to the TV. Devices 512, 514 and 516 represent mobile terminals. The mobile terminal 512 shows an example of a screen for synchronizing and displaying the content being executed on the TV 520. The mobile terminal 514 displays another example of a screen for synchronizing and displaying the content being executed on the TV 520 And the mobile terminal 516 shows another example of a screen for synchronously displaying the content that is being executed on the TV 520. [

The mobile terminal 512 receives information for synchronizing the TV to the mobile terminal such as contents currently being broadcast, augmentation information, and status information from the TV 520, and displays the received information at a resolution suitable for the display of the mobile terminal . At this time, the mobile terminal 512 may provide personalized augmented information to the user of the mobile terminal 512 based on the received augmented information, such as a user profile or user preference information. In other words, if the mobile terminal user is a father of forty family members, and if it is a daughter of ten family members, the information taste may be different. Personalized personalized enhanced information may be provided reflecting the personal information taste.

For example, the mobile terminal 512 may provide the location of the broadcast content and OST information, as shown. The photographing place and the OST information can be selected by the user, and can be other information that is set by the selection, or information that can be interacted to perform the function. In this case, when the user of the mobile terminal 512 wants to know more detailed information about the content currently being broadcast on the TV, he can select the shooting location or the OST information and receive detailed information connected to the selected augmentation information.

The user of the mobile terminal 514 may pause the TV content without disturbing other members as shown by the mobile terminal 514 or may broadcast the broadcast content of another channel through channel zapping Or individually operate the broadcast guide for the content currently being broadcasted.

Also, as illustrated by the mobile terminal 516, if enhanced information is interaction information associated with a purchase and payment service, a user of the mobile terminal 516 may use services such as purchase and payment using the augmentation information have.

In this manner, in the mobile terminal synchronized with the TV 520, the user can individually perform operations such as content control, device control, and application control without interfering with the TV viewing by other members watching the TV 520 . As described above, the user of the mobile terminal 512, 514, 516 synchronizes the augmented information and the content state retrieved by the user to the TV 520 again, . ≪ / RTI >

6 is a diagram showing an example of the configuration of the first device of FIG.

The first device 110 includes a control unit 610, a user interface unit 620, a photographing unit 630, a broadcast receiving unit 640, a storage unit 650, a communication unit 660, an output unit 670, 680 < / RTI >

The controller 610 may be configured to control the overall operation of the first device 110 and may include means for decoding audio and video streams. The control unit 610 may perform an operation for performing synchronization for sharing content of another device capable of content synchronization and for allowing content to be executed by the control unit 610 to be shared by another device.

The user interface unit 620 receives the user input signal and transmits the user input signal to the control unit 610. The user interface unit 620 may include various user input means such as a keypad, a touch pad, and a jog shuttle. The touch pad for sensing the touch gesture of the user of the user interface unit 620 is integrated with the display unit 672 of the output unit 670 for displaying the output information so that the user input processing result can be directly provided through the display unit 672 . ≪ / RTI > In addition, the user interface unit 620 may be configured to allow the user to feel tactile, force, and movement feelings through the user input unit when the user input is received using the haptic technique. For example, the user interface unit 620 may generate tactile feedback to convey a feeling that the user directly manipulates the device when the user input signal is received.

The photographing unit 630 may include a camera, such as a lens, an image sensor such as a CCD and a CMOS, and an analog-to-digital converter. The photographing unit 630 obtains a video signal generated by capturing an object, converts the obtained video signal into a digital signal, and transmits the digital signal to the controller 610. [ The control unit 610 may perform an image processing operation for image processing such as noise reduction and color tone processing on the image converted into the digital signal. Here, the control unit 610 may output the captured image through the display unit 672 to provide the user with a continuous image indicating the currently captured state.

The broadcast receiver 640 may receive the broadcast program such as terrestrial or satellite, or receive the broadcast program through a bidirectional channel such as the Internet. In the case of receiving a broadcast program through a broadcast signal, the broadcast receiver 640 includes a tuner unit (not shown) for tuning and demodulating a broadcast signal input to a specific channel selected by the user and outputting a transport stream, And a demultiplexer (not shown) that demultiplexes the multiplexed transport stream into a video stream and an audio stream.

The storage unit 650 stores the operating system, application programs, and data of the first device 110. At least one of user profile information and user preference information including information on the user such as the user age and gender of the first device 110 may be stored in the storage unit 650. User profile information and user preference information can be preset and stored in the storage unit 650 and can be added, changed and deleted by the user input signal. In addition, the storage unit 650 may store contents such as moving pictures, music files, and text of the first device 110, and enhancement information for the contents.

The communication unit 660 may be configured to communicate with various servers such as an external device connected through a network or the augmented reality support server 130, the content providing server 150 and the web server 160 of Fig. The communication unit 660 may include a first communication module configured to communicate with a server connected via the Internet and a second communication module configured to communicate with another device capable of content synchronization. For example, the communication unit 660 may communicate with at least one of various communication protocols such as Bluetooth, Zigbee, Wireless LAN, Home Radio Frequency (UWB), Ultra Wide Band (UWB) As shown in FIG.

The output unit 670 may include a display unit 672 and a speaker 674. The microphone 680 may receive an externally input acoustic signal and may transmit the acoustic signal to the controller 610.

The control unit 610 may track another device capable of synchronization that is registered in advance within a range of distance that can be synchronized and control the communication unit to connect with another device before synchronization is performed.

The control unit 610 acquires content, status information on the content, and augmentation information related to the content from another device, and shares the content of the other device, when receiving a synchronization command for sharing the content being executed by another device, By using the acquired information to perform content synchronization with another device, a screen including content and augmentation information can be configured. The control unit 610 can perform the operation set to the selected information when the information configured to enable the interaction among the enhancement information is selected according to the user input signal.

The control unit 610 selects (or filters) some of the augmentation information among the augmentation information associated with the content based on at least one of the user profile and the user preference information, and provides the selected augmentation information to the display unit 672 together with the content can do.

The enhancement information includes playback control information for controlling the playback of the content, and the control unit 610 can control the playback of the synchronized content in accordance with the user input signal for selecting the playback control information. In addition, the control unit 610 is configured to perform other applications capable of interacting with the user in parallel with content synchronization. Other applications capable of interacting with a user may include at least one of an Internet search application and a social networking application.

The control unit 610 may be configured to acquire the shared content directly via another path, instead of continuously receiving the shared content from another device, in order to share the content of another device. For example, when the content being executed by another device is real-time broadcast content, the control unit 610 acquires meta information of the real-time broadcast content from another device, and the broadcast receiving unit 640 , And can receive and receive real-time broadcast contents through a broadcasting network.

The meta information about the content can be obtained by analyzing a synchronization command for sharing the content being executed by another device and an operation state of the device when the synchronization command is input. Here, the meta information about the content may include at least one of a web address where the content is located and broadcast channel information capable of receiving the content. The meta information about the content may be information of a part of the content (for example, a frame of a partial section of the moving picture, an acoustic signal of a section of the music file), and may have various forms .

In addition, the control unit 610 does not perform communication with another device, but recognizes content and status information executed by another device, acquires the recognized content, and acquires content of another device As shown in FIG.

For example, when a synchronization command for sharing content that is being executed by another device is input while capturing an image through the shooting unit 630, the control unit 610 controls the capture unit 630 to capture the captured image when the synchronization command is input And obtain meta information about the content being executed by another device. The control unit 610 analyzes the sound signal using the sound signal input to the microphone 680 for a predetermined time as content meta information and outputs the music file or the moving picture file including the analyzed sound signal to the storage unit 650 You can search.

When the control unit 610 acquires the meta information about the content, the controller 610 acquires additional augmenting information from the augmented reality support server 130 (130 in Fig. 1) using the obtained meta information, It is possible to construct a screen by rendering additional augmentation information on the content being displayed and display the configured screen through the display unit 672. [

In addition, the control unit 610 receives a synchronization command for allowing a content to be executed by the control unit 610 to be shared by another device, and transmits the content, status information, and content that are being executed by the control unit 610 to another device The communication unit 660 can be controlled to transmit the enhancement information. In this case, when the content of another device is synchronized and the shared content is transmitted to the other device and is shared, the controller 610 can transmit only the reinforcement information currently provided to another device.

7 is a diagram showing an example of the configuration of the control unit of Fig.

The control unit 610 may include a synchronization performing unit 710, an augmenting information processing unit 720, a playback control unit 730, and a parallel application performing unit 740.

The synchronization performing unit 710 may transmit synchronization information to another device to share and synchronize contents of another device.

Upon receiving a synchronization command for sharing content that is being executed by another device, the synchronization performing unit 710 recognizes the content executed in the other device and the status information indicating the execution status of the content to acquire the content and status information , And can perform content synchronization with another device using the acquired content and status information. The output unit 670 of FIG. 6 can output the synchronized contents.

When the photographing unit 630 of FIG. 6 photographs a content execution screen that another device is executing, the synchronization performing unit 710 performs at least one of object extraction, pattern recognition, and character recognition on the content execution screen, It is possible to recognize the content executed in another device and the status information indicating the execution status of the content.

When the audio input unit such as the microphone 680 receives the audio signal for the content being played by the device, the synchronization performing unit 710 extracts the feature information from the input audio signal, and recognizes the audio data corresponding to the feature information can do.

The storage unit 650 may store at least one content. In this case, the synchronization performing unit 710 can retrieve and acquire the recognized content from the storage unit.

The storage unit 650 may store the content so that the content corresponding to the feature information generated by the object extraction, pattern recognition, or character recognition result of the synchronization performing unit 710 is searched. In addition, the storage unit 650 may be configured to retrieve audio data corresponding to the feature information extracted from the audio signal. Here, the feature information on the audio is an audio finger print, which means an audio parameter or a model that most effectively summarizes the audio object. In order to identify each audio data or audio file, And may be a parameter indicating characteristics inherent to the audio signal extracted from the audio signal.

When the communication unit 660 of FIG. 6 communicates with a content providing server (150 of FIG. 1) that retrieves and provides the content using the content metadata, the synchronization performing unit 710 receives, from the content providing server, The user can obtain the retrieved contents through the Internet.

Next, an operation in which the synchronization executing unit 710 receives the synchronization command for sharing the web application as the content being executed by another device and shares the execution state of the web application that the other device is executing will be described. Here, the user of the web application in which the other device is running and the user of the first device 110 may be the same user.

The web application can provide information through a communication with a web server (160 in FIG. 1) that provides the web service through the communication unit 660. For example, a web application can have many different types, such as a messenger, a Twitter, a blog, and a web browser that provides a particular web site. Upon receiving a synchronization command for sharing a web application as content that another device is executing, the photographing unit 630 can acquire a web application execution screen of another device.

The synchronization performing unit 710 may perform at least one of object extraction, pattern recognition, and character recognition on the web application execution screen of another device to identify the web application. In addition, the synchronization performing unit 710 may acquire user information used to receive the web service of the web server 160. [ The user information may include user login information such as a user ID, a password, and other authentication information used to receive a web service from the web server.

The synchronization performing unit 710 accesses the web server 160 using the user information and receives web application status information indicating the execution status of the web application in which another device is running from the web server 160 through the communication unit 660 Can be obtained. The synchronization performing unit 710 can reproduce the execution state of the web application in which the other device is executing, using the identified web application and the acquired web application status information.

When the storage unit 650 stores user information used for receiving a web service from a web server using at least one web application, the synchronization performing unit 710 acquires user information from the storage unit 650 .

When the communication unit 660 communicates with another device and acquires user information from another device, the synchronization performing unit 710 can acquire user information from another device. The synchronization executing unit 710 accesses the web server 160 using the user information obtained from another device and displays the execution status of the web application that the other device is running from the web server 160 through the communication unit 660 The web application status information can be obtained and the execution status of the web application being executed by another device can be reproduced using the acquired web application status information.

For example, if the web application is a messenger program that provides one-to-one conversation service, the synchronization performing unit 710 can identify the messenger program from the content execution screen of another device and execute the identified messenger program. At this time, the identified messenger program executed by the synchronization performing unit 710 may be a program that can be supported by the web server communicating with the web application of another device, and may not be the same program as the messenger program executed by the other device. For example, the messenger program executed by the synchronization performing unit 710 may be a homogeneous program optimized to be executed by the synchronization performing unit 710. [ The synchronization performing unit 710 accesses the web server 160 using the user information through the executed messenger program and acquires web application status information that is being executed on another device from the web server 160, The user can receive the information about the other party user who is talking with the other party, and continue conversation with the other party user who has been communicating through the other device.

In addition, the synchronization performing unit 710 acquires content, status information, and augmentation information related to the content from another device, and shares the content of another device, when receiving a synchronization command for sharing content that is being executed by another device , And by using the acquired information to perform content synchronization with another device, a screen including content and augmentation information can be configured. In addition, the synchronization executing unit 710 receives a synchronization command for allowing the other device to share the content being executed by the control unit 610, and transmits the content, status information, and content, which are being executed by the control unit 610, And may control the communication unit 660 to transmit the related enhancement information.

The enhancement information processing unit 720 may filter some enhancement information among the enhancement information associated with the content based on at least one of the user profile and the user preference information and provide the filtering result to the synchronization performing unit 710. [ Accordingly, the synchronization performing unit 710 can construct the screen by rendering the filtered augmented information with the content.

The playback control unit 730 may be configured to control playback of the shared content without changing the state of the other device. For example, when the other device is a TV, the playback control unit 730 can control the playback of the shared content to operate the TV content without disturbing other users watching the TV.

The parallel application performing unit 740 is configured to perform another application capable of interacting with a user in parallel with content synchronization. For example, the parallel application execution unit 740 may perform an Internet search related to TV contents without interfering with a user of the TV, which is another device that the user of the first device 110 shares with the TV content, / Social networking, such as messenger, to communicate with the delay watching the same broadcast, such as enabling the connected applications can be enabled on the TV. By individually manipulating the information and manipulating the apparatus, not only can the personal interaction space be provided without disturbing the other members watching the TV contents, but also the functions more suitable for the near-far interaction than the remote interaction such as browsing and manipulation It is possible to provide a more intuitive proximity interaction to the user.

Although the configuration of the first device 110 has been described above, the second device 120 may also be configured to perform the functions of the first device 110. [ For example, if the second device is a TV, the second device 120 provides content, augmentation information, and status information to the first device 110 to synchronize with the first device 110, The mobile contents provided from the mobile terminal 120 can be synchronized and displayed on the TV screen.

Figure 8 is a flow diagram illustrating an example of a method by which another device may synchronize a running web application.

The first device receives a synchronization command to share the web application running on the second device (810).

Upon receipt of the synchronization command for sharing the web application as the content being executed by the second device, the first device acquires the web application execution screen of the second device (820).

The first device performs at least one of object extraction, pattern recognition, and character recognition on the acquired web application execution screen of another device to identify the web application (830). The first device executes the identified web application (840).

The first device obtains the user information used to receive the web service (850). The user information may be stored in the first device and may be obtained from another device.

The first device accesses the web server using the executed web application and the user information, and acquires web application status information indicating the execution status of the web application that the other device is running from the web server (860).

Using the identified web application and the acquired web application status information, another device reproduces the execution status of the running web application (870).

One aspect of the present invention may be embodied as computer readable code on a computer readable recording medium. The code and code segments implementing the above program can be easily deduced by a computer programmer in the field. A computer-readable recording medium includes all kinds of recording apparatuses in which data that can be read by a computer system is stored. Examples of the computer-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical disk, and the like. The computer-readable recording medium may also be distributed over a networked computer system and stored and executed in computer readable code in a distributed manner.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention. Therefore, the scope of the present invention should not be limited to the above-described embodiments, but should be construed to include various embodiments within the scope of the claims.

Claims (20)

An output section;
A photographing unit; And
The image processing apparatus comprising: an image acquisition unit configured to acquire,
Acquiring the identified content and content related information,
Acquiring personalized content related information based on the content related information and user profile information,
And a control unit for outputting the personalized content related information and the screen including the content through the output unit,
The content-
And information indicating an execution state of the content being executed in the external device. The method according to claim 1,
And an audio input unit,
Wherein,
And identifies the content based on the audio signal received via the audio input and the captured image. The method according to claim 1,
Wherein,
And obtains the identified content and content related information from the external device. The method of claim 3,
Wherein,
And configures and outputs a screen running on the external device based on the information indicating the execution state of the content and the content. A method of controlling a device,
Identifying content that is being executed on an external device based on the captured image;
Obtaining the identified content and content related information;
Obtaining personalized content related information based on the content related information and user profile information; And
And outputting the personalized content related information and the screen including the content,
The content-
And information indicating an execution state of the content being executed by the external device. 6. The method of claim 5,
Further comprising receiving an audio signal,
Wherein identifying the content comprises:
And identifying the content based on the received audio signal and the photographed image. 6. The method of claim 5,
The step of acquiring the content and the content-
And obtains the identified content and content-related information from the external device. 8. The method of claim 7,
The step of outputting the screen may include:
And a control unit configured to configure and output a screen running on the external device based on the information indicating the execution state of the content and the content. delete delete delete delete delete delete delete delete delete delete delete delete

Images