KR20140060682A - Multimedia device connected to at least one network interface and method for processing data in multimedia device - Google Patents

Abstract

A multimedia device and a control method thereof are disclosed in the present specification. According to an embodiment of the present invention, the multimedia device connected to at least one network interfaces comprises a first network interface module for receiving a broadcasting signal in which audio data and video data are multiplexed; a video processor for processing video data within the multiplexed broadcasting signal; an audio processor for processing the audio data within the multiplexed broadcasting signal; a display module for outputting the processed video data; a speaker for outputting the processed audio data; a detecting module for detecting the predetermined-specific event; a list generation module for generating a list including at least one stopped image data in which the predetermined-specific data is connected to the video data of the detected point; a controller for storing the integrated data in which the specific-stopped image data and random information are mapped within the list in a memory; and a second network interface module for transmitting the integrated data stored in the memory to an external device.

Description

TECHNICAL FIELD [0001] The present invention relates to a multimedia device connected via at least one network interface and a control method thereof. [0002] MULTIMEDIA DEVICE CONNECTED TO AT LEAST ONE NETWORK INTERFACE AND METHOD FOR PROCESSING DATA IN MULTIMEDIA DEVICE [

The present invention relates to a multimedia device (for example, a TV, a DTV, an IPTV, a network TV, a smart TV, etc.), and more particularly to a specific solution of a multimedia device and a control method thereof connected to at least one network interface .

Recently, with the development of multimedia devices, a variety of multimedia devices connected to various networks have emerged. In addition, existing multimedia devices have only the function of processing broadcast data received from a broadcasting station, but currently there is a growing discussion about a technique for processing data received from a plurality of sources together in real time.

An embodiment of the present invention is to provide an interface for mapping video data received from an external source and arbitrarily input information and storing them in a memory in a short time.

According to another aspect of the present invention, there is provided a solution for mapping desired information to video data through various methods in the process of generating one integrated data.

Further, another object of the present invention is to provide a GUI (Graphic User Interface) technique that is more convenient and quick for a user in editing the one integrated data.

According to another aspect of the present invention, there is provided a method for delivering newly created integrated data to a specific device or group in real time.

A multimedia device connected to at least one network interface according to an embodiment of the present invention includes a first network interface module for receiving a broadcast signal in which audio data and video data are multiplexed and a second network interface module for receiving video data in the multiplexed broadcast signal A speaker for outputting the processed audio data; a display module for outputting the processed video data; a speaker for outputting the processed audio data; A list generating module for generating a list including at least one still image data associated with the video data at a time point at which the preset specific event is detected; The image data and any state data integration of the mapping information, and the controller for storage in the memory, and a second network interface module for transmitting the integrated data stored in the memory, to the external device.

A method of controlling a multimedia device connected to at least one network interface according to an exemplary embodiment of the present invention includes displaying video data received from an external device through a first network interface, Outputting a list including at least one still image data associated with the moving image data; and storing integrated data in a state in which specific still image data in the list is mapped with arbitrary information, in a memory And transmitting the integrated data stored in the memory to an external device via a second network interface.

According to an embodiment of the present invention, there is provided an interface for mapping video data received from an external source and arbitrarily inputted information and storing them in a memory in a short time.

According to another embodiment of the present invention, there is provided a solution for mapping desired information to video data through various methods in the process of generating one integrated data.

Further, according to another embodiment of the present invention, a graphical user interface (GUI) technique is provided to the user in a process of editing the integrated data.

According to another embodiment of the present invention, there is provided a method for delivering newly created integrated data to a specific device or group in real time.

1 is a detailed block diagram of components of a multimedia device according to an exemplary embodiment of the present invention.
FIG. 2 is a block diagram showing the control unit shown in FIG. 1 in more detail.
3 is a view schematically showing an appearance of a remote controller according to an embodiment of the present invention.
4 is a detailed block diagram of the components of the remote controller shown in FIG.
FIG. 5 is a detailed block diagram of components of a multimedia device according to another embodiment of the present invention. Referring to FIG.
6 is a block diagram showing the detec- tion module shown in Fig. 5 in more detail.
7 is a diagram illustrating a process of detecting a voice signal derived from a user by a multimedia device according to another embodiment of the present invention.
8 is a diagram illustrating a process of detecting a specific gesture of a user by the multimedia device according to another embodiment of the present invention.
9 is a diagram illustrating a process in which a multimedia device according to another embodiment of the present invention detects a specific command transmitted from a remote control.
FIG. 10 shows a screen output by the multimedia device according to another embodiment of the present invention when a specific command is detected according to the embodiment shown in FIG. 7 to FIG.
11 is a view for explaining a list of captured images provided by a multimedia device according to another embodiment of the present invention.
12 is a view for explaining a process of dividing a memo area provided by a multimedia device into at least two or more memo areas according to another embodiment of the present invention.
13 is a process diagram for explaining an example in which a multimedia device according to another embodiment of the present invention inputs text data for mapping with a captured image.
14 is a process diagram for explaining another example in which a multimedia device according to another embodiment of the present invention inputs text data for mapping with a captured image.
15 is a process diagram for explaining another example in which a multimedia device according to another embodiment of the present invention inputs text data for mapping with a captured image.
16 is a diagram for explaining a process of editing a captured image by a multimedia device according to another embodiment of the present invention.
17 is a view for explaining an example of inputting a voice signal for mapping a captured image with a multimedia device according to another embodiment of the present invention.
18 is a diagram for explaining a process of storing the integrated data including the captured image in a random memory according to another embodiment of the present invention.
FIG. 19 is a diagram for explaining a process in which a multimedia device according to another embodiment of the present invention transmits integrated data including a captured image to an arbitrary device or group. FIG.
20 is a flowchart for explaining an embodiment for controlling a multimedia device.
FIG. 21 is a flowchart detailing step S2030 shown in FIG.
22 is a flowchart for explaining another embodiment for controlling the multimedia device.

Hereinafter, the present invention will be described in more detail with reference to the drawings.

The suffix "module" and " part "for components used in the following description are given merely for ease of description, and the" module "and" part "

Meanwhile, the multimedia device described in this specification is an intelligent electronic device that adds a computer support function to a broadcast reception function, for example, and is equipped with an Internet function while being faithful to a broadcast reception function, A pad or a space remote controller. With the support of wired or wireless internet function, it can be connected to internet and computer and can perform functions such as e-mail, web browsing, banking or game. A standardized general-purpose OS can be used for these various functions.

Accordingly, the multimedia device described in the present invention can perform various user-friendly functions because various applications can be freely added or deleted on a general-purpose OS kernel, for example. More specifically, the multimedia device may be, for example, an Internet TV, an IPTV, a network TV, an HBBTV, a smart TV, and an open hybrid TV (OHTV), and may be applied to a smart phone, a PC, Do.

BRIEF DESCRIPTION OF THE DRAWINGS The above and other features and advantages of the present invention will be more apparent from the following detailed description taken in conjunction with the accompanying drawings, in which: FIG.

As used herein, terms used in the present invention are selected from general terms that are widely used in the present invention while taking into account the functions of the present invention, but these may vary depending on the intention or custom of a person skilled in the art or the emergence of new technologies. In addition, in certain cases, there may be a term arbitrarily selected by the applicant, in which case the meaning thereof will be described in the description of the corresponding invention. Therefore, it is intended that the terminology used herein should be interpreted based on the meaning of the term rather than on the name of the term, and on the entire contents of the specification.

1 is a detailed block diagram of components of a multimedia device according to an exemplary embodiment of the present invention.

1, a multimedia device 100 according to an exemplary embodiment of the present invention includes a broadcast receiving module 101, a demodulator 102, a network interface 103, an external device interface 104, A control unit 106, a video output unit 107, an audio output unit 108, a power supply unit 109, a user interface 111, and the like. Meanwhile, the multimedia device 100 is designed to perform data communication with the remote controller 110, and the remote controller 110 will be described later in detail with reference to FIG. 3 and FIG.

The broadcast receiving module 101 can be designed, for example, as an RF tuner or an interface for receiving broadcast data from an external device such as a STB.

The broadcast receiving module 101 may receive an RF broadcast signal of a single carrier according to an Advanced Television System Committee (ATSC) scheme or an RF broadcast signal of a plurality of carriers according to a DVB (Digital Video Broadcasting) scheme.

The demodulator 102 receives the digital IF signal DIF converted by the broadcast receiving module 101 and performs a demodulation operation.

For example, when the digital IF signal output from the broadcast receiving module 101 is of the ATSC scheme, the demodulator 102 performs 8-VSB (8-Vestigal Side Band) demodulation, for example. Also, the demodulator 102 may perform channel decoding.

The external device interface 104 is an interface that enables data communication between the external device and the multimedia device 100. The external device interface 104 can be connected to an external device such as a DVD (Digital Versatile Disk), a Blu ray, a game device, a camera, a camcorder, a computer (notebook) The external device interface 104 transmits external video, audio, or data signals to the control unit 106 through a connected external device. Also, the control unit 206 can output the video, audio, or data signal processed by the control unit 206 to an external device.

The external device interface 104 may include a USB terminal, a CVBS (Composite Video Banking Sync) terminal, a component terminal, an S-video terminal (analog), a DVI (Digital Visual Interface) terminal, Terminal, an RGB terminal, a D-SUB terminal, and the like.

The network interface 103 provides an interface for connecting the multimedia device 100 to a wired / wireless network including the Internet network. The network interface 103 may include an Ethernet terminal or the like for connection to a wired network and may be a WLAN (Wireless LAN) (Wi-Fi ), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), and HSDPA (High Speed Downlink Packet Access) communication standards.

The network interface 103 can transmit or receive data to another user or other electronic device via the network to which it is connected or another network linked to the connected network.

The memory 105 may store a program for each signal processing and control in the control unit 106, and may store signal-processed video, audio, or data signals. The memory 105 may also function for temporary storage of video, audio, or data signals input from the external device interface 104 or the network interface 103. [ The memory 105 also stores various OSs, middleware, and platforms, for example.

The user interface 111 transmits a signal input by the user to the control unit 106 or transmits a signal from the control unit 106 to an external device (for example, the remote controller 110). For example, the user interface 111 may be provided with various functions such as power on / off, channel selection, screen setting, and the like from the remote controller 110 according to various communication methods such as a radio frequency (RF) communication method and an infrared And is designed to receive and process a control signal or transmit a control signal from the control unit 106 to the remote controller 110. [

The control unit 106 demultiplexes the input stream or processes the demultiplexed signals through the broadcast receiving module 101, the demodulating unit 102, the network interface 103, or the external device interface 104 , And can generate and output signals for video or audio output. The control unit 106 will be described later in detail with reference to FIG.

The video output unit 107 converts a video signal, a data signal, an OSD signal processed by the control unit 106, a video signal and a data signal received from the external device interface 104 into R, G, and B signals, respectively Thereby generating a driving signal. The video output unit 107 may be a PDP, an LCD, an OLED, a flexible display, a 3D display, or the like.

The audio output unit 108 receives a signal processed by the control unit 106, for example, a stereo signal, a 3.1 channel signal, or a 5.1 channel signal, and outputs the audio signal. The audio output unit 108 may be implemented by various types of speakers.

The power supply unit 109 supplies corresponding power to the multimedia device 100. In particular, a control unit 106 that can be implemented in the form of a system-on-chip (SOC), a video output unit 107 for video display, and an audio output unit 108 for audio output .

Particularly, the following description will be given in connection with the embodiment of the present invention.

Under the control of the control unit 106, the network interface 103 connects to a specific server. Further, the IR code values received from the specific server are designed to be stored in the memory 105. On the other hand, the IR code values are values necessary for controlling the electronic device connected to the external device interface 104.

Lastly, the IR code value corresponding to the specific function of the specific electronic device is transmitted to the remote controller 110 through the user interface 111 at the request of the remote controller 110. As described above, the Bluetooth communication method can be employed at this time.

FIG. 2 is a block diagram showing the control unit shown in FIG. 1 in more detail.

2, the control unit 206 of the multimedia device includes a demultiplexing unit 210, an image processing unit 220, an OSD generating unit 240, a mixer 250, a frame rate converting unit 255, A formatter 260, and the like. It is also within the scope of the present invention to further design a voice processing unit (not shown) and a data processing unit (not shown).

The demultiplexer 210 demultiplexes the input stream. For example, when an MPEG-2 TS is input, it can be demultiplexed into video, audio, and data signals, respectively.

The image processing unit 220 may perform image processing of the demultiplexed image signal. To this end, the image processing unit 220 may include a video decoder 225 and a scaler 235. The video decoder 225 decodes the demultiplexed video signal, and the scaler 235 performs scaling so that the resolution of the decoded video signal can be output from the video output unit. The video signal decoded by the video processing unit 220 is input to the mixer 250.

The OSD generation unit 240 generates an OSD signal according to a user input or by itself. Accordingly, the mixer 250 can mix the OSD signal generated by the OSD generator 240 and the decoded video signal processed by the image processor 220. [

The mixed signal is provided to formatter 260. Since the decoded broadcast video signal or the external input signal and the OSD signal are mixed, the OSD can be overlaid on the broadcast image or the external input image.

A frame rate converter (FRC) 255 can convert a frame rate of an input image. For example, a frame rate of 60 Hz is converted to 120 Hz or 240 Hz.

The formatter 260 receives the output signal of the frame rate conversion unit 255, changes the format of the signal so as to be suitable for the video output unit, and outputs the signal. For example, the R, G, and B data signals may be output as low voltage differential signaling (LVDS) signals or mini-LVDS signals.

An OSD for selecting an electronic device and a function to be controlled by a remote controller according to an embodiment of the present invention is generated by the OSD generation unit 240 described above. Further, the mixer 250 described above is additionally designed to make the general broadcast screen and the OSD overlaid.

3 is a view schematically showing an appearance of a remote controller according to an embodiment of the present invention.

A pointer 301 corresponding to the movement of the remote controller 310 is displayed on the screen of the multimedia device 300 as shown in FIG. The user can move or rotate the remote controller 310 left and right (FIG. 3 (b)) or up and down (FIG. 3 (c)). In such a remote controller 310, since the pointer 301 is moved and displayed according to the movement in the 3D space, it may be called a space remote controller.

3 (b), when the user moves the remote controller 310 to the left, the pointer 301 displayed on the screen of the multimedia device 300 also moves to the left. Information on the movement of the remote controller 310 sensed through the sensor of the remote controller 310 is transmitted to the multimedia device 300. The multimedia device 300 can calculate the coordinates of the pointer 301 from the information on the motion of the remote controller 310. [ The multimedia device 300 may display the pointer 301 so as to correspond to the calculated coordinates.

3 (c), when the user moves the remote controller 310 downward, the pointer 301 displayed on the screen of the multimedia device 300 also moves downward.

Therefore, it is possible to quickly select a specific area on the screen of the multimedia device 300 using the remote controller 310 according to an embodiment of the present invention.

4 is a detailed block diagram of the components of the remote controller shown in FIG.

4, the remote controller 410 includes a wireless communication unit 414, a user input unit 415, a sensor unit 417, an output unit 416, a power supply unit 411, a storage unit 412, And a control unit 413, and the like.

The wireless communication unit 414 is designed to be capable of communicating with any external device. In particular, according to an embodiment of the present invention, the RF module 414a is designed to perform data communication with the multimedia device 400, and the IR module 414b is configured to communicate with an external electronic device (e.g., STB) .

Accordingly, it is possible to implement the remote controller 410 to perform the role of a repeater for forwarding the IR infrared code value received from the multimedia device 400 to the STB.

In addition, according to one embodiment of the present invention, the remote controller 410 transmits a signal containing information on the motion of the remote controller 410 to the multimedia device 400 through the RF module 414a.

In addition, the remote controller 410 can receive the signal transmitted by the multimedia device 400 through the RF module 414a. In addition, the remote controller 410 can transmit commands to the multimedia device 400 via the IR module 414b, such as power on / off, channel change, volume change, and the like, if necessary.

The user input unit 415 may include a keypad, a button, a touch pad, or a touch screen.

The sensor unit 417 may include a gyro sensor 417a or an acceleration sensor 417b. The gyro sensor 417a can sense information about the motion of the remote controller 410. [ For example, the gyro sensor 417a can sense information about the operation of the remote controller 410 on the basis of the x, y, and z axes. The acceleration sensor 417b can sense information on the moving speed of the remote controller 410 and the like. Meanwhile, a distance measuring sensor may be additionally provided, so that the distance to the multimedia device 400 can be sensed.

The output unit 416 may output an image or voice signal corresponding to an operation of the user input unit 415 or a signal transmitted from the multimedia device 400. [ For example, the output unit 416 includes an LED module 416a that is turned on when a user input unit 415 is operated or a signal is transmitted to or received from the multimedia device 400 through the wireless communication unit 414, a vibration module 416b An audio output module 416c for outputting sound, or a display module 416d for outputting an image.

The power supply unit 411 supplies power to each component of the remote controller 410. The power supply unit 411 can reduce the power waste by interrupting the power supply when the remote controller 410 has not moved for a predetermined time.

The storage unit 412 may store various types of programs, application data, and the like necessary for the control or operation of the remote controller 410. The control unit 413 controls various matters related to the control of the remote controller 410. For example, the control unit 413 transmits a signal corresponding to a predetermined key operation of the user input unit 415 or a signal corresponding to the motion of the remote controller 410 sensed by the sensor unit 417 to the radio communication unit 414, To the multimedia device 400 or to the STB.

FIG. 5 is a detailed block diagram of components of a multimedia device according to another embodiment of the present invention. Referring to FIG. It is also within the scope of the present invention to delete or add some components of the multimedia device shown in Fig. 5, with reference to the embodiments described in Figs. 1 to 4 above.

5, a multimedia device according to another embodiment of the present invention includes a first network interface module 510, a display module 520, a video processor 530, an audio processor 540, A speaker 550, a controller 560, a second network interface module 570, a detection module 580, a list generation module 590, a memory 500, and the like. Further, the controller 560 may be designed to additionally include an OCR (Optical Character Recognition) module 561.

The first network interface module 510 of the multimedia device connected to at least one network interface shown in FIG. 1 receives a broadcast signal in which audio data and video data are multiplexed.

The video processor 530 processes the video data in the multiplexed broadcast signal, and the audio processor 540 processes the audio data in the multiplexed broadcast signal.

The display module 520 outputs the processed video data, and the speaker 550 outputs the processed audio data.

The detec- ting module 580 is designed to detect a preset specific event, and a concrete implementation method and an embodiment of the detec- tion module 580 will be described later in detail with reference to Figs. 6 to 9 would.

The list generation module 590 generates a list including at least one still image data associated with the video data at the time when the preset specific event is detected. A method of implementing the list will be described later in detail with reference to FIG.

The controller 560 stores the integrated data in a state in which specific still image data in the list is mapped to arbitrary information in the memory 500. The second network interface module 570 is designed to transmit the integrated data stored in the memory 500 to an external device. The operation process of the memory 500 will be described later in more detail with reference to FIG. 18, and the operation process of the second network interface module 570 will be described later in more detail with reference to FIG.

In addition, according to another embodiment of the present invention, when the preset specific event is detected, the controller 560 controls the speaker 550 to maintain the output of the processed audio data, Is designed to control the display module 520 to stop outputting the processed video data, and to store the video data in the memory 500. By designing in this way, there is an advantage that the user can continuously and continuously check the broadcast data of the channel that the user has watched or the audio data of the data from the external source.

According to another embodiment of the present invention, when the preset specific event is detected, the controller 560 divides the screen of the multimedia device into at least two areas, The indicator is designed to be converted into a first graphic image, and when the indicator is located in a second area including an editing tool, the indicator is designed to be converted into a second graphic image. With reference to the above-described embodiment, it will be described later in detail with reference to FIG.

The OCR module 561 is designed to automatically extract the text included in the specific still image data. Therefore, there is an advantage that the user does not need to directly input the text data for mapping with the captured image.

In addition, the multimedia device according to another embodiment of the present invention further includes a user interface for inputting arbitrary information for mapping with the specific still image data. The arbitrary information includes at least one of text, audio data, and graphic images.

The at least one still image data associated with the video data corresponds to at least one or more images captured retroactively from a point in time when the specific event is detected, retrospectively. Hereinafter, this will be described below with reference to FIG.

Furthermore, the controller 560 is designed to control the display module 520 to display at least one or more options for specifying an external device. In this regard, it will be described later with reference to FIG.

The preset specific event corresponds to at least one of a gesture of a user using the multimedia device, a voice, or a specific command generated by a remote controller, for example. In this regard, it will be described later with reference to FIG. 7 to FIG.

6 is a block diagram showing the detec- tion module shown in Fig. 5 in more detail. Hereinafter, with reference to FIG. 6, a command detection method for initially executing a process of capturing a specific screen, a process of mapping and storing arbitrary data, and the like will be described.

6 may correspond to the detec- tion module 580 shown in Fig. 5, and the controller 660 shown in Fig. 6 may correspond to the controller 560 shown in Fig. 5 It may respond.

For example, when an arbitrary voice signal is detected through a microphone 683 receiving a voice signal of a user, the voice recognition module 684 determines that the detected voice signal corresponds to a predetermined specific voice signal Or not. If it corresponds to a predetermined specific voice signal (for example, "capture "," capture ", "stop "," stop ", etc.), it is designed to capture the current screen under the control of the controller 660. Hereinafter, a supplementary explanation will be given with reference to FIG.

On the other hand, when an arbitrary hand motion is photographed through a camera that captures an image including a user's gesture, the gesture recognition module 682 determines whether the photographed hand motion corresponds to a predetermined gesture . If the predetermined gesture corresponds to the preset gesture, the controller 660 controls to capture the current screen. With reference to FIG. 8, a supplementary explanation will be given below.

On the other hand, the remote controller command receiving module 685 for receiving an IR signal or the like from the above-described remote controller in FIGS. 3 to 4 determines whether or not a specific command has been received. If it is determined that the command is a predetermined command, the controller 660 controls the screen to capture the current screen. With reference to FIG. 9, a supplementary explanation will be given below.

7 is a diagram illustrating a process of detecting a voice signal derived from a user by a multimedia device according to another embodiment of the present invention. Hereinafter, a process of capturing a current screen according to a specific voice of a user will be described with reference to FIG.

As described above, the multimedia device 700 according to another embodiment of the present invention includes a speech recognition engine. 7, when the user 730 located near the multimedia device 700 has uttered a specific voice 731, the corresponding voice 730 is transmitted through the microphone 720 mounted on the multimedia device 700, A signal related to a particular voice 731 is received.

In addition, when the received specific voice 731 corresponds to a predetermined command, the multimedia device 700 is designed to automatically capture a current screen (video data from a broadcast screen or other external source) and the like.

On the other hand, any type of algorithm can be used as the speech recognition engine, and the specific algorithm does not limit the scope of the present invention.

For example, a dynamic time warping (DTW) algorithm is a method of determining similarity by comparing a representative pattern with a given input pattern. In other words, even if the same word is uttered, it has a different duration depending on the speaker, emotion, and surrounding environment. Thus, nonlinear optimization of the discrepancy of the continuous length is performed based on the partial optimization, . The dynamic fit method algorithm has an advantage that the task contents of the speech recognition system can be easily changed since the recognition target vocabulary is mainly used for small isolated word recognition and can easily generate the reference pattern.

On the other hand, the Hidden Markov Model (HMM) algorithm is one of the most widely used methods for speech recognition due to its high recognition rate and convenient learning performance. It is composed of transition probability modeling the temporal change of speech and output probability modeling the spectrum change. Rather than using the similarity between the input pattern and the output pattern, the similarity of the model is calculated by using a probabilistic estimate with a given model. Therefore, it is advantageous to analyze the vocabulary and voice of a relatively long sentence.

8 is a diagram illustrating a process of detecting a specific gesture of a user by the multimedia device according to another embodiment of the present invention. Hereinafter, a process of capturing a current screen according to a specific gesture of a user will be described with reference to FIG.

As described above, the multimedia device 800 according to another embodiment of the present invention includes a gesture recognition / analysis module. 8, when the user 830 located near the multimedia device 800 takes a specific gesture 831, the camera 820 mounted on the multimedia device 800 may receive the specific gesture 831, A gesture 831 related signal is received.

Further, when the received specific gesture 831 corresponds to a predetermined command, the multimedia device 800 is designed to automatically capture a current screen (video data from a broadcast screen or other external source) and the like.

On the other hand, any type of algorithm can be used as the camera and gesture recognition / analysis module described above, and the specific algorithm does not limit the scope of the present invention.

There are two types of gesture recognition methods. One is camera image based 2D recognition and the other is 3D based gesture recognition method.

The former camera image-based gesture recognition includes a process of acquiring a user's image with a camera, detecting only a meaningful gesture, and analyzing and recognizing the meaning of the gesture while continuously tracking the gesture.

On the other hand, the latter 3D-based recognition method can analyze a user's gesture more stereoscopically by adding an element to the former method, that is, using depth (depth) information.

Therefore, when the user's hands captured by using a 2D camera or a 3D camera have a rectangle shape of a square shape, they are designed to be recognized as an instruction to execute the capture function according to the present invention.

9 is a diagram illustrating a process in which a multimedia device according to another embodiment of the present invention detects a specific command transmitted from a remote control. Hereinafter, a process of capturing a current screen according to a specific signal received from the remote controller will be described with reference to FIG.

As described above, a remote controller for controlling the multimedia device 900 according to another embodiment of the present invention may be used. 9, the user 930 selects a specific hotkey 932 by using a remote controller 931 located within a distance capable of data communication with the multimedia device 900. As shown in FIG. At this time, the specific IR signal is received through the IR signal receiving unit 920 mounted in the multimedia device 900.

Further, when the received specific IR signal corresponds to a predetermined command, the multimedia device 900 is designed to automatically capture a current screen (video data from a broadcast screen or other external source) and the like.

9, it is assumed that the user 930 selects a specific hotkey 932 of the remote controller 931. However, the scope of the present invention is not limited by the above description.

For example, as described above with reference to FIG. 4, since the remote controller 931 is equipped with the position sensor, it is possible to recognize the case where the remote controller 931 takes a specific motion. Or if the touch of the user is detected through the touch sensor mounted on the remote controller 931, designing to transmit the same IR signal as that when the specific hot key 932 is selected is also within the scope of the present invention I will say.

FIG. 10 shows a screen output by the multimedia device according to another embodiment of the present invention when a specific command is detected according to the embodiment shown in FIG. 7 to FIG.

According to the embodiment shown in Figs. 7 to 9 described above, the multimedia device receiving the instruction to capture the current screen from the user provides the editable screen shown in Fig. Each of the items shown in FIG. 10 may be interpreted as an independent scope of rights, and will be described in detail below.

For example, when video data transmitted from a broadcasting station of a specific channel is output or video data transmitted from another external source (ex: BD player, STB, home theater, etc.) is output, And displays the image in one area (b).

Also, a tool box containing various items 1010, 1020, 2030, 1040, 1050, 1060, 1070 for editing the captured image is displayed in another area (a) of the screen. Further, in order to solve the problem that the captured image is partially overlapped due to the above-described toolbox, the hide / unfold option 1080 is displayed together.

On the other hand, one of the features of the present invention is that the graphic image of the indicator is designed to be different in one area (b) where the captured image is displayed and in another area (a) in which the tool box is displayed. For example, as shown in Fig. 10, an indicator 1092 in one area where the captured image is displayed has a first graphic image, and an indicator 1091 in another area in which the tool box is displayed So as to have a second graphic image. Therefore, there is a technical effect that it is possible to provide the user with a form only in the area b in FIG. 10, and to quickly provide information that the items can be selected only in the area a in FIG.

Furthermore, as shown at the bottom of the screen of Fig. 10, a watermark 1090 indicating the source of the captured image is designed to be automatically generated. Of course, it is also within the scope of the present invention to design the watermark 1090 stored in the memory to be arbitrarily editable by the user.

Finally, the function of each item arranged in the toolbox area (a) shown in Fig. 10 will be described in more detail.

It is one of the advantages of the technique to design the user to input any data for mapping with the captured image into various methods after the current screen is captured according to an embodiment of the present invention.

For example, when the first item 1020 is selected, the speech recognition engine is automatically executed and the voice signal that the user utters is mapped to the currently captured screen image and stored in memory. In this regard, it will be described later in detail with reference to FIG.

When the second item 1030 is selected, a graphic image may be added in an overlaid form on the currently captured screen image, or an edit may be performed to crop only a portion of the entire screen image. On the other hand, when the third item 1040 is selected, the function of deleting the graphic image added by the second item is performed again.

When the fourth item 1050 is selected, the integrated data (or memo data) in the form of a captured image and arbitrary text data or the like is mapped to another external device. In this regard, it will be described later in detail with reference to FIG.

When the fifth item 1060 is selected, the integrated data (or memo data) in the form of a captured image and arbitrary text data is stored in a memory. In this regard, it will be described in more detail below with reference to FIG.

When the sixth item 1070 is selected, an instruction to return to the screen that was executed immediately before the current screen is executed. Accordingly, when an unintended graphic image is added by mistake, the user can quickly return to the original screen at once without using the third item 1040.

When the seventh item 1010 is selected, at least two background images or captured images are displayed. The reason for designing such an option is that some time interval may occur between when the user issues a capture command and when the captured image is actually captured. Particularly, when the whole screen is changed to a totally different screen, it is expected that the seventh item 1010 is useful. With reference to FIG. 11, a detailed description will be given later.

11 is a view for explaining a list of captured images provided by a multimedia device according to another embodiment of the present invention.

10, it is possible to add an additional graphic image 1160 to the screen image captured using the indicator 1150 whose position changes according to the control of the remote controller.

However, since there may be a small amount of time gap between the time when the user inputs the capture command and the time when the captured image is captured, it is not possible to exclude the possibility that only a captured screen image desired by the user is provided.

Accordingly, the multimedia device according to the embodiment of the present invention is designed to continuously store video data for a predetermined time in a buffer or a memory. Also, after the predetermined time has elapsed, the video data for the previous time is deleted from the buffer or the memory, thereby improving the memory processing efficiency compared to the conventional art.

That is, the first captured image 1110 shown in the left area of FIG. 10 displays a still image frame of 0.25 seconds before the captured image, and the second captured image 1120 displays a still image frame of 0.5 seconds before the captured image The third captured image 1130 displays a 0.75-second previous still image frame from the captured time point, and the fourth captured image 1140 displays a one-second previous still image frame from the captured time point. That is, it is advantageous to further increase the possibility that the user selects a desired captured image by additionally providing four history lists at intervals of 0.25 seconds based on the viewpoint of the captured image.

Of course, it is also within the scope of the present invention to design the specific time unit described above so as to be changeable according to the design of the person skilled in the art or the choice of the user.

When the memo pad 1170 of FIG. 11 is selected, the memo window that the user can input is enlarged as shown in FIG. The reason for designing in this way is to solve the case where it is difficult for the user to distinguish the captured image from the text color.

When the user finishes editing the memo window shown in Fig. 12, the memo 1170 of Fig. 11 may be deleted or retained, and all belong to the scope of the present invention.

12 is a view for explaining a process of dividing a memo area provided by a multimedia device into at least two or more memo areas according to another embodiment of the present invention.

As shown in FIG. 12, in the area (b) for the user to input text or the like, an indicator or pointer holds the first graphic image 1231. On the other hand, as described above, in the area (a) for the user to select any item in the toolbox, the indicator or pointer is changed to the second graphic image 1232. Further, the toolbox may be completely hidden on the screen by using the hidden / unfolded option 1210 of the toolbox, thereby expanding the text input area as needed.

Accordingly, the user can input any text in the specific area 1220 using the indicator 1230 on the area b shown in Fig. Furthermore, it is also within the scope of the present invention to design a virtual QWERTY keyboard to be displayed in the area b.

In addition, according to another embodiment of the present invention, even if the user does not input the memo-creating time, the user can use the timer built in the device to set the time at which the memo was created ) Is automatically displayed and stored in the memory. Therefore, there is an advantage that the status of the memo can be grasped by the time zone in the future.

13 is a process diagram for explaining an example in which a multimedia device according to another embodiment of the present invention inputs text data for mapping with a captured image.

First, as shown in FIG. 13A, the multimedia device 1300 outputs video data input from a broadcasting station or an external source. At this time, it is assumed that, in accordance with one of the above-described embodiments (for example, the user's specific gesture or voice, etc.), the currently output video data is captured and only the still image is output.

Furthermore, for example, when a specific hot key (e.g., a "Quick Memo" button) attached to the remote controller is selected, a first area 1320 in which text or the like can be input as shown in FIG. A second area 1310 is provided which simultaneously provides a tool for editing the captured image.

Accordingly, when the user merely inputs a desired text into the first area 1320, the multimedia device generates integrated data in which the captured image and the input text are mapped.

14 is a process diagram for explaining another example in which a multimedia device according to another embodiment of the present invention inputs text data for mapping with a captured image.

First, as shown in Fig. 14A, the multimedia device outputs video data input from a broadcasting station or an external source. At this time, it is assumed that the currently output video data is captured according to one of the above-described embodiments (for example, the user's specific gesture or voice), and only the still image is output.

13, a pre-stored icon list 1410 including an icon 1411 for selecting a quick memo is output at the bottom of the screen as shown in FIG. 14 (a).

In this case, when the quick memo-related icon 1411 is selected using the remote controller, for example, as shown in FIG. 14 (b), a first area 1430 for inputting text or the like, A second area 1420 is provided which is provided with a tool for editing images.

Accordingly, when the user merely inputs a desired text into the first area 1430, the multimedia device generates integrated data in which the captured image and the input text are mapped.

15 is a process diagram for explaining another example in which a multimedia device according to another embodiment of the present invention inputs text data for mapping with a captured image. 13 and 14, it is assumed that the embodiment of the present invention outputs video data received from a broadcasting station or a CP (Content Provider). On the other hand, in the case of taking an image through a camera mounted on or connected to a multimedia device .

First, as shown in FIG. 15A, a multimedia device captures a user, objects, etc. around the multimedia device through a camera and corresponding video data 1510 is displayed on the screen. At this time, a menu 1520 including options related to camera settings and the like is output, and the moment the user selects the specific option 1521, the captured image is displayed instead of the video data.

Further, the user enters a command to enter a note through one of the above-described embodiments (e.g., a gesture, voice, or hotkey button on the remote controller).

Therefore, as shown in FIG. 15B, an area 1540 in which a text or the like can be input is displayed at a predetermined position, and a tool box 1530 including items for editing the captured image Simultaneously. Of course, it is also within the scope of the present invention to design the two areas 1530 and 1540 to display sequentially or in reverse order according to the needs of those skilled in the art.

16 is a diagram for explaining a process of editing a captured image by a multimedia device according to another embodiment of the present invention.

First, as shown in FIG. 16A, a captured image is displayed in a first area, and a tool box including items capable of editing the captured image is displayed together with a second area 1610 do. At this time, some of the first area and the second area 1610 may be output in an overlaid form, or may be designed to have transparency to provide the user with various options as much as possible.

Further, when the indicator 1611 whose position changes according to the control of the remote controller selects the specific item 1612 of the tool box 1610 shown in FIG. 16A, Screen. As described above, the graphic images of the indicator 1611 shown in FIG. 16A and the indicator 1630 shown in FIG. 16B are set differently.

Accordingly, it is possible to add the shape 1620 of editing the currently captured image using the changed graphical image indicator 1630. [ It is also within the scope of the present invention that only the portion of the selected shape 1620 is cropped and stored in the memory or that the entire captured image is reduced / enlarged.

17 is a view for explaining an example of inputting a voice signal for mapping a captured image with a multimedia device according to another embodiment of the present invention.

As described above, according to the embodiment of the present invention, the user's voice can be designed to be mapped to the captured image and stored in the memory instead of the text inputting method or the captured image editing method.

For example, as shown in FIG. 17, in a toolbox 1710 that includes items for editing a captured image, the user is able to select a particular item 1712. As the selection method, the user places the indicator 1711 in the specific item 1712 using the remote controller. From this point on, the speech recognition engine installed in the multimedia device is designed to operate. Accordingly, there is an advantage that unnecessary speech recognition data processing steps need not be performed.

As shown in FIG. 17, the user 1720 only has to comment on any information that he maps and stores with the captured image in a voice form (e.g., "Let's meet tomorrow" 1721) The currently captured image and corresponding voice signal are automatically stored in memory. Of course, it is also within the scope of the present invention to design a multimedia device that recognizes a user's voice signal to be converted into a text form again.

18 is a diagram for explaining a process of storing the integrated data including the captured image in a random memory according to another embodiment of the present invention. The integrated data represented in the present specification may mean, for example, new memo data in the form of a combination of the captured image and the additional information. In particular, the additional information includes at least one of text, as well as voice or graphic images.

Hereinafter, in FIG. 18, the options for storing the previously-described integrated data (single data in which the captured image and the additional information are mapped) are stored in a specific memory desired by the user will be described.

First, when capturing and inputting additional information (text) for video data is completed and integrated data is generated, a tool box 1810 for editing is displayed as shown in FIG. 18 (a). At this time, an indicator (or pointer) 1811 is used to select a specific item 1812 in the tool box 1810. The specific item 1812 performs a function of storing newly generated integrated data.

For example, as shown in Fig. 18 (b), an optional graphic image 1820 for selecting a memory to be stored is displayed. A memory (ex: TV internal memory) in the multimedia device is set as a default value, but an option to select an external memory (ex: X-TICK 2) connected to the multimedia device is also provided. Therefore, the user can quickly store the integrated data in the external memory as well as the internal memory of the multimedia device.

Finally, when the storage in the specific memory is completed, a message 1830 indicating that the storage has been successfully completed is output on the specific area of the predetermined position as shown in FIG. 18C.

18, a process of simply storing newly generated integrated data in a memory has been described. However, in the case where a user quickly transfers or transmits integrated data (which may be referred to as quick memo) to an external specific device . In this regard, it will be described later with reference to FIG.

FIG. 19 is a diagram for explaining a process in which a multimedia device according to another embodiment of the present invention transmits integrated data including a captured image to an arbitrary device or group. FIG.

Hereinafter, in Fig. 19, options for sharing the previously described integrated data (single data in which the captured image and the additional information are mapped) are shared by a specific device or server desired by the user will be described.

First, when capturing and inputting additional information (text) for video data is completed and integrated data is generated, a tool box 1910 for editing is displayed as shown in FIG. 19 (a). At this time, a specific item 1912 in the tool box 1910 is selected by using an indicator (or pointer) 1911. The specific item 1912 carries out a function of transmitting the newly generated integrated data to an arbitrary device, a server, or the like.

For example, as shown in FIG. 19 (b), an optional graphic image 1920 for transmitting newly generated integrated data is displayed. Accordingly, the user can conveniently select a method such as e-mail / SNS (Social Network Service) / MMS, or preset a device or a server to be transmitted and executed prior to the execution of the capture function. Belongs.

Particularly, in the case of designing to be set in advance, the user has the effect that the additional information mapped on the screen that he / she is viewing can be transmitted to other people more quickly.

20 is a flowchart for explaining an embodiment for controlling a multimedia device. It is also within the scope of the present invention to supplement the analysis of FIG. 20 with reference to the embodiments described above in FIGS. 1 to 19 above.

The method of controlling a multimedia device connected to at least one network interface according to an exemplary embodiment of the present invention includes displaying moving image data received from an external device through a first network interface (S2010).

In addition, the multimedia device detects predetermined events (S2020), and outputs a list including at least one still image data associated with the moving image data (S2030).

In addition, the multimedia device stores the integrated data in a state in which the specific still image data in the list is mapped to any information in the memory (S2040), and transmits the integrated data stored in the memory to the second network interface To the external device (S2050).

The preset specific event corresponds to at least one of a gesture of a user using the multimedia device, a voice, or a specific command generated by a remote controller, for example. The moving picture data received from the outside corresponds to any one of a broadcast signal received from a broadcasting station or video data photographed by a camera of the multimedia device.

FIG. 21 is a flowchart detailing step S2030 shown in FIG.

If the preset specific event is detected, the step of outputting the list shown in FIG. 20 (S2030) may include dividing the screen of the multimedia device into at least two areas (S2033), determining the position of the indicator (S2032), converting the indicator into the first graphic image (S2033) when the indicator is located in the first area including the still image (S2033) And converting the indicator into a second graphic image (S2034) if the indicator is located in the second area including the second graphic image.

Further, although not shown in FIGS. 20 and 21, the method further includes receiving arbitrary information for mapping the specific still image data according to another embodiment of the present invention. The arbitrary information includes at least one of, for example, text, audio data, or graphic image. Also, at least one or more still image data associated with the moving image data corresponds to at least one or more images captured retroactively, for example, from a point in time when the specific event is detected, retrospectively.

20 and 21, according to another embodiment of the present invention, it is designed to further include displaying at least one or more options for specifying an external device.

22 is a flowchart for explaining another embodiment for controlling the multimedia device. It is also within the scope of the present invention to supplement the analysis of FIG. 20 with reference to the embodiments described above in FIGS. 1 to 19 above.

First, the multimedia device according to another embodiment of the present invention receives an input command for executing a quick memo (S2210), and captures moving image video data currently being output (S2220).

In step S2230, it is determined whether the currently captured screen image is selected. If not, at least one snapshot or the like prior to the captured time is displayed in step S2240. A specific snapshot of the at least one snapshot is selected (S2250).

On the other hand, if the determination result is selected, an editing tool is provided (S2260), and only a specific region is selected from the entire captured screen (S2270). Of course, it is also within the scope of the present invention that the user is not provided with a resizing function but is designed to store the captured screen as it is.

(E.g., text, voice, etc.) for the memo (S2280), and then transmits the integrated data including all of the additional information mapped to the captured image in the predetermined e-mail, MMS or SNS format (S2290).

The apparatus and the control method according to an embodiment of the present invention may implement other embodiments by combining the above-described drawings and the drawings or by combining matters obvious to those skilled in the art, which also falls within the scope of the present invention.

Meanwhile, the method of operating the electronic device of the present invention can be implemented as a code that can be read by a processor on a recording medium readable by the processor included in the electronic device. The processor-readable recording medium includes all kinds of recording apparatuses in which data that can be read by the processor is stored. Examples of the recording medium that can be read by the processor include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and may also be implemented in the form of a carrier wave such as transmission over the Internet . In addition, the processor-readable recording medium may be distributed over network-connected computer systems so that code readable by the processor in a distributed fashion can be stored and executed.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the present invention.

500: Memory
510: first network interface module
520: display module
530: Video processor
540: Audio Processor
550: Speaker
560: Controller
570: second network interface module
580: Detecting module
590: List generation module

Claims (19)

A multimedia device connected via at least one or more network interfaces,
A first network interface module for receiving a broadcast signal in which audio data and video data are multiplexed;
A video processor for processing video data in the multiplexed broadcast signal;
An audio processor for processing audio data in the multiplexed broadcast signal;
A display module for outputting the processed video data;
A speaker for outputting the processed audio data;
A detecting module for detecting a predetermined event;
A list generating module for generating a list including at least one still image data associated with the video data at a point in time when the preset specific event is detected;
A controller for storing the integrated data in a state in which the specific still image data in the list and the arbitrary information are mapped; And
A second network interface module for transmitting the integrated data stored in the memory to an external device,
Lt; / RTI > The method according to claim 1,
When the preset specific event is detected,
The controller comprising:
Controls the speaker to maintain the output of the processed audio data,
And to control the display module to stop outputting the processed video data, and to store the video data in the memory. The method according to claim 1,
When the preset specific event is detected,
The controller comprising:
Dividing the screen of the multimedia device into at least two areas,
If the indicator is located in a first area containing a still image, converting the indicator into a first graphic image,
The multimedia device being designed to convert the indicator to a second graphic image if the indicator is located in a second area including an editing tool. The method according to claim 1,
An OCR (Optical Character Recognition) module for extracting text included in the specific still image data
Lt; / RTI > The method according to claim 1,
A user interface for inputting arbitrary information for mapping the specific still image data,
Lt; / RTI > 6. The method of claim 5,
The above-
Wherein the multimedia device comprises at least one of text, audio data, or graphic images. The method according to claim 1,
The at least one still image data associated with the video data,
Wherein the predetermined event corresponds to at least one image captured retroactively from a point in time when the specific event is detected. The method according to claim 1,
The controller comprising:
The multimedia device being designed to control the display module to display at least one or more options for specifying an external device. The method according to claim 1,
The predetermined specific event is a predetermined event,
Wherein the multimedia device corresponds to at least one of a gesture of a user using the multimedia device, a voice, or a specific command generated by a remote controller. The method according to claim 1,
The multimedia device comprising:
A multimedia receiver, a DTV, a network TV, an Internet TV, or a smart TV. A method of controlling a multimedia device connected to at least one or more network interfaces,
Displaying moving picture data received from outside via a first network interface;
Detecting a preset specific event;
Outputting a list including at least one still image data associated with the moving image data;
Storing integrated data in a state in which specific still image data in the list is mapped to arbitrary information in a memory; And
Transmitting the integrated data stored in the memory to an external device via a second network interface
Gt; a < / RTI > 12. The method of claim 11,
When the preset specific event is detected,
The step of outputting the list may include:
Dividing the screen of the multimedia device into at least two areas;
Converting the indicator to a first graphic image if the indicator is located in a first area containing a still image; And
If the indicator is located in a second area including an editing tool, converting the indicator to a second graphic image
Gt; a < / RTI > 12. The method of claim 11,
Receiving the arbitrary information for mapping with the specific still image data
Further comprising the steps of: 14. The method of claim 13,
The above-
Wherein the multimedia data comprises at least one of text, audio data, or graphic images. 12. The method of claim 11,
The at least one still image data associated with the moving image data,
Wherein at least one image captured after retreating the predetermined past period from the point at which the specific event is detected corresponds to the captured image. 12. The method of claim 11,
Displaying at least one or more options for specifying an external device
Further comprising the steps of: 12. The method of claim 11,
The predetermined specific event is a predetermined event,
Wherein the multimedia device corresponds to at least one of a gesture of a user using the multimedia device, a voice, or a specific command generated by a remote controller. 12. The method of claim 11,
The moving picture data received from the outside includes:
Wherein the multimedia device corresponds to any one of a broadcast signal received from a broadcasting station or video data photographed by a camera of the multimedia device. A computer-readable recording medium having recorded thereon a program for executing the method according to any one of claims 11 to 18.

Images