KR20100135131A - Method of processing image in a display apparatus and the display apparatus - Google Patents

Abstract

PURPOSE: A method for processing an image in a display apparatus and the display apparatus are provided to capture at least a part from the displaying entire picture and use at least one reference frame among previous frames, thereby converting the captured image into a super resolution image. CONSTITUTION: An output unit(113) displays at least one processed image on a screen. A storage unit stores previous frames based on a frame of the displaying image while the image is displayed on the screen. The storage unit captures the image of at least a partial picture. The storage unit stores next frames after the frame of the captured image. A super resolution processing unit(111) converts the resolution of the captured image into a super resolution image.

Description

Image processing method and display device in a display device {Method of processing image in a display apparatus and the display apparatus}

The present invention relates to a display apparatus for processing an image being viewed and an image processing method in the display apparatus.

In general, a display device such as a television processes and displays an input video signal.

The display device may process and display a broadcast signal received through a broadcast network, may process and display a video signal input through a communication network such as the Internet, or may process and display a video signal input through an external input terminal. You may.

SUMMARY OF THE INVENTION An object of the present invention is to provide a method for processing an image in a display apparatus and a display apparatus for capturing at least a portion of an image being displayed on a display apparatus and upscaling the resolution of the captured image and providing the same to a user. .

According to an aspect of the present invention, there is provided a method of processing an image, the method including: storing previous M frames based on a frame of the image while displaying an image on a screen of the display device; Capturing at least a portion of the entire screen being displayed and storing N frames thereafter based on a frame of the captured image when a capture command is input while displaying the image; Upscaling the resolution of the captured image using at least one of the previous M / post N frames as a reference frame; Storing the upscaled image on a corresponding storage medium according to a storage command of the upscaled image; And displaying the upscaled image on at least a portion of a screen in a corresponding display type according to a display command of the upscaled image.

The upscaling may include upscaling the captured image to a super resolution image.

The upscaling may include: obtaining motion information between the captured image and the reference frame by performing motion estimation by receiving at least one of the captured image and the previous M / N frames after the captured image; Mapping motion information of each reference frame to super resolution coordinates; And obtaining a pixel value of a super resolution image from pixel information of a reference frame mapped to the super resolution coordinates, and outputting a super resolution image.

The storing may include displaying, on a menu screen, storage medium information that can be stored in the display apparatus according to a storage command of the upscaled image; And storing the upscaled image in the storage medium selected by the user on the menu screen.

The displaying may include displaying at least one display type on a menu screen according to a display command of the upscaled image; And displaying the upscaled image according to the display type selected by the user on the menu screen.

The display may include displaying the upscaled image on the entire screen when the full screen is selected among the display types, and displaying the upscaled image on the entire screen when the simultaneous screen is selected. It is characterized by.

The displaying may further include setting the upscaled image as a background screen of the display device according to a background screen designation command of the upscaled image.

The display apparatus according to an embodiment of the present invention, when at least one of a broadcast received through at least one of a broadcasting network and a communication network, an image input from an external device through an external input terminal, and an image input from a storage medium are processed and input. Output unit for displaying this on the screen; At least one of a volatile internal storage medium, a nonvolatile internal storage medium, and a nonvolatile external storage medium, wherein the previous M frames are stored based on a frame of the image being displayed while the image is displayed on the screen, and a capture command A storage unit for capturing an image of at least some screens of the entire screen and storing N frames thereafter based on a frame of the captured image; And a super resolution processor that converts the resolution of the captured image into a super resolution image by using at least one of the previous M / n N frames as a reference frame. The storage unit stores the super resolution image in a corresponding storage medium according to a storage command of the super resolution image, and the output unit displays the super resolution image in at least a portion of a screen in a corresponding display type according to a display command of the super resolution image. Can be displayed.

The super resolution processor includes: a motion estimator configured to obtain motion information between the captured image and the reference frame by performing motion estimation by receiving at least one of the captured image and the previous M / n N frames after the captured image; An interpolator for mapping motion information of each reference frame to super resolution coordinates; And an image reconstruction unit configured to obtain a pixel value of the super resolution image while removing blur and noise from pixel information of the reference frame mapped to the super resolution coordinates, and output a super resolution image.

Here, M and N may have the same value or may have different values. M is zero or greater than zero. N is zero or greater than zero.

Other objects, features and advantages of the present invention will become apparent from the following detailed description of embodiments taken in conjunction with the accompanying drawings.

According to the present invention, while displaying an image on the screen of the display device, and stores the previous M frames based on the frame of the image, if a capture command is input while displaying the image, at least a portion of the entire screen being displayed Captures and stores subsequent N frames based on frames of the captured image, and converts the captured image into a super resolution image by using at least one of the previous M / N after frames as a reference frame. . By doing so, the captured image having low image quality has an effect of increasing size and resolution. In addition, after storing the super resolution image in a storage medium of the display device, it can be used as a wallpaper on another external device, or can be used as a background screen of the display device. In addition, a continuous capture section may be set, and a continuous still may be configured by converting an image captured during the set continuous capture section to super resolution. As described above, the present invention can be variously applied to the captured image converted to super resolution.

Hereinafter, with reference to the accompanying drawings, preferred embodiments of the present invention that can specifically realize the above object will be described. At this time, the configuration and operation of the present invention shown in the drawings and described by this will be described as at least one embodiment, by which the technical spirit of the present invention and its core configuration and operation is not limited.

The terms used in the present invention were selected as widely used general terms as possible in consideration of the functions in the present invention, but may vary according to the intention or custom of the person skilled in the art or the emergence of new technologies. In addition, in certain cases, there is a term arbitrarily selected by the applicant, and in this case, the meaning will be described in detail in the corresponding description of the invention. Therefore, it is intended that the terms used in the present invention should be defined based on the meanings of the terms and the general contents of the present invention rather than the names of the simple terms.

The present invention enables the display device to capture at least a portion of an image being displayed and upscale the resolution of the captured image to provide the user.

In particular, the present invention allows to provide a user with upscaling the resolution of the captured image to super resolution.

The display device of the present invention is a device capable of providing a user after receiving and processing a broadcast signal transmitted by any one of terrestrial waves, satellites, and cables. In addition, if the display device is an Internet Protocol TV (IPTV), it may be provided to a user after receiving and processing a broadcast signal through an internet network. That is, the image displayed through the display device of the present invention may be input through a broadcasting network such as terrestrial wave, satellite, or cable, may be input through a communication network such as the Internet, or may be input through an external input terminal. Various external devices may be connected to the external input terminal. For example, external devices include camcorders, DVD players, VTRs, digital cameras, mobile phones, personal computers (PCs), surveillance cameras, and the like.

The display device may have a storage medium such as an HDD mounted therein to serve as an internal storage medium.

In addition, an external storage medium may be connected to the display device through the external input terminal. The external storage medium may be an HDD, a USB memory, or the like. As another example, the mobile phone, PC, etc. may be used as an external storage medium.

1 is a block diagram showing an embodiment of a television system of the display device according to the present invention.

1, the control unit 100, the tuner unit 101, the demodulator 102, the demultiplexer 103, the video decoder 104, the display processor 105, the audio decoder 106, the external input processor 108 ), A memory 109, an internal storage medium 110, a super resolution processing unit 111, a user input unit 112, an output unit 113, and a recording / reproducing unit 114. At this time, each unit is connected through the internal bus 120.

The memory 109 is a volatile memory capable of storing a plurality of frames. For example, SDRAM, DRAM, or the like may be used as the memory. A hard disk drive (HDD) may be mounted in the display device to be used as the internal storage medium 110.

In addition, an external device may be connected to the external input processor 108 through an external input terminal, and / or an external storage medium 121 may be connected.

That is, the HDD or USB memory may be connected to the external input terminal of the external input processor 108 to be used as the external storage medium 121.

In addition, if the external input terminal is capable of bidirectional communication and an external device connected to the external input terminal can store an image output from the display device through the external input terminal, the external device may be used as an external storage medium. For example, an image stored in a mobile phone or a digital camera may be displayed on a display device, or an image displayed on the display device may be output to a mobile phone or a digital camera and stored.

The present invention may store broadcast signals received by the display apparatus using at least one of the internal storage medium 110 or the external storage medium 121. According to an embodiment of the present invention, the storing is performed through any one of instant recording, scheduled recording, and time shift. The time shift may also be referred to as a time machine. In addition, in the present invention, the image which is upscaled to the super resolution may also be stored using at least one of the internal storage medium 110 or the external storage medium 121. In another embodiment, if an external device connected through an external input terminal includes a storage medium, and the external input terminal is capable of bidirectional communication, the upscaled image in super resolution may be stored in the external device. At this time, the external device may be a mobile phone, a PC, a digital camera, a camcorder, a DVD player, a VTR, or the like.

The tuner 101 receives a broadcast signal of a specific channel transmitted by any one of terrestrial waves, satellites, and cables, and outputs it to the demodulator 102. In this case, the specific channel may be set in advance in the system or may be a channel selected by the user through the user input unit 112. The user may select a specific channel by performing viewing reservation, recording reservation, channel change, etc. using an input device such as a remote controller, a touch screen, a mouse, or the like. The channel selected by the user is input to the controller 100 through the user input unit 112, and the controller 100 controls the tuner unit 101 so that the broadcast signal of the desired channel is tuned.

The demodulator 102 demodulates the broadcast signal of the channel tuned by the tuner 101. The demodulator 102 demodulates the inverse of the modulation scheme performed on the received broadcast signal. For example, if the modulation scheme is a VSB scheme, demodulation is performed in a VSB scheme. If the modulation scheme is an OFDM scheme, demodulation is also performed in an OFDM scheme.

The broadcast signal demodulated by the demodulator 102 is input to the demultiplexer 103. The demultiplexer 103 separates the audio and video streams multiplexed on the input broadcast signal and outputs the audio and video streams to the corresponding decoders 104 and 106 through the memory 109. The broadcast signal may be multiplexed with a data stream, but since the data stream is not a feature of the present invention, it will not be described in the present invention. On the other hand, if a plurality of programs are multiplexed on the broadcast signal input to the demultiplexer 103, one of the programs is selected and the audio and video streams are separated from the selected program.

The video decoder 104 performs video decoding on a video stream input using reference frames stored in the memory 109, restores the video stream to a state before compression, and outputs the same to the display processor 105. The video decoded data is also stored in the memory 109 for use as a reference frame in the video decoding process of the next frame. The audio decoder 106 performs audio decoding on the input audio stream, restores it to a state before compression, and outputs it to the output unit 113.

For example, the video decoder 104 may decode the video stream using at least one of an MPEG-2 video decoding algorithm, an MPEG-4 video decoding algorithm, an H.264 decoding algorithm, an SVC decoding algorithm, and a VC-1 decoding algorithm. have. The audio decoder 106 is an AC-3 decoding algorithm, MPEG-2 audio decoding algorithm, MPEG-4 audio decoding algorithm, AAC decoding algorithm, AAC + decoding algorithm, HE AAC decoding algorithm, AAC SBR decoding algorithm, MPEG surround decoding algorithm, At least one of the BSAC decoding algorithms may be used to decode the audio stream.

The display processing unit 105 outputs the video data restored to the state before compression in the video decoder 104 to the output unit 113 in synchronization with the display synchronization signal. In this case, the display processing unit 105 may convert the video data according to the display format and output the video data to the output unit 113. If the video data is an I, P, or B picture structure, the display processing unit 105 rearranges the output data. You can also output The memory 109 is also used at this time.

The output unit 113 includes a speaker, a TV screen, and the like, a video signal is output through a screen, and an audio signal is output through an internal or external speaker.

On the other hand, the user can select any one of instant recording, scheduled recording, and time shift, and the storage / playback unit 114 is currently displaying or scheduled if any one of instant recording, scheduled recording, and time shift is selected by the user. The broadcast of the corresponding channel received at the predetermined time is stored in the storage medium. The time shift may be automatically performed by the display device. In another embodiment, when the phone can receive a call, the device may automatically perform a time shift operation when a call is received while displaying an image input from a broadcast or external device. In addition, a time shift operation may be performed by using an input device such as a remote controller.

In this case, the storage / reproducing unit 114 may store the image before decoding in the storage medium in consideration of the capacity of the storage medium, and then decode and output the image to the screen during reproduction. Alternatively, when the image to be stored is an analog signal or an uncompressed signal, the image may be compressed using an encoder and then stored in a storage medium. To this end, the storage / reproducing unit 114 may further include an encoder therein.

As another example, the storage / reproducing unit 114 may store a high quality image as it is in a storage medium in consideration of the image quality of the image to be reproduced. When the captured image according to the present invention is converted into a super resolution image and stored in a storage medium, the super resolution image may be stored as it is or may be compressed and stored.

The storage medium may be any one of the internal storage medium 110 and the external storage medium 121, and other external devices connected through an external input terminal, for example, a mobile phone, a camcorder, a digital camera, a VTR, and a recordable device. It may be a DVD player. According to an embodiment of the present invention, it is stored in at least one of the internal storage medium 110 and the external storage medium 121. In this case, the user may allow the user to select at least one of the internal storage medium 110 and the external storage medium 121, or if a recording command is input after setting one storage medium as a default through a display device, a corresponding video is input. May be stored in a storage medium set as a default. For example, if the internal storage medium 110 is set as a default, the image is stored in the internal storage medium 110 by immediate recording, scheduled recording, or time shift operation.

Subsequently, when the user selects playback of an image stored in the internal or external storage media 110 and 121, the storage / reproducing unit 114 reads the image to be reproduced from the corresponding storage medium and outputs the image to the demultiplexer 103 or output unit ( Output to 113).

In this case, when the playback of the stored video is selected by the time shift function, an OSD-type guide bar may be displayed and the recording information and the playback time may be displayed.

12 shows an embodiment of a guide bar according to the present invention.

The guide bar may include a position indicator indicating a current position.

The position indicator is a visual indicator of the progress of the current playback in proportion to the total storage time. The position indicator may be any means that can visually display the current state position, in an embodiment may be an arrow, a line or a figure. The playback time can be changed by moving the position indicator forward and backward. At this time, it is also possible to move the position indicator directly, it is also possible to move using the control key. As a method of directly moving, a touch screen, a mouse pointer, a remote controller, or a direction key of a display device may be used. This is only an embodiment, and any means for inputting a signal is possible.

The guide bar may also include a cache bar. The cache bar is a semantic unit that visually illustrates the state of being stored. That is, the cache bar increases in proportion to the storage state.

Time information may be further displayed on the information bar. For example, the time information may be play time information of the entire recorded program, current play time information, time information at the time of recording, and the like.

In addition, if the display device has a simultaneous screen function, the currently viewed image and the time shifted image may be simultaneously displayed on one screen according to a user's selection.

The simultaneous picture function is a function of dividing one screen into a main screen and one or more sub-screens to display independent images, and includes a picture in picture (PIP), a picture out picture (POP), a double window (or a twin picture), and the like. There is this. The PIP means that one or more sub-screens are displayed in the main screen, and the POP means that one or more sub-screens are displayed on the left (or right) side of the main screen. In addition, a double window means splitting one screen into left and right screens, displaying a main screen image on the divided left screen, and a subscreen image on the right screen. In this case, the size of the left and right screens may or may not be the same.

When the storage space for the time shift is fixed in the storage medium, when the storage space for the time shift becomes full, the current display is deleted by first deleting the stored data and storing the data at the location or overwriting the data. You can save the video.

Meanwhile, the external input processor 108 may include various external input terminals. The external input processor 108 may include at least one of a digital video interface (DVI) terminal, a high definition multimedia interface (HDMI) terminal, an S terminal, a component terminal, a composite terminal, and a universal serial bus (USB) terminal. The external input terminal may transmit a video signal in only one direction or a video signal in both directions. The external input terminals are embodiments to help the understanding of the present invention, and any external input terminal capable of providing an external image to the external input processing unit 108 can be provided. Examples will not be limited.

In addition, various types of external devices may be connected to the external input terminals. For example, a personal computer (PC), camcorder, surveillance camera, digital camera, mobile phone, DVD player, VTR and the like can be connected. In addition, various types of external devices may be connected through the USB terminal, or an external storage medium may be connected. For example, a mobile phone, a digital camera, or the like can be connected through the USB terminal. HDD, USB memory, etc. may also be connected through the USB terminal. In addition, the external input processor 108 may include a USB hub to connect various external devices or storage media through one USB terminal.

According to an embodiment of the present invention, a storage medium such as an HDD, a USB memory, or the like connected through a USB terminal of the external input processor 108 may be used as the external storage medium 121.

The external input processor 108 may receive an external video signal from an external device connected to an external input terminal. In addition, the external input processor 108 may receive an image signal from an external storage medium 121 connected to an external input terminal.

The external input processor 108 may include a color coordinate converter to convert the color coordinates of the input image signal into the color coordinates of the display device. For example, if the color coordinate of the video signal input from an external device is RGB and the color coordinate of the display device is YCbCr, the color coordinate converter converts the RGB of the video signal input from the external device to YCbCr.

The external input processor 108 may include a format converter to convert the format of the input video signal into the format of the display apparatus. For example, if the format of the video signal input from the external device is 640 * 480 and the format of the display device is 1920 * 1080, the format converter may convert the format of the video signal input from the external device to 1920 * 1080. have. The format conversion function may be performed by the display processor 105. In addition, if the frame rate of the image input from the external device and the frame rate of the display device do not match, the external input processor 108 or the display processor 105 may convert the frame rate according to the display device.

The external input processing unit 108 may include at least one codec algorithm to provide an external image input through an external input terminal to the user through the output unit 113. The codec algorithm may be embedded in the external input processor 108 at the time of shipment of the display device, or may be downloaded using an external device such as a PC or an external storage medium. In addition, the existing codec algorithm may be upgraded using the external device or the external storage medium.

The external input processing unit 108 receives a movie or UCC (User Created Contents) video of various file formats through an external device such as a camcorder or a PC or an external storage medium 121, encodes the corresponding codec algorithm, and outputs the output unit ( And display via 113).

The external input processing unit 108 may be equipped with a wireless function, and may receive a picture or video stored in a mobile phone, a PC, etc. supporting the function, encode the corresponding codec algorithm, and display the same through the output unit 113. That is, a picture or video stored in a mobile phone, a PC, etc. can be input from a display device such as a TV and displayed without a cable connection.

Meanwhile, while watching an image displayed through the screen of the display device, the user may use the capture function. The image displayed on the screen of the display device may correspond to a broadcast program received from a broadcasting station, an advertisement, or the like, and may also correspond to an image input from an external device or a storage medium. The capture function may be performed by inputting a specific button of an input device, for example, a remote controller.

According to an embodiment of the present invention, when the capture function is selected, the user may select the entire screen or a part of the entire screen that is currently displayed.

In another embodiment of the present invention, the capture section can be designated by the user to generate a continuous still cut.

The captured image may be displayed on a display device or may be stored in the internal storage medium 110 or the external storage medium 121 of the display device. Alternatively, the data may be stored in another storage medium through an external device. A detailed description of the display and storage method will be given later.

In this embodiment, the present invention upscaling the resolution of an image captured during a specific screen or a specific section.

In an embodiment, the resolution of the upscaled image is super resolution.

To this end, the present invention includes a super resolution processing unit 111.

That is, according to the embodiment of the present invention, the captured image is converted into a super resolution image by using a plurality of reference frames. In this embodiment, the reference frame is the previous M frames and the subsequent N frames based on the frames of the captured image. The M and N values may vary depending on the capacity of the memory and the processing capability of the super resolution processor 111. The M and N may have the same value or may have different values. In addition, when converting to a super resolution image, the higher the number of reference frames, the higher the quality of the image converted to the super resolution. Therefore, as M and N increase, the image quality of the image converted to super resolution becomes better. However, as the number of reference frames increases, the converting time to a super resolution image becomes longer. Therefore, the number of reference frames may be adjusted in consideration of image quality and execution time.

In the present invention, the M + N reference frames may be stored in the memory 109 used for the video decoding, or may be stored in the internal storage medium 110 or the external storage medium 121. In the present invention, it is assumed that the capacity of the memory 109 is sufficient, and M + N reference frames are stored in the memory 109 according to an embodiment. In this case, a portion of the storage area of the memory 109 may be allocated and used as an area for super resolution conversion.

In addition, the storage area for the super resolution conversion may be divided into an area for storing the previous M frames and an area for storing the N frames thereafter. To do this, each area must have enough capacity to store M or N frames.

In another embodiment, if the capacity of the memory 109 is not sufficient, only the previous M frames are stored as reference frames based on the currently displayed image frame, and the images captured using the M reference frames are stored. You can also convert to a super resolution image. In this case, image quality is lower than that of an image converted to super resolution using M + N reference frames.

And since the user does not know when to request a capture, the present invention always stores the previous M frames in the memory 109 based on the image frame currently being displayed.

At this time, the previous M frames are continuously stored in the FIFO (First Input First Output) method. For example, suppose you want to store four previous frames for super resolution conversion, and you want to save a new previous frame when all four previous frames are stored. Save or overwrite the new frame with the oldest previous frame. In this case, the M frames immediately before the image frame currently being displayed are always stored in the storage area of the memory 109 storing the previous frame.

As soon as the user inputs a capture command, N frames are stored in the memory 109 from the currently displayed image frame and the next frame of the current image frame.

When the capture command is input as described above, when not only the frame of the image displayed on the screen but also the previous M frames and the following N frames are stored in the memory 109, the super resolution processing unit ( 111 converts the resolution of the captured image into super resolution using the previous M frames and the subsequent N frames as reference frames.

2 is a detailed block diagram illustrating an embodiment of a super resolution converter, and may include a motion estimator 201, an interpolator 202, and an image reconstructor 203.

The motion estimator 201 includes M previous frames (Y-1 to YM) and N subsequent frames (Y + 1 to Y + N) based on the captured image, that is, the current frame (Y0) and the current frame. , M + N + 1 frames are input, motion estimation is performed, and the result is output to the interpolator 202. In this case, the M + N previous / post frames are used as reference frames. That is, the motion estimator 201 obtains a motion vector with the current frame by performing motion estimation between the current frame Y0 and each reference frame. The motion vector represents a direction and a magnitude of a motion between the current frame and the reference frame. This process is sometimes referred to as obtaining warping information from the modeling process. According to an embodiment of the present invention, the motion vector is obtained up to a sub pixel unit and the rotation as well as the translation. The reason for performing the motion estimation in the sub-pixel units is that the exact position in the middle of the pixel can be obtained when mapping to the super-resolution image grid when there is a motion vector value in the sub-pixel unit.

The interpolator 202 maps the super-resolution image grid based on the motion estimation result of the motion estimation unit 201, and outputs the result to the image reconstruction unit 203.

That is, the interpolator 202 performs a process of mapping each M + N frames to super resolution coordinates using the motion vector obtained by the motion estimator 201. In this case, as the information of the reference frame is evenly distributed among the pixels, and the number of the reference frames is larger, the image quality of the image converted to super resolution is improved.

The image reconstruction unit 203 performs a process of obtaining a pixel value of a super resolution image from pixel information of a reference frame mapped to super resolution coordinates. At this time, the image reconstructor 203 obtains a super resolution image while removing blur and noise. That is, the image restoration unit 203 may remove blur and noise at the same time because the image restoration unit 203 restores the super resolution image using the concept of image restoration rather than simple interpolation. In this way, a clearer and clearer super resolution image can be obtained.

When the captured image is converted to the super resolution image as described above, the resolution of the captured image is converted to the super resolution, and the size of the captured image is also enlarged.

3 is a flowchart illustrating an embodiment of a process of capturing an image and converting the captured image into a super resolution image according to the present invention.

That is, when the image signal received from the tuner or the outside is processed and displayed on the screen (S301), the previous M frames are stored in the memory 109 based on the frame displayed on the current screen (S302).

At this time, if a capture command is input by the user (S303), the current frame that was displayed on the screen when the capture command is input and N frames thereafter are stored in the memory 109 based on the current frame (S304). Here, the current frame becomes an image captured by the capture command.

If all N frames are stored in the memory 109 after the current frame and the current frame in step S304, capturing by using the previous M frames and the next N frames stored in the memory 109 as reference frames. The resolution of the captured image, that is, the current frame, is converted to super resolution (S305).

An example of a method of converting the captured image into a super resolution image is shown in FIGS. 2 and 4.

The image converted to the super resolution in S305 may be stored in a storage medium or displayed on a screen (S306).

Here, the user may select whether to store the image converted to super resolution on a storage medium, display on the screen, and on which storage medium to store the image, and if so, how to display the image, and display at least one operation on the display device. May be set as a default, and the set operation may be automatically performed.

4 is a flowchart showing an embodiment of a method of converting a captured image into a super resolution image according to the present invention, and shows an example of converting the captured image into a super resolution image by applying FIG. 2.

That is, when the capture command is input, the variable n is set to -M (S401), and the nth frame is loaded from the memory 109 (S402). After the motion vector is obtained by performing motion estimation between the current frame and the loaded n-th frame (S403), the motion vector is stored (S404). According to an embodiment of the present invention, the motion vector is obtained up to a sub pixel unit and the rotation as well as the translation. Then, it is determined whether the loaded frame is the Nth frame (S405). Otherwise, the variable n is increased by 1 and the process returns to S402 to continue the motion estimation process (S406).

If it is determined as the Nth frame in S405, since the motion estimation process between the current frame, the previous M frames, and the subsequent N frames is performed, the process proceeds to S407. In step S407, a process of mapping each M + N frames to super resolution coordinates is performed using the motion vector stored in step S404.

When a process of mapping to super resolution coordinates (or grids) is performed in S407, a super resolution image is obtained from pixel information of a reference frame mapped to the super resolution coordinates while removing blur and noise (S408).

The super resolution conversion operation shown in the block diagram of FIG. 2 and the flowchart of FIG. 4 may be performed by any one of hardware, firmware, middleware, and software, or a combination of at least two of them.

5 is a flow chart of the present invention showing in more detail the process of capturing the image being displayed, the process of converting the captured image to a super-resolution image to store and / or display.

That is, when the image signal received from the tuner or the outside is processed and displayed on the screen (S501), the user who is watching this may request to capture a specific screen or a specific section using the remote controller (S502). In this case, the previous M frames are always stored in the memory 109 based on the frame displayed on the current screen regardless of the capture request.

If it is determined in S502 that the capture command is input, the user can select the entire image or a part of the entire image currently being displayed as the captured image (S503).

Therefore, the image to be captured may be a full screen or may be part of the full screen. In addition, only one image may be captured, or a plurality of images may be captured by specifying a section to be captured. In addition, when capturing a plurality of images by specifying a section, a continuous still cut may be generated.

When generating the continuous still cut, audio of the still cut may be output through the speaker at the normal speed as it is, or audio may be output through the speaker at the same double speed according to the double speed of the continuous steel cut. Or you can convert it to text and display it as a subtitle on the screen when the still cut is displayed. The audio output in the text form may be displayed horizontally or vertically on the screen on which the still cut is displayed, or may be displayed by giving a special effect like a bubble. The method of converting the audio into the text may use a known technique, and the method of displaying the audio converted into the text on the screen may be changed by the system designer, and thus the present invention will not be limited to the above-described embodiment.

6 illustrates an embodiment of a menu for allowing a user to select a captured image when an input of a capture command is input in the form of an On Screen Display (OSD). In FIG. 6, all video items, some video items, continuous capture items, and exit items are displayed. If the entire image item is selected, the entire screen being displayed is captured, and if the partial image item is selected, only some of the screens selected by the user are captured. If the continuous capture item is selected, a plurality of screens are captured during the period specified by the system or the user. If the exit item is selected, the capture function is terminated.

Since the number of menu items and the functions to be executed vary by the designer, the present invention will not be limited to the number and function of the menu items. Similarly, the position and shape of the menu item are displayed as one embodiment, and the shape or display position of the menu item may be changed by a designer, and thus the present invention is not limited to the example.

That is, when a capture command is input, a menu as shown in FIG. 6 is displayed, and a user captures an entire image currently displayed on the screen, a portion of the entire image, or a plurality of images using an input device such as a remote controller. It may be selected whether to capture an image (S503). The image selected in S503 is converted into a super resolution image (S504).

If the entire image is selected in S503, the entire image currently displayed on the screen is converted to the super resolution image, and if the partial region is selected from the entire image, the image of the partial region selected by the user is displayed. It is converted to super resolution video.

In this case, when some image items are selected by the user in S503, a cursor appears on the screen so that the user can select some desired images from all the images. In addition, in order to help the user to understand, using an input device such as a remote control to display a guide phrase to drag the area to be captured in one embodiment.

Then, the user can move the cursor to the start position of the image to be captured by using a remote controller and then select the area to be captured by dragging the screen.

The form of the drag may be a quadrangle, another form, for example, a circle or a triangle, or may be set by the user. According to the embodiment of the present invention, a rectangle is set as a default.

In this case, the position where the cursor is located becomes the start position of the area to be captured, that is, the upper left corner of the rectangle obtained by dragging. In this case, as the screen is dragged by using the remote controller or the like, the shape of the quadrangle becomes larger based on the start position, and thus the size of the capture area may be determined using the remote control.

When the captured image is selected through the above process, the position information of the captured image is output to the controller 100. The location information of the captured image may be a starting position of a rectangle obtained by dragging, that is, coordinate information of an upper left corner point, width information of the rectangle, height information of the rectangle, or each vertex of the rectangle. It may be coordinate information of.

The controller 100 controls the memory 109 so that only a part of the captured image of the current frame is output to the super resolution processor 111. The super resolution processor 111 converts the resolution of the partial image into super resolution by referring to the previous M frames and the subsequent N frames. In addition, some images converted to super resolution may be stored in a storage medium or displayed on a screen.

According to another embodiment of the present invention, even if some images of the entire image are selected as the captured images, the super resolution processing unit 111 outputs all current frames to perform super resolution conversion, and then stores the converted images in super resolution. When displaying, only the image corresponding to the captured image may be stored or displayed by referring to the location information of the captured image.

Meanwhile, when the continuous capture item is selected on the menu screen of FIG. 6, a guide bar as shown in FIG. 12 may be displayed to select a capture section. The user may designate a section to be captured by using a location indicator in the guide bar or by using a cursor or a key of an input device allocated for a designated section.

If continuous capture is selected while watching the image stored in the storage medium through the reproduction request, the capture section may be set to the previous and subsequent sections based on the currently displayed screen. However, if continuous capture is selected while watching an image received in real time, the previous image cannot be obtained, and thus, the capture section can be set only for the subsequent section. In an embodiment, the capture section is set using a guide bar. The continuous capture function is most effective when applied to an image reproduced by a time shift operation.

Alternatively, the capture device may automatically set a capture section. Depending on the type of the currently displayed image, it may be set from the previous M frame to the subsequent M frame based on the current frame, may be set to the next M frame based on the current frame, or may be set as time. In addition, it is possible to capture all frames from the previous M frame to the next M frame, or to capture only a certain period or a specific frame. For example, you can capture the screen you are watching once every second, or you can only capture the I picture. Since this is only one embodiment, the present invention will not be limited to the above-described embodiments.

When the captured image is converted into the super resolution image in S504, the user is asked whether to save the super resolution image through a menu screen (S505). In this case, if the user selects storage, the super resolution image is stored in the storage medium (S506).

Here, the menu screen may be configured to simply select whether or not to store the storage, or, if there are a plurality of storage media capable of storing the super resolution image, the user may select at least one of the plurality of storage media. If only the presence or absence of the storage is selected as in the former, the embodiment stores the super resolution image in a storage medium set as a default in the system. In another embodiment, the system may search for a storage medium that can be stored, select an available storage medium, store a super resolution image in the selected storage medium, and inform the storage medium information to a user. In another embodiment, the super resolution image may be automatically stored in the storage medium without asking the user whether it is stored.

In FIG. 7, the HDD and the USB memory are mounted as an external storage medium 121 on the outside of the display device, and the HDD is mounted as the internal storage medium 110 on the inside of the display device so that the user can select one of the storage media. Shows the menu configuration. That is, in FIG. 7, an external storage medium 1 item, an external storage medium 2 item, an internal storage medium item, and an exit item are displayed. In this case, each storage medium item may further display detailed information of the storage medium so that a user can easily recognize what the storage medium is. The capacity of each storage medium can also be displayed. For example, the total capacity, the capacity already used, and the remaining capacity (that is, the storage capacity) may be displayed for each storage medium.

FIG. 8 illustrates embodiments in which a total capacity, an already used capacity, and a storage capacity are displayed for any one of the storage media.

In a first embodiment of a method of displaying a capacity of a storage medium, a region may be visually illustrated by dividing an area into an already used space and a storage space in the figure. At this time, the figure may be a planar figure and a three-dimensional figure.

In a second embodiment of the method for displaying the capacity of a storage medium, the total storage space, the space already used, or the storage space can be represented in units of capacity. In this case, the total storage space capacity, the already used space capacity or the storage space capacity may be selectively indicated. That is, the storage capacity can be indicated by a combination that allows the user to know the capacity of the storage space.

In a third embodiment of a method of displaying a capacity of a storage medium, the total storage space and the storage space can be represented in units of time. In this case, the time of the entire storage space may be selectively indicated. Also, the recordable time may vary depending on the recording quality (quality). For example, it can be stored for 75 hours in the HD class, but can be stored for 100 hours in the SD class. By outputting each of these, the user can efficiently utilize the storage space by appropriately selecting the recording quality (image quality).

In a fourth embodiment of the method for displaying the capacity of a storage medium, the total storage space, already used space, or storage space may be represented in percentage units. In this case, the capacity percentage list may be combined and selectively displayed to allow a user to know a storage space.

On the other hand, in the menu configuration as shown in FIG. 7, if the user selects the external storage medium 1, the super resolution image is stored in the selected external storage medium 1. If the user selects the exit item, the super resolution image is not stored in any storage medium.

The super resolution image stored in the storage medium as described above may be later read and displayed on a screen, set as a background screen of the corresponding display device, or used as a background screen on another external device, for example, a PC or a mobile phone. That is, if the storage medium storing the super resolution image is an external storage medium, the external storage medium may be connected to an external device such as a PC or a mobile phone, and the super resolution image connected to the external storage medium may be read and set as a wallpaper. have.

For example, a user-produced UCC video or a video taken by a mobile phone has a low image quality and a small screen size. In this case, after displaying the UCC video, video taken by a mobile phone, etc. on the display device, at least one screen is converted into a super resolution video through a capture command to increase the image quality and to enlarge the screen size. If the super resolution image is stored in an external storage medium and the external storage medium is connected to a PC or a mobile phone, the super resolution image may be viewed on a PC or a mobile phone, or the super resolution image may be designated as a wallpaper. have.

On the other hand, if the storage is not selected in S505 or the storage of the super resolution image is completed in S506, the user is asked whether to display the super resolution image through a menu screen (S507). If the user selects a display at this time, the super resolution image is displayed on the screen (S508).

Here, the menu screen may be configured to simply select whether there is a display, or may display one or more display types, and when one of the user selects the display, a super resolution image may be displayed in the selected type.

If only the presence or absence of the display is selected as in the former, the display of the super resolution image by the display type set by default in the system is an embodiment. In another embodiment, the system may automatically display the super resolution image on the screen without asking for the display.

In FIG. 9, a display type includes a full screen, a picture in picture (PIP), a double window, and a menu configuration for allowing a user to select one of the display types. That is, in FIG. 9, the full screen item, the PIP item, the double window item, and the exit item are displayed. In this case, each display type item may further display, for example, an icon when an image is displayed in the corresponding display type to help the user understand. If the exit item is selected by the user at this time, the super resolution image is not displayed in any display type.

If a part of the entire screen is captured and converted into a super resolution image as shown in FIG. 10A, and the full screen item is selected in the menu configuration of FIG. 9, the super resolution image is shown in FIG. 10B. As shown in the full screen. At this time, if the size of the image converted to the super resolution is larger than the screen of the display device, it is not possible to display all the image converted to the super resolution on one screen. In this case, by receiving a direction signal by using a direction key of the remote control or a drag input of the touch screen, by moving the super-resolution image displayed on the screen in the corresponding direction, it is possible to see the part not currently displayed on the screen.

In the menu configuration of FIG. 9, when the PIP item is selected, an image currently being viewed is displayed on at least one of the main screen and the sub screen, and a super resolution image may be displayed on the remaining screen. In this case, the user can simultaneously view the image being watched and the captured image converted to super resolution.

For example, as shown in (c) of FIG. 10, the main screen continues to display a video being watched (for example, a broadcast being received in real time), and a sub-screen displays a super resolution video or vice versa. The super resolution image can be displayed, while the sub-screen can continuously display the image being viewed.

In the menu configuration of FIG. 9, the display method is the same as the PIP method even when a double window is selected. However, in the PIP method, a small subscreen exists in the main screen, whereas the double window divides one screen into left and right screens. In this case, the size of the left and right screens may or may not be the same.

If the size of the image converted to super resolution is larger than the screen of the display device even in the PIP or the double window method, and none of them are displayed on one screen, the direction signal is input by the direction keys of the remote control or the drag input of the touch screen. By moving the image displayed on the screen in the corresponding direction, it is possible to view the part not currently displayed on the screen.

When the super resolution image is displayed with the selected display type as described above, a guidance phrase such as 'set as wallpaper' may be displayed to set the super resolution image as a background screen of the corresponding display device.

In addition, when the size of the image converted to the super resolution is larger than the screen size currently being displayed, the size of the super resolution image may be resized to the screen size of the display device automatically or by the user's selection.

On the other hand, if the display device according to the present invention has a simultaneous picture function such as PIP or double window, the captured image converted to super resolution is automatically displayed on any one of the simultaneous picture without asking the user's selection, and the other The screen can display an image that the user was watching. In this case, the user can watch the broadcast on one screen while viewing the captured image on the other screen. If continuous capture is selected, continuous stills are displayed on one of the simultaneous screens.

In addition, the present invention may display the image of each step sequentially when the captured image is converted into a super resolution image. That is, when the captured image is converted into a super resolution image, since the size and resolution, that is, the depth, are different for each step, the resolution and size of the displayed image are different for each step as shown in FIG. 11.

In this case, the user may select one of each step by using an input device such as a remote controller, or may store the image of the selected step in a storage medium or use it as a background screen.

Alternatively, when a part of the image converted to the super resolution is selected, the part of the image may be converted back into the super resolution image. By repeating this process, you can easily identify areas that are shot too far and are too small or too dim to see.

In addition, assuming that the display device has a call function and that there is a phone number in the super resolution converted image, a call may be made by clicking the phone number. As another example, assuming that the display device is connected to the Internet network and there is a trade name in the image converted to the super resolution, when the trade name is clicked, the homepage of the clicked trade name is connected, and a purchase or the like is performed on the homepage. It can also be done.

The present invention described so far is not limited to the above-described embodiments, and can be modified by those skilled in the art as can be seen from the appended claims, and such modifications are the scope of the present invention. Belongs to.

It will be apparent to those skilled in the art that various modifications and variations can be made in the present invention without departing from the spirit or scope of the invention.

Therefore, the technical scope of the present invention should not be limited to the contents described in the embodiments, but should be defined by the claims.

1 is a block diagram showing an embodiment of a display device according to the present invention;

FIG. 2 is a detailed block diagram illustrating an embodiment of the super resolution processing unit of FIG. 1. FIG.

3 is a flowchart showing an embodiment of an image capture and a method for converting a captured image to super resolution according to the present invention;

4 is a detailed flowchart illustrating an embodiment of a method of converting a captured image into super resolution according to the present invention;

5 is a detailed flowchart illustrating one embodiment of an image capture and a method for converting a captured image into super resolution according to the present invention;

6 is a diagram illustrating an example of a menu screen for performing an image capture according to the present invention.

7 is a diagram illustrating an example of a menu screen for storing a super resolution image according to the present invention.

8 illustrates embodiments of a method for displaying a capacity of a storage medium storing a super resolution image according to the present invention.

9 illustrates an example of a menu screen for displaying a super resolution image according to the present invention.

10 is a view showing embodiments in which a super resolution image is displayed according to a display type according to the present invention.

11 is a diagram illustrating a process of converting a captured image into a super resolution image according to an embodiment of the present invention.

12 illustrates an embodiment of a guide bar for storage / playback and continuous capture in accordance with the present invention.

* Description of the symbols for the main parts of the drawings *

100 control unit 101 tuner unit

102: demodulation unit 103: demultiplexer

104: video decoder 105: display processing unit

106: audio decoder 108: external input processing unit

109: memory 110: internal storage medium

111: super resolution processing unit 112: user selection unit

113: output unit 114: storage / playback unit

120: internal bus 121: external storage medium

Claims (14)

While displaying an image on the screen of the display device, the previous M (M

Storing 0) frames; If a capture command is input while displaying the image, at least a portion of the entire screen being displayed is captured, and N (N) is based on a frame of the captured image.

Storing 0) frames; Upscaling the resolution of the captured image using at least one of the previous M / post N frames as a reference frame; Storing the upscaled image on a corresponding storage medium according to a storage command of the upscaled image; And And displaying the upscaled image on at least a portion of a screen in a corresponding display type according to a display command of the upscaled image. The method of claim 1, wherein the upscaling step And upscaling the captured image to a super resolution image. The method of claim 2, wherein the upscaling step Obtaining motion information between the captured image and a reference frame by performing motion estimation by receiving at least one of the captured image and the previous M / n after N frames of the captured image; Mapping motion information of each reference frame to super resolution coordinates; And And obtaining a pixel value of a super resolution image from pixel information of a reference frame mapped to the super resolution coordinates, and outputting a super resolution image. The method of claim 3, wherein the motion estimation step The image processing method of the display device, characterized in that for performing motion estimation up to a sub-pixel unit. The method of claim 3, wherein the outputting the super resolution image An image processing method of a display apparatus, characterized by obtaining pixel values of a super resolution image while removing blur and noise. The method of claim 1, The captured image is at least one of a broadcast received through at least one of a broadcasting network and a communication network, an image input from an external device through an external input terminal, and an image input from a storage medium. The method of claim 1, wherein the storing step Displaying storage medium information that can be stored in the display device on a menu screen according to a storage command of the upscaled image; And And storing the upscaled image on the storage medium selected by the user on the menu screen. The method of claim 1, wherein the displaying step Displaying at least one display type on a menu screen according to a display command of the upscaled image; And displaying the upscaled image according to the display type selected by the user on the menu screen. The method of claim 8, wherein the displaying step When the full screen is selected among the display types, the upscaled image is displayed on the entire screen, and when the simultaneous screen is selected, the upscaled image is displayed on the entire screen on any one of the main and sub-screens. Image processing method of display device. The method of claim 8, wherein the displaying step And setting the upscaled image as a background screen of the display device according to a background screen designation command of the upscaled image. The method of claim 1, wherein the capturing step Setting a continuous capture section including at least one of a time before / after the image being displayed when the capture command is input; And And continuously capturing an image according to a preset condition during the set continuous capture period. The method of claim 11, When displaying the image captured during the continuous capture period and converted to the super resolution on the screen, the audio associated with the image is output in the form of at least one of audio, text. An output unit configured to display at least one of a broadcast received through at least one of a broadcasting network and a communication network, an image input from an external device through an external input terminal, and an image input from a storage medium, and to be processed and inputted; At least one of a volatile internal storage medium, a nonvolatile internal storage medium, and a nonvolatile external storage medium, wherein the previous M (M) is based on a frame of the image being displayed while the image is displayed on the screen.

0) frames are saved, and if a capture command is input, images of at least some of the entire screens are captured, and N (N) is based on the frames of the captured images.

A storage unit storing 0) frames; And A super resolution processor configured to convert the resolution of the captured image into a super resolution image by using at least one of the previous M / n N frames as a reference frame, The storage unit stores the super resolution image in a corresponding storage medium according to a storage command of the super resolution image, and the output unit displays the super resolution image in at least a portion of a screen in a corresponding display type according to a display command of the super resolution image. Display device characterized in that for displaying. The method of claim 13, wherein the super resolution processing unit A motion estimator for receiving motion information between the captured image and the reference frame by performing motion estimation by receiving at least one of the captured image and the previous M / n after N frames of the captured image; An interpolator for mapping motion information of each reference frame to super resolution coordinates; And And an image reconstructor configured to obtain a pixel value of the super resolution image while removing blur and noise from pixel information of the reference frame mapped to the super resolution coordinates, and output a super resolution image.

Images