KR20080018396A - Computer-readable medium for recording mobile application and personal computer application for displaying display information of mobile communications terminal in external display device - Google Patents

Abstract

A computer-readable medium recording mobile application and PC application for displaying the display information of a mobile communication terminal on an external display device are provided to display mobile contents, being displayed on the mobile communication terminal, on the external display device in real time, thereby facilitating the development and production of the mobile contents and promoting effectively the mobile contents. A screen capture module(212) copies image data(230), displayed on the LCD(Liquid Crystal Display) of a mobile communication terminal, in a frame buffer, in which the image data resides, and alternately stores the image data in two buffers, and compares the image data stored in the two buffers to divide the image data into time-variant image data, image data which is changed in process of time, and time-invariant image data, image data which is not changed in process of time, and transmits only the time-variant image data. An image data converting/compressing module receives only the time-variant image data out of the image data from the screen capture module, and compresses the time-variant image data in a smaller size for the optimization of transmission speed in transmitting the time-variant image data with a PC(Personal Computer). An image data transmission module(216) transmits the compressed time-variant image data to the PC from the mobile communication terminal through USB(Universal Serial Bus) communication.

Description

Computer-Readable Medium for Recording Mobile Application and Personal Computer Application for Displaying display Information of Mobile Communications Terminal in External Display Device}

1 is a diagram schematically illustrating a system for displaying display information of a mobile communication terminal in real time on an external display device according to an embodiment of the present invention;

2 is a view briefly showing a mobile platform structure and a mobile application configuration of a mobile communication terminal according to an embodiment of the present invention;

3 is a diagram illustrating pixel-based reconstruction of a screen for data compression according to a preferred embodiment of the present invention;

4 is a view showing a frame structure, a tile structure and a cell structure according to a preferred embodiment of the present invention;

5 is a diagram illustrating a conversion and compression process of image data according to a preferred embodiment of the present invention;

6 is a diagram showing the configuration of a PC application according to a preferred embodiment of the present invention;

7 is a diagram illustrating an example of a UI of a PC application according to a preferred embodiment of the present invention;

8 is a diagram illustrating a blur and sharpening process of an image according to an exemplary embodiment of the present invention.

<Description of Symbols for Main Parts of Drawings>

110: mobile communication terminal 120: PC

130: external display device 140: Internet

210: mobile application 212: screen capture module

214: image data conversion / compression module 216: image data transmission module

220: mobile platform 230: mobile content

510: PC application 512: image data receiving module

514: display module 516: control module

The present invention relates to a computer-readable recording medium recording a mobile application and a PC application for displaying display information of a mobile communication terminal on an external display device. In more detail, the configuration, execution state, and operation state of the mobile content running on the mobile communication terminal may be expanded / expressed in real time on an external display device such as a plasma display panel (PDP), an LCD, a television (TeleVision), or the like. The present invention relates to a computer-readable recording medium recording a mobile application and a PC application for generating and storing a moving image of mobile content running on a mobile communication terminal.

Mobile content is a combination of wireless communication and digital content, and the mobile content industry can be defined as producing new added value through mobile content. In other words, movies, games, information contents, and educational contents are digitized, distributed and serviced through wireless communication networks, not wired, and are emerging as the core of the ubiquitous environment where people can listen to the desired information anytime and anywhere. In addition, content that is distributed using digital satellite broadcasting, digital multimedia broadcasting (DMB) services, and the like also belongs to mobile content.

Although mobile contents fall under the category of digital contents, they show different characteristics from existing wired internet-oriented digital contents due to characteristics of mobility, location-based, customer discrimination, and personality of wireless internet.

Mobile contents can be classified into four categories: first, information-based services such as news, weather, and location information, and second, ringtones, character downloads, movies, VOD (Video On Demand), MOD (Music On Demand), etc. Is an entertainment service. The third is a mobile game service using a mobile communication terminal, and the fourth is a communication service for sending various messages and emails using the mobile communication terminal.

In the conventional planning and development of such mobile contents, in the development situation, the development is carried out using software called an emulator, which is a PC program, so that it is possible to work on a general PC screen without any inconvenience. However, since the "emulator" is similar to the mobile terminal environment and is not a real mobile terminal, it can only confirm whether or not the developed program is executed, and whether or not it operates on software when mobile contents are applied to the actual mobile terminal. , The compatibility with the hardware, and whether there is a problem in the execution of the mobile content, etc. are to be tested directly by looking at the LCD (Liquid Crystal Display) screen of the mobile communication terminal of about 1 to 2 inches. After all, the development of mobile content is a very inconvenient task that can only be done while viewing a small screen size of about 1 to 2 inches.

In addition, even in the promotion of mobile content, since the LCD screen of the mobile communication terminal running the mobile content is shown to the customer, there is a problem in that it is impossible to simultaneously demonstrate the mobile content to a large number of customers. Since there is no system for providing a video or a real-time enlarged screen for the mobile content, there is a problem in that it is not possible to properly promote the mobile content.

In order to solve the above-described problems, the present invention, the configuration, execution state and the state of the operation implemented in the mobile content running on the mobile communication terminal in an external display device such as plasma display panel (PDP), LCD, TV (TeleVision) An object of the present invention is to provide a computer-readable recording medium recording mobile applications and PC applications for real-time zooming / displaying or generating and storing moving images of mobile contents running on a mobile communication terminal.

In order to achieve the above object, the present invention, in the mobile application of the mobile communication terminal in conjunction with a personal computer (PC) for controlling to display the image being displayed in the mobile communication terminal on the external display device, Copy the image data from the frame buffer where the image data displayed on the LCD (Liquid Crystal Display) reside and alternately store them in two preset buffers, and simultaneously compare the image data stored in the two buffers. A screen capture module that separates time-varying image data that is image data that changes over time and time-varying image data that is image data that does not change over time, and transmits only the time-varying image data; Receiving only the time-varying image data of the image data from the screen capture module, and when transmitting the time-varying image data to the PC, image data conversion to compress the time-varying image data to 1/4 size for optimization of the transmission speed / Compression module; And image data transmission module for transmitting the time-varying image data compressed to 1/4 to the PC through Universal Serial Bus (USB) communication. A computer readable recording medium having recorded thereon a mobile application for display is provided.

In addition, according to another object of the present invention, a PC for managing to display the compressed image data from the mobile communication terminal to display on an external display device including a plasma display panel (PDP), LCD and CRT (Cathode-Ray Tube) ( Personal computer),

An image data receiving module configured to receive the image data from the mobile communication terminal by performing a universal serial bus (USB) communication with the mobile communication terminal;

Receives the image data from the image data receiving module, converts the YUV method of the image data into RGB image data, enlarges the image by using a bilinear interpolation method, blurs the filter and sharpens the image. A display module for performing image processing using at least one of the filters, and then transmitting the RGB image data to the external display device to display an image being displayed on the mobile communication terminal in real time on the external display device; And

Controls whether or not to execute the image processing performed by the display module according to each command input by a user through a UI (User Interface) of the PC application, and is image data that captures an image being displayed on the external display device. Stores captured image data, connects all the captured image data through the command, or creates a video by using a method of inserting and missing part of the captured image data, and receiving the command to display on the external display device. It provides a computer-readable recording medium recording a PC application for displaying the display information of the mobile communication terminal on an external display device, characterized in that it comprises a control module for recording the image being in real time.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. First of all, in adding reference numerals to the components of each drawing, it should be noted that the same reference numerals are used as much as possible even if displayed on different drawings. In addition, in describing the present invention, when it is determined that the detailed description of the related well-known configuration or function may obscure the gist of the present invention, the detailed description thereof will be omitted.

1 is a diagram schematically illustrating a system for displaying display information of a mobile communication terminal in real time on an external display device according to an exemplary embodiment of the present invention.

A system for displaying display information of a mobile communication terminal in real time on an external display device according to a preferred embodiment of the present invention includes a mobile communication terminal 110, a PC 120, an external display device 130, the Internet 140, and the like. It includes.

The mobile communication terminal 110 is a terminal which is connected to the mobile communication network through a wireless access network (not shown) to perform a normal voice call by wireless communication with different mobile communication terminals, and includes a KVM (Kilobyte Virtual Machine) and SK-VM ( It is a terminal equipped with a mobile platform including any one of SK-Virtual Machine (GVM), General Virtual Machine (GVM), Binary Run-time Enveronment for Wireless (BREW), and Wireless Internet Platform for Interoperability (WIPI).

In the present invention, the mobile communication terminal 110 stores display information displayed by the mobile communication terminal 110, that is, image data captured in real time, and stores the data format of the stored image data in RGB (Red). It is equipped with a mobile application that converts, compresses, and transmits to YUV (Y: luminance signal, U: luminance signal and red component difference, V: luminance signal and blue component) from Green-Blue) to PC 120. . Here, the image includes either a moving image or a still image.

Detailed description of the mobile application according to the preferred embodiment of the present invention will be made with reference to FIGS.

In the present invention, the mobile communication terminal 110 is interlocked with the PC 120 through USB (Universal Serial Bus) communication, and the image data of the image being displayed on the mobile communication terminal 110 through the mobile application to the PC 120 send.

The PC 120 is a personal computer including any one of a desktop computer and a notebook computer. In the present invention, the mobile terminal 110 receives the converted and compressed image data from the YUV and converts the image data format into RGB. In addition, a PC application for filtering the image data converted into RGB, enlarging the resolution of the image data and transmitting the image data to the external display device 130 or registering the image data in a web page through the Internet 140 may be installed. A detailed description of a PC application according to a preferred embodiment of the present invention will be made with reference to FIGS. 6 to 8.

In the present invention, the PC 120 interworks with the mobile communication terminal 110 by USB communication, converts the data format of the image data transmitted from the mobile communication terminal 110 from YUV to RGB, and converts the image data into RGB. The image data is filtered and the resolution of the image data is enlarged and transmitted to the external display device 130 or the image data is registered in the web page through the Internet 140.

The external display device 130 is a display device including a plasma display panel (PDP), an LCD and a cathode-ray tube (CRT). The external display device 130 receives image data from the PC 120 and is being displayed on the mobile communication terminal 110. Display images in real time.

2 is a diagram schematically illustrating a mobile platform structure and a mobile application of a mobile communication terminal according to a preferred embodiment of the present invention.

In the present invention, the mobile platform includes KVM, GVM, BREW, WIPI, and WI-TOP, but for convenience of description, the description is limited to WIPI and WI-TOP. In addition, since the mobile platform structure in the present invention is a known technology, a separate description of the mobile platform structure will be omitted.

In the present invention, the mobile application 210 is a screen capture module 212, an image data conversion / compression module 214 and an image data transmission module 216 for capturing display information displayed on the mobile communication terminal 110, that is, image data ), And operates on the upper portion of the mobile platform 220 and the lower portion of the mobile content 230 including games, photos, and the like.

In the mobile application 210 of the present invention, the screen capture module 212 and the image data transmission module 216 are implemented by an API (Application Process Invocation) function provided by WIPI. In the present invention, since the mobile platform is limited to WIPI and WI-TOP for convenience of description, it is limited to using the API provided by WIPI. However, when the present invention is actually applied, the mobile platform includes KVM, GVM and BREW. Among the API functions according to the present invention, the screen capture module 212 and the image data transmission module 216 may be implemented using API functions that perform the same functions as the API functions provided by the present invention.

The screen capture module 212 according to the present invention copies image data from a frame buffer in which image data displayed on the LCD screen of the mobile communication terminal 110 resides, and alternately stores the image data in two preset buffers. The image data stored in the two buffers are transmitted to the image data conversion / compression module 214. Here, the screen capture module 212 performs a function of obtaining color data of entire pixels of the screen from the frame buffer, that is, copying image data using an API function provided by WIPI called "MC_grpGetRGBPixels".

When sending the image data to the image data conversion / compression module 214, the screen capture module 212 compares the image data stored in the two buffers with time-varying image data, which is image data that changes over time, and does not change over time. Time-varying image data, which are non-image data, is separated, and only time-varying image data is transmitted to the image data conversion / compression module 214. Here, the time-varying image data and the time-invariant image data are divided into tile units. The tile will be described in detail with reference to FIG. 3A.

 The image data conversion / compression module 214 receives only the time-varying image data from the image data of the full screen from the screen capturing module 212, and transmits the time-varying image data to the PC 120 in order to optimize the transmission speed. Compress the data to 1/4 size.

The reason why the time-varying image data is compressed to 1/4 size in the present invention is that when the mobile application 210 transmits the image data from the mobile terminal 110 to the PC 120 via USB communication, the amount of image data to be transmitted is mobile communication. This is because the amount of data that can be transmitted from the terminal 110 to the USB communication. Detailed description thereof is as follows.

When transferring image data from the mobile communication terminal 110 to the PC 120 via USB communication, the premise of the assumption is first, that the resolution of the screen is 240 × 320 pixels for the currently used mobile communication terminal 110 ( Pixel), and since the data of each pixel is 24 bits, the amount of data to be transmitted in one frame is 230 kbytes.

Second, in the case of the "MC_srlWrite" function provided by WIPI, only experimental values of 65, 535 bytes can be transmitted at a time.

Third, the image data displayed on the mobile communication terminal 110 is usually about 5 to 10 frames per second, at least about 3 to 5 frames per second can be normal demonstration.

Fourth, the USB standard of the mobile communication terminal 110 that is currently used is USB 1.1, USB 1.1 has a maximum transmission speed of 12 Mbps. In practice, however, the maximum transmission speed is not satisfied, and in particular, since the central processing unit (CPU) of the mobile communication terminal 110 transmits other tasks in parallel, it is actually about 0.8 to 2.4 Mbps, that is, about 100 to 300 kbytes per second. Is possible.

Combining the above cases, in order to transmit 3 to 5 frames per second within 100 to 300 kbytes, the size of the captured image data should be about 20 to 100 kbytes to enable real-time demonstration on the external display device 130. The conclusion is that the time-varying image data is compressed to a quarter size.

A method of compressing time-varying image data to 1/4 size will be described with reference to FIGS. 3 and 4.

First, FIG. 3 is a diagram illustrating pixel reconstruction of a screen for data compression according to an exemplary embodiment of the present invention.

First, as shown in FIG. 3A, a tile having a 16 × 16 pixel size is used for the entire screen having a size of 240 × 320 pixels, as shown in FIG. 3A, without using the pixel data of the screen of the mobile communication terminal 110 to compress time-varying image data. Tile is divided into 15 × 20, or 300 tiles. The tile is again defined as a cell having a size of 4x4 pixels, and configured to be divided into 4x4, that is, 16 cells.

4 is a view showing a frame structure, a tile structure and a cell structure according to a preferred embodiment of the present invention.

4A shows the frame structure. In the frame structure, the header of 6 Bytes is the frame ID (IDentification), which is the frame separator of 2 Bytes, debug (Debug) separator of the 2 Bytes, and the number of transport tiles of 2 Bytes. The size of a tile is 192 Bytes.

4B shows a tile structure in which the tile number occupies 2 Bytes and the size per cell is 12 Bytes.

4C shows a cell structure in which each pixel has a size of 3 Bytes, which is determined by an RGB to YUV conversion and compression method in the image data conversion / compression module 214. Here, the YUV method is generally used in data transmission because it can express a color similar to the RGB method with smaller data than the RGB method, and the YUV color model is also called YCbCr, and Y: Luminance and Cb. : Blueness, Cr: Redness.

5 is a diagram illustrating a conversion and compression process of image data according to an exemplary embodiment of the present invention.

The YUV method is a method in which one U and V component is allocated to four Y components, and the conversion relationship with the RGB method is expressed by Equation 1.

Y = 0.3R + 0.59G + 0.11B

U = (B-Y) × 0.493

V = (R-Y) × 0.577

Since the data modified in the YUV method has one U and V component for four Y components of each pixel, the size of the data is reduced from 12 Bytes (RGB method) to 6 Bytes (YUV conversion). The reason for assigning one U and V component to the four pixel data is that the color component does not have a large difference with respect to four pixels that are very adjacent to the brightness component.

In order to compress the time-varying image data compressed from 12 Bytes to 6 Bytes (1/2) to 1/2 again, the data of a 4-pixel cell is transformed as follows.

The Y, U, and V components are data each having a size of 1 Bytes, which is 8 bits of data, of which 4 bits are forcibly converted to 0. Performing such a forced compression method causes a loss of data, so that a slight change in color occurs when time-varying image data is displayed on the external display device 130, but the change does not significantly affect the entire screen.

When the Y, U, and V components are respectively compressed into 4 bits, the size of six data sets of four Y components and one U and V component is 3 Bytes. That is, 1/4 of 12 Bytes.

When the time-varying image data compressed to 1/4 is transmitted, the screen size of the mobile communication terminal 110 is 240 × 320 × 6 ÷ 8 = 57600 Bytes for one frame.

When the image data conversion / compression module 214 compresses the time-varying image data from 12 Bytes to 3 Bytes, the image data transmission module 216 uses the "MC_srlWrite" function provided by the WIPI API to convert the time-varying image data. Transfer from the mobile communication terminal 110 to the PC 120 by USB communication.

As described above, the mobile application 210 including the screen capture module 212, the image data conversion / compression module 214, and the image data transmission module 216 is an image of an image displayed by the mobile communication terminal 110. In order to properly transmit the data to the PC 120, only the time-varying image data of the image data is converted and compressed and transmitted to the PC 120.

On the other hand, if the mobile platform 220 is a WI-TOP platform, when the WI-TOP is mounted on the mobile communication terminal 110, the capacity of the image data is much smaller than when the WIPI is mounted. Since the screen resolution of the mobile communication terminal 110 equipped with WI-TOP is only 1/4 compared with that of the WIPI, the amount of data is only about 1/4.

Therefore, when the mobile platform 220 is a WI-TOP platform, the image data is transmitted to the PC 120 without conversion and compression.

6 is a diagram showing the configuration of a PC application according to a preferred embodiment of the present invention.

The PC application 610 according to the preferred embodiment of the present invention includes an image data receiving module 612, a display module 614, and a control module 616.

The image data receiving module 612 may perform USB communication with the mobile communication terminal 110 to receive time-varying image data compressed to 1/4 from the mobile communication terminal 110 and transmit the data to the display module 614. As described above, the time-varying image data is image data in which data in units of tiles is changed over time among the image data of the entire screen of the mobile communication terminal 110.

The display module 614 receives quarterly compressed time varying image data from the image data receiving module 612 and transmits the 1/4 compressed time varying image data to the external display device 130. Convert time-varying image data of the YUV method compressed to 1/4 to image data of the RGB method. Here, the conversion process from the YUV method to the RGB method may be performed by performing the reverse process of the RGB to YUV conversion and compression process in the mobile application 210.

The display module 614 converting the time-varying image data to the RGB method performs image processing of the time-varying image data converted to the RGB method as follows. Here, image processing is a processing step of processing an image more beautifully.

First, the display module 614 enlarges an image of time-varying image data converted by the RGB method. In this case, a method for performing image magnification is performed by using a bilinear interpolation method.

The reason for using the fold linear interpolation method in the present invention is that the resolution of the mobile communication terminal 110 is often 240 × 320, and the resolution of the external display device 130 is 1024 × 768. This is because when the time-varying image data transmitted by 2 times is enlarged, an image suitable for the resolution of the external display device 130 can be displayed.

In addition, the display module 614 smoothly processes the original image of the time-varying image data converted in the RGB method using a blur filter, and sharpens the image using the sharpening filter. Here, the blur filter is a filter that blurs the focus of the image to reduce the blemishes of the image and eliminates the pixel cracking that occurs when the image is stretched. The smart blur filter, the radial blur filter, and the Gaussian ( And one of a Gaussian blur filter and a motion blur filter. On the other hand, the sharpening filter in the present invention uses a high-boost filter. An example of such blurring and sharpening will be described with reference to FIG. 8.

After the image processing is completed, the display module 612 transmits the time-varying image data converted by the RGB method to the external display device 130, so that the image displayed by the external display device 130 on the mobile communication terminal 110 in real time. Display it.

The control module 616 receives a user's command to control whether the blur processing and sharpening processing performed by the display module 612 are executed, and receives an image of the image being displayed on the external display device 130 by receiving the user's command. The captured image data is stored as captured image data. In this case, the control module 616 may connect the pre-stored captured image data through a command input by a user to produce a video, or may produce a video by a method of partially inserting and missing a portion of the captured image data.

In addition, the control module 616 receives a user's command to record the video being displayed on the external display device 130 in real time.

In addition, the control module 616 includes various background picture data. For the remaining part of the screen of the external display device 130 that does not display an image, any one of the various background picture data is input by a user input command. By transmitting to the external display device 130, the external display device 130 displays the background picture to the remaining portion.

As described above, the control module 616 receives a user's command and performs its function. Here, the user's command is input through the UI (User Interface).

7 is a diagram exemplarily illustrating a UI of a PC application according to an exemplary embodiment of the present invention.

The UI in the present invention includes an image processing selection portion, a video production and editing selection portion through image capture and image combination, a background picture selection and termination selection portion, and a video production and reproduction selection portion.

8 is a diagram illustrating a blur and sharpening process of an image according to an exemplary embodiment of the present invention.

8A is an image before performing the blur and sharpening processing, 8B is an image after performing the blur processing, and 8C is an image after performing the sharpening processing.

The above description is merely illustrative of the technical idea of the present invention, and those skilled in the art to which the present invention pertains may make various modifications and changes without departing from the essential characteristics of the present invention. Therefore, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention but to describe the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments. The protection scope of the present invention should be interpreted by the following claims, and all technical ideas within the equivalent scope should be interpreted as being included in the scope of the present invention.

As described above, according to the present invention, since the mobile content being displayed on the mobile communication terminal can be displayed in real time on an external display device, the mobile content can be easily developed and produced, and the effective promotion of the mobile content can be achieved. There is.

Claims (5)

In the mobile application of the mobile communication terminal in conjunction with a personal computer (PC) for controlling to display the image being displayed on the mobile communication terminal to the external display device, Copying the image data from the frame buffer in which the image data displayed on the liquid crystal display (LCD) of the mobile communication terminal resides, alternately storing the image data in two preset buffers, and simultaneously storing the image data in the two buffers. A screen capture module configured to compare data to separate time-varying image data that is image data that changes over time and time-varying image data that is image data that does not change over time, and to transmit only the time-varying image data; Image data conversion / compression that receives only the time-varying image data among the image data from the screen capture module and compresses the time-varying image data to a smaller size for optimizing a transmission speed when transmitting the time-varying image data to the PC. module; And Display information of the mobile communication terminal to the external display device, characterized in that it comprises an image data transmission module for transmitting the time-varying image data compressed to a smaller size from the mobile communication terminal to the PC by USB (Universal Serial Bus) communication A computer readable recording medium recording a mobile application for display. The method of claim 1, The image data conversion / compression module converts the red-green-blue (RGB) format of the time-varying image data into a YUV format and converts 8-bit Y, U, and V components into 4 bits, respectively, to convert the time-varying image data. A computer-readable recording medium recording a mobile application for displaying display information of a mobile communication terminal on an external display device, characterized in that the compression is to a quarter size. In a PC application of a personal computer (PC) which receives compressed image data from a mobile communication terminal and manages to display it on an external display device including a plasma display panel (PDP), an LCD, and a cathode-ray tube (CRT), An image data receiving module configured to receive the image data from the mobile communication terminal by performing a universal serial bus (USB) communication with the mobile communication terminal; Receives the image data from the image data receiving module, converts the YUV method of the image data into RGB image data, enlarges the image by using a bilinear interpolation method, blurs the filter and sharpens the image. After performing image processing using one or more of the filters, the display module for transmitting the RGB image data to the external display device to display the image being displayed in the mobile communication terminal in real time on the external display device ; And Controls whether or not to execute the image processing performed by the display module according to each command input by a user through a UI (User Interface) of the PC application, and is image data that captures an image being displayed on the external display device. Stores captured image data, connects all the captured image data through the command, or creates a video by using a method of inserting and missing part of the captured image data, and receiving the command to display on the external display device. Control module for recording the image being in real time And a PC application for displaying the display information of the mobile communication terminal on the external display device. The method of claim 3, wherein The image data is a computer-readable recording recording a PC application for displaying the display information of the mobile communication terminal on the external display device, characterized in that the time-varying image that changes over time of the image data of the full screen of the mobile communication terminal media. The method of claim 3, wherein The control module includes various background picture data, and transmits any one of the various background picture data to the external display device by the command for the remaining part of the screen of the external display device that does not display the image. And a PC application for displaying the display information of the mobile communication terminal on the external display device, wherein the external display device displays a background picture on the remaining portion.

Images