JP2016057630A - Screen transfer device, and screen transfer method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To satisfy need for transferring a screen, which is displayed on a mobile device, to another device and displaying the screen.SOLUTION: A system interconnect 40 interconnects a main memory 50, a display controller 60, and a dedicated memory 70. The display controller 60 reads frame data from a frame buffer 52 provided in the main memory 50 and generates a screen output image for displaying on a display. A path is provided for: taking out the screen output image from the display controller 60 as a transfer screen image for transferring the screen output image to another device; and writing the screen output image back to the dedicated memory 70 without going through the system interconnect 40.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for transferring a screen image.

  High-quality images such as playing games and simulations using high-quality 3D computer graphics and playing video content that combines live-action and computer graphics on personal computers and game machines The use of graphics is spreading.

  On the other hand, the display performance of a display panel of a mobile device such as a smartphone or a tablet terminal is high, and high-quality computer graphics and video are also used in the mobile device.

  Since the screen of a mobile device is generally small, there is a need to display the screen of the mobile device on a large screen such as a monitor of a television or a stationary game machine. There is also a need to connect a mobile device to other devices so that family members and friends can view the same screen, or share a screen among multiple users and enjoy applications such as games.

  The present invention has been made in view of these problems, and an object of the present invention is to provide a technique for transferring and displaying a screen image generated by a certain device to another device.

  In order to solve the above-described problem, a display controller according to an aspect of the present invention is a display controller that reads frame data from a frame buffer and generates a screen output image for display on a display. There is provided a path for taking out as a transfer screen image for transfer to the device and writing it to a dedicated memory provided separately from the frame buffer.

  Another aspect of the present invention is a screen transfer apparatus. The apparatus includes a main memory provided with a frame buffer, a display controller that reads out frame data from the frame buffer and generates a screen output image for display on a display, and a dedicated memory provided separately from the main memory; And a system interconnect interconnecting the main memory, the display controller and the dedicated memory. A path for taking out the screen output image from the display controller as a transfer screen image for transferring to another device and writing it back to the dedicated memory without going through the system interconnect is provided.

  Yet another embodiment of the present invention is also a screen transfer device. The apparatus includes a main memory provided with a frame buffer, a display controller that reads frame data from the frame buffer and generates a screen output image for display on the display, and the screen output image generated by the display controller. A dedicated memory for storing a transfer screen image for transfer to the device, and an encoder for compressing and encoding the transfer screen image. The encoder compresses and encodes the transfer screen image without performing inter-frame prediction.

  Yet another embodiment of the present invention is a screen transfer method. This method is a method for transferring a screen image in a device in which a main memory, a display controller, and a dedicated memory are interconnected by a system interconnect, in which the display controller receives frame data from a frame buffer provided in the main memory. Reading and generating a screen output image for display on a display; and taking out the screen output image from the display controller as a transfer screen image for transferring to another device, without passing through the system interconnect Writing back to the dedicated memory; and compressing and encoding the transfer screen image written back to the dedicated memory and transferring the image to the other device.

  It should be noted that any combination of the above-described constituent elements and the expression of the present invention converted between a method, an apparatus, a system, a computer program, a data structure, a recording medium, and the like are also effective as an aspect of the present invention.

  According to the present invention, a screen image generated by a certain device can be efficiently transferred to another device and displayed.

It is a block diagram of a system for transferring a screen from a mobile device to a TV / monitor. It is a block diagram of a system that transmits a screen from a mobile device to a stationary device. It is a block diagram of the system which transmits a screen from another mobile device to another mobile device. It is a block diagram of a system in which a distribution platform transfers a screen to a cloud service client via a cloud server. It is a block diagram of a system for uploading screen data from a mobile device to a video sharing service. It is a block diagram of a mobile device. It is a figure explaining the example of mounting of the screen transfer function in the system LSI of FIG. It is a figure explaining another example of mounting of the screen transfer function in the system LSI of FIG. It is a figure explaining the structure which shares the exclusive memory of FIG. 2 between a screen transfer process and a downward compatibility process. It is a block diagram of the display controller of FIG. FIG. 5A to FIG. 5C are diagrams for explaining screen images taken out from the display controller using the transfer write-back path. It is a figure explaining the structure of the display controller provided with the image output of 2 channels. FIG. 7A to FIG. 7C are diagrams for explaining an application example of a display controller with two output channels. FIG. 8A to FIG. 8C are diagrams illustrating another application example of the display controller with two output channels. FIG. 9A to FIG. 9C are diagrams illustrating still another application example of the display controller with two output channels. It is a flowchart explaining the procedure of the software handshake between a screen transmission system and a screen reception system.

  The screen transfer system according to the embodiment transfers a screen from a device in real time to another device. Further, when the user gives some operation input to the transfer destination device, the operation content is transferred to the transfer source screen transfer system, and an update screen reflecting the operation content is transferred to the transfer destination device. Several configuration examples of the screen transfer system according to the embodiment will be described with reference to FIGS. 1A to 1E.

  FIG. 1A is a configuration diagram of a system for transferring a screen from the mobile device 100 to the TV / monitor 300.

  Mobile device 100 and TV / monitor 300 are connected via a network such as a wireless LAN. The mobile device 100 is a portable device such as a portable game machine, a smartphone, or a tablet as an example, and has a screen transfer function 200 described later.

  The screen transfer function 200 compresses and encodes a screen generated by the display controller of the mobile device 100 (not limited to a screen displayed on the display of the mobile device 100) by a moving image compression encoding method such as MPEG-4 AVC. The data is transferred to the TV / monitor 300 via a network such as a wireless LAN.

  The TV / monitor 300 has an AVC decoder 400, decodes the compressed screen data received from the mobile device 100, and displays it on the large screen display. Thereby, the image viewed on the small screen of the mobile device 100 can be displayed on the large screen of the TV / monitor 300.

  FIG. 1B is a configuration diagram of a system that transmits a screen from the mobile device 100 to the stationary device 310.

  The mobile device 100 and the stationary device 310 are connected by a network such as a wireless LAN. The screen transfer function 200 of the mobile device 100 compresses and encodes the screen generated by the display controller, and transfers it to the stationary device 310 via a network such as a wireless LAN. The stationary device 310 is, for example, a stationary game machine, a television receiver, a recorder, a set top box for CATV (cable television).

  The stationary device 310 includes an AVC decoder 400, decodes the compressed screen data received from the mobile device 100, and transmits to the stationary device 310 an HDMI (High Definition Multimedia Interface) (registered trademark) or MHL (Mobile High-definition Link). ) Or the like for display on the connected TV / monitor 300. Thereby, the image viewed on the small screen of the mobile device 100 can be displayed on the large screen of the TV / monitor 300. In addition, if the stationary device 310 has a recording function, the video transferred from the mobile device 100 can be recorded.

  FIG. 1C is a configuration diagram of a system that transmits a screen from the mobile device 100 to another mobile device 110.

  The first mobile device 100 and the second mobile device 110 are connected by a network such as a wireless LAN. The first mobile device 100 has a screen transfer function 200, and the second mobile device 110 has an AVC decoder 400. The screen transfer function 200 compresses and encodes the screen displayed on the display of the first mobile device 100, and transfers it to the second mobile device 110 via a network such as a wireless LAN. The second mobile device 110 decodes the compressed screen data received from the first mobile device 100 and displays it on the display panel.

  Furthermore, the second mobile device 110 receives an input from the user and transmits the input content to the first mobile device 100. The first mobile device 100 generates a screen reflecting the user input received from the second mobile device 110 and transfers the screen to the second mobile device 110. Thereby, in the case of a competitive game, a game screen is shared between the first mobile device 100 and the second mobile device 110, and a plurality of users can participate in the game. In addition, by sharing a screen between the first mobile device 100 and the second mobile device 110, a plurality of users can have a net meeting.

  FIG. 1D is a configuration diagram of a system in which the distribution platform 502 transfers a screen to the cloud service client 510 via the cloud server 500.

  The distribution platform 502 has a screen transfer function 200 and is connected to the cloud server 500 via an interface such as PCI Express (PCIe) or USB. The screen transfer function 200 compresses and encodes a screen desired to be transferred to the cloud service client 510 and supplies it to the cloud server 500. The cloud server 500 transmits the compressed screen data to the cloud service client 510 via the Internet.

  The cloud service client 510 has an AVC decoder 400, decodes the compressed screen data, and displays it on the display. Further, the cloud service client 510 receives an input from the user and transmits the input content to the cloud server 500. The distribution platform 502 generates a screen reflecting the input contents of the user received by the cloud server 500, compresses and encodes the screen, and supplies it to the cloud service client 510. The cloud service client 510 receives from the cloud service client 510 a screen that reflects the input content of the user.

  FIG. 1E is a configuration diagram of a system for uploading screen data from the mobile device 100 to the video sharing service 600.

  The mobile device 100 has a screen transfer function 200, compresses and encodes the screen displayed on the display, and transmits the compressed image to the video sharing service 600 via the Internet. As a result, the screen displayed on the display of the mobile device 100 is uploaded to the video sharing service 600 as a video.

  The transmitting-side mobile device 100 having the screen transfer function 200 and the distribution platform 502 described in the configuration examples of FIGS. 1A to 1E are collectively referred to as a “screen transmission system”. The receiving TV / monitor 300 having the AVC decoder 400, the stationary device 310, the mobile device 110, the cloud service client 510, and the like are collectively referred to as a “screen receiving system”. Hereinafter, the configuration and processing procedure will be described by taking the mobile device 100 as an example of the “screen transmission system” and the TV / monitor 300 as an example of the “screen reception system”, but the “screen transmission system” and “screen reception” will be described. Examples of “system” are not limited to these.

  FIG. 2 is a configuration diagram of the mobile device 100. Mobile device 100 includes a system LSI 270. The system LSI 270 includes a CPU 10, a graphics processing unit (GPU) 20, a peripheral 30, a main memory 50, a display controller (DISPC) 60, a dedicated memory 70, an AVC encoder 80, and an I / O 90. These components are connected to each other by a system interconnect 40.

  The CPU 10 executes an application and controls each configuration of the system LSI 270. The GPU 20 executes graphics processing, generates a moving image, and stores it in the frame buffer 52 of the main memory 50. The peripheral 30 controls various peripheral devices.

  The system interconnect 40 is a connection for connecting system components to each other, and has an arbitrary physical form such as a shared bus, a crossbar switch, a ring bus, a tree-like network, and a data transmission method is a handshake method or a packet-based network. Arbitrary methods can be adopted, and these are exemplifications, and the physical form and data transfer method of the system interconnect 40 are not limited to those described here.

  The mobile device 100 includes a touch panel 280 including an LCD (Liquid Crystal Display) / OLED (Organic Light Emitting Diode) panel 230 and a touch pad 240.

  The main memory 50 includes a plurality of frame buffers 52 that store frames of moving images. The DISPC 60 reads and converts the images stored in the plurality of frame buffers 52 in units of lines, generates a final screen image at a constant refresh rate, and outputs it to the LCD / OLED panel 230. As an option, the DISPC 60 can output the final screen image to the HDMI / MHL connector 260 and display the screen on an external display connected by HDMI / MHL.

  The DISPC 60 according to the present embodiment is provided with a dedicated path (referred to as “transfer write-back path”) for taking out the generated screen image as a transfer screen image. The DISPC 60 outputs, as an initiator (bus master), a transfer screen image to the dedicated path or the system interconnect 40 via the transfer write-back path, and the transfer screen image is provided exclusively for the main memory 50. It can be written back to the memory 70.

  The AVC encoder 80 reads the transfer screen image stored in the dedicated memory 70, compresses and encodes it using a moving image compression encoding method such as MPEG-4 AVC, and writes the compressed and encoded screen data to the dedicated memory 70.

  Here, in MPEG-4 AVC, inter-frame prediction is normally performed using an I frame (Intra-coded Frame), a P frame (Predicted Frame), and a B frame (Bi-directional Predicted Frame). The AVC encoder 80 may compress and encode an image using only I frames without performing inter-frame prediction in order to ensure real-time screen transfer.

  When inter-frame prediction is performed, the amount of data after compression can be reduced and the transfer bandwidth can be suppressed. However, it takes a long time to perform prediction, and the latency of screen transfer increases. Therefore, in the AVC encoder 80 according to the present embodiment, in view of the fact that connection between devices such as WLAN and Bluetooth (trademark) has been speeded up, priority is given to reducing latency at the expense of compression efficiency. A method of compressing and encoding an image with only I frames was used without performing prediction. When there is a restriction on the bandwidth of the connection between devices, the image quality is lowered, but the required bandwidth can be reduced by increasing the compression rate so that the data amount is equivalent to the inter-frame prediction.

  Note that, in applications where latency is not a problem, as in the case of uploading a video to the video sharing service 600 shown in FIG. 1E, compression encoding may be performed by normal inter-frame prediction with priority on the compression rate.

  Since MPEG-4 AVC does not perform inter-frame prediction and compression coding of moving images using only I frames is permitted by the standard, a moving image compressed and encoded using only I frames can be used in a standard-compliant decoder. But it can be decrypted. Therefore, dedicated hardware and software are not required on the screen receiving system side, and the transfer screen image can be decoded only by installing the general-purpose AVC decoder 400, so that it can correspond to a wide range of devices.

  The I / O 90 reads the compressed screen data from the dedicated memory 70, outputs it to the external connection interface such as the WLAN antenna 220a, the Ethernet (registered trademark) connector 220b, the USB port 220c, and the PCIe slot 220d, and transmits it to the outside. . These external connection interfaces are also used when audio data and general-purpose processing results are transmitted to the outside.

  The I / O 90 accepts user input from the button / keyboard / sensor 250 and the touch pad 240 and supplies the user input to the CPU 10. The CPU 10 outputs data input from the user as needed to an external connection interface such as the WLAN antenna 220a, the Ethernet connector 220b, the USB port 220c, and the PCIe slot 220d via the I / O 90 and transmits the data to the outside.

  The old generation machine compatible hardware 210 is a hardware configuration of an old generation machine for maintaining compatibility with an old generation machine such as an old generation game machine. For example, a CPU, an FPU that performs vector operations, such as a VFPU, GPU, Built-in circuits such as DMA (Direct Memory Access) controller. The old generation machine compatible hardware 210 is connected to the system interconnect 40 and uses the dedicated memory 70 as a buffer for reading and writing data.

  The configuration of the screen transfer function 200 implemented in the system LSI 270 will be described with reference to FIGS. 3A to 3C.

  FIG. 3A is a diagram for explaining an implementation example of the screen transfer function 200 in the system LSI 270. In this implementation example, the DISPC 60, the dedicated memory 70, the AVC encoder 80, and the I / O 90 surrounded by the dotted line 120 in the system LSI 270 are the main components of the screen transfer function 200 of the screen transmission system described with reference to FIGS. 1A to 1E. Acts as a block.

  The DISPC 60 reads frame data from the plurality of frame buffers 52 of the main memory 50 in units of lines, performs conversion processing such as enlargement, reduction, and composition in accordance with the size of the LCD / OLED panel 230, and refreshes the LCD / OLED panel 230. The final screen data is generated at the rate and displayed on the LCD / OLED panel 230.

  On the other hand, the DISPC 60 is provided with a dedicated path 62 for extracting a screen image that has been subjected to conversion processing such as enlargement, reduction, and composition and writing it back to the dedicated memory 70, and the screen image after the conversion processing is stored in the dedicated memory 70. Written back. The AVC encoder 80 reads out the converted screen image stored in the dedicated memory 70 from the dedicated memory 70 via the dedicated path 82 and compresses and encodes the compressed and encoded screen image via the dedicated path 82. To write to the dedicated memory 70. The I / O 90 reads the compression-encoded screen image from the dedicated memory 70 via the dedicated path 92 and transfers it to the transfer destination device as a transfer screen image.

  In the implementation example of FIG. 3A, dedicated paths 62, 82, and 92 are used for reading / writing data from / to the dedicated memory 70 until the screen image generated by the DISPC 60 is extracted, compressed and encoded, and output to the outside as a transfer screen image. The system interconnect 40 is not interposed. That is, the dedicated memory 70 and the dedicated paths 62, 82, and 92 are used for the data flow of the compressed transfer screen image, and the data path is separated from the normal data processing using the system interconnect 40. As a result, the screen transfer process can be executed without squeezing the data transfer band of the system interconnect 40. In addition, since the dedicated memory 70 is directly accessed through the dedicated paths 62, 82, and 92, and it is not necessary to access the main memory 50 via the system interconnect 40, the screen transfer process can be executed with low latency.

  FIG. 3B is a diagram illustrating another implementation example of the screen transfer function 200 in the system LSI 270. In this implementation example, the DISPC 60, the dedicated memory 70, the AVC encoder 80, the I / O 90, and the system interconnect 40 surrounded by the dotted line 130 in the system LSI 270 are transferred to the screen of the screen transmission system described with reference to FIGS. 1A to 1E. Acts as the main block of the function 200.

  In the implementation example of FIG. 3B, the screen interconnect generated by the DISPC 60 is extracted, compressed and encoded, and the system interconnect 40 is used to read and write data to the dedicated memory 70 until it is output to the outside as a transfer screen image. Is different.

  The system interconnect 40 reserves a bandwidth by the QoS function for the data flow of the transfer screen image, and separates the data path from the normal processing. By this QoS control, even if the system interconnect 40 is shared between the screen transfer process and the normal process, the screen transfer process is affected by the normal process, and the screen image transfer is delayed and the real-time property is lost. Conversely, it is possible to avoid a situation in which the data transfer band of the system interconnect 40 is compressed by the screen transfer process, the data process of the normal process is delayed, and the response is deteriorated.

  FIG. 3C is a diagram illustrating a configuration in which the dedicated memory 70 is shared between the screen transfer process and the backward compatibility process.

  The old generation machine compatible hardware 210 is a hardware block of an old generation machine such as an old generation game machine. The old generation machine compatible hardware 210 reads / writes data using the dedicated memory 70 instead of the main memory 50. Here, the dedicated memory 70 is designed to have the same latency and bandwidth as the video memory installed in the previous generation machine in order to maintain compatibility with the previous generation machine. This is because the main memory 50 accessed via the system interconnect 40 has a larger latency than the video memory installed in the previous generation machine and is not suitable for maintaining compatibility.

  When the old generation machine compatible hardware 210 operates, the dedicated memory 70 is used as a video memory, and the GPU of the old generation machine compatible hardware 210 writes drawing data into the dedicated memory 70. On the other hand, as described with reference to FIGS. 3A and 3B, the screen transfer process needs to be operated in real time without being affected by the normal process. For this purpose, the dedicated memory 70 is mounted separately from the main memory 50, and the transfer is performed. The screen image is written into the dedicated memory 70. One dedicated memory 70 is shared between the old generation machine compatibility process and the screen transfer process, and both the purpose of maintaining compatibility with the old generation machine and the purpose of guaranteeing the real-time property of the screen transfer process are Achieved at the same time. As a result, the circuit scale can be reduced, and the old generation machine compatibility process and the screen transfer process can be executed efficiently.

  FIG. 4 is a configuration diagram of the DISPC 60. The DISPC 60 reads frame data in units of lines from the plurality of frame buffers 52 of the main memory 50 via the system interconnect 40, performs format conversion, bit expansion processing, scaling processing, and color space conversion, and synthesizes an image by alpha blending. The final screen data is generated and output to the panel. The DISPC 60 pipeline updates the screen at the display screen refresh rate. For example, if the refresh rate is 60 Hz, the final screen is generated 60 times per second.

  The path a (symbol 160a) includes a format conversion / bit expansion unit 150a, a scaling processing unit 152a, and a color space conversion unit 154a. The frame a read from the first frame buffer 52 is subjected to format conversion, bit In this configuration, expansion processing, scaling processing, and color space conversion are performed.

  The path b (reference numeral 160b) includes a format conversion / bit expansion unit 150b, a scaling processing unit 152b, and a color space conversion unit 154b, and format conversion, bit conversion is performed on the frame data read from the second frame buffer 52. In this configuration, expansion processing, scaling processing, and color space conversion are performed.

  The format conversion / bit extension units 150a and 150b read various format information of the frame buffer, convert the data format, and extend it to a bit length used in subsequent processing.

  The scaling processing units 152a and 152b enlarge or reduce the frame image according to the display size, and improve the image quality using a bilinear filter or the like.

  The color space conversion units 154a and 154b convert the color space of the frame in accordance with the display specifications. For example, the frame data is converted into a pixel format such as RGB or YCbCr.

  The alpha blending unit 156 alpha blends the input of the frame data after the conversion of the path a and the frame data after the conversion of the path b, generates a composite image, and outputs the composite image to the MIPI interface unit 158. The MIPI interface unit 158 outputs the image received from the alpha blend unit 156 to the panel as final screen data.

  FIG. 4 shows a configuration example based on input of frame data from two frame buffers. However, when three or more frame buffers are supported, the format conversion / bit extension unit 150a surrounded by a dotted line 160a, scaling is shown. Three or more configurations of the processing unit 152a and the color space conversion unit 154a may be provided in parallel, and the alpha blending unit 156 may have a multi-input one-output configuration that receives and synthesizes inputs from three or more passes.

  The DISPC 60 is provided with a transfer write-back path 64 for extracting the generated screen image as a transfer screen image. The output of the path a and the path b, that is, the screen image after the frame data from the frame buffer is subjected to format conversion, bit expansion, scaling processing, and color space conversion are output as path a output (path a-out) and path b output (path b-out). Further, the output of path c, that is, the screen image after alpha blending the output of path a and the output of path b, is taken out as path c output (path c-out). The screen images of the path a output, the path b output, and the path c output are transferred to the dedicated memory 70 via the dedicated path 62 in the mounting example of FIG. 3A and via the system interconnect 40 in the mounting example of FIG. 3B. Is written back as a screen image.

  By writing back the screen images of the path a output and the path b output to the dedicated memory 70 via the dedicated path 62 or the system interconnect 40, the screen images before the two-screen composition can be individually taken out. By writing back the path c output to the dedicated memory 70 via the system interconnect 40, it is possible to take out the screen image after the two-screen composition.

  FIG. 5A to FIG. 5C are diagrams for explaining screen images taken out from the DISPC 60 by using the transfer write-back path 64.

  FIG. 5A shows a screen image of path a output, which is a moving image stream here. FIG. 5B shows a screen image of pass b output, which is a menu screen or a game screen here. FIG. 5C is a screen image of the path c output, and here is an image obtained by combining the moving image of the path a output with the menu screen of the path b output. By using the transfer write-back path 64 of the DISPC 60, these three types of screen images can be taken out.

  Several application examples of the three types of screen images taken out as transfer screen images will be described. As an application example 1, a composite image of path c output is displayed on a display panel of a screen transmission system having a screen transfer function 200, and the same screen image is transferred to a screen reception system having an AVC decoder 400. For example, this method is used when it is desired to share the screen between the transmission side and the reception side, or when it is desired to view the transmission side screen on the large screen display on the reception side.

  As application example 2, a composite image of path c output is displayed on the display panel of the screen transmission system, and a moving image stream of path a output is transferred to the image reception system. For example, this method is suitable when a menu screen is displayed on the mobile device 100 on the transmission side and an operation is desired at hand while watching a movie on a large screen display. The menu screen may be generated and displayed separately in the screen receiving system without being displayed on the mobile device 100 on the transmission side.

  As application example 3, a composite screen of path c output is displayed on the display panel of the screen transmission system, one of the two screen reception systems transfers the screen of path a output, and the other of path b output. Transfer the screen. For example, a moving image can be displayed on the display of one screen receiving system, and a game screen can be displayed on the display of the other screen receiving system.

  As application example 4, the image of any path output is not displayed on the display panel of the screen transmission system, and the composite image of path c output is transferred to the screen reception system. This method is used when it is desired to switch the screen viewed on the mobile device 100 on the transmission side to the large screen of the TV / monitor 300 on the reception side.

  Although FIG. 4 shows the configuration of the DISPC 60 having one channel image output, two image outputs may be provided. FIG. 6 is a diagram for explaining the configuration of the DISPC 60 provided with 2-channel image output. The configuration of the first channel CH1 (symbol 162) and the configuration of the second channel CH2 (symbol 164) are the same as those in FIG. As shown in FIG. 2, in a normal use, the final screen data of the first channel CH1 is supplied to the LCD / OLED panel 230, while the final screen data of the second channel CH2 is supplied to the HDMI / MHL connector 260. And displayed on a monitor connected by HDMI / MHL.

  Also in the DISPC 60 of FIG. 6, with respect to the first channel CH1, the paths path a-out and path b-out for extracting the images before alpha blending of the path a and the path b of the first channel CH1, and the path c of the first channel CH1 A path path c-out is provided for extracting the composite image after alpha blending. Also, for the second channel CH2, after the alpha blend of the path a-out and path b-out for extracting the image before the alpha blend of the path a and the path b of the second channel CH2, and the path c of the second channel CH2. A path path c-out for extracting the composite image is provided. As in FIG. 4, these screen images of path output are transferred to the dedicated memory 70 via the dedicated path 62 in the implementation example of FIG. 3A and via the system interconnect 40 in the implementation example of FIG. 3B. Will be written back as

  If the output 2-channel DISPC 60 of FIG. 6 is used, two types of screen images are generated and taken out from the transfer write-back path 65 (path a-out, path b-out, path c-out) as transfer screen images. Can be transferred to the destination device.

  FIG. 7A to FIG. 7C are diagrams for explaining an application example of the DISPC 60 having two output channels. In the first channel CH1, a screen A with a low resolution of 1280 pixels by 720 pixels shown in FIG. 7A is generated as a screen for the mobile device 100 on the transmission side. On the other hand, in the second channel CH2, a high-resolution screen B of 1980 horizontal by 1080 pixels shown in FIG. 7B is generated as a screen for the TV / monitor 300 on the receiving side. In the first channel CH1, the screen is updated in accordance with the refresh rate of the panel of the mobile device 100, and in the second channel CH2, the screen is updated in accordance with the refresh rate of the TV / monitor 300.

  As shown in FIG. 7C, the screen A of the first channel CH1 of the DISPC 60 is output to the display panel of the mobile device 100 on the transmission side, while the screen B of the second channel CH2 is transferred from the write-back path for transfer. The screen image is taken out and compressed and encoded, and then transferred to the transfer destination TV / monitor 300. The TV / monitor 300 receives the compression-encoded screen image, decodes it by the AVC decoder 400, and displays it on the display. As a result, the low-resolution screen A is displayed on the mobile device 100 on the transmission side, and the high-resolution screen B is displayed on the TV / monitor 300 on the reception side. However, when it is not necessary to display the screen A on the mobile device 100 on the transmission side, the screen A may not be displayed.

  In this way, if the output 2-channel DISPC 60 is used, a screen with the same image but different resolution and refresh rate can be generated. Therefore, a low-resolution screen is directly output to the mobile device 100, while a high-resolution screen is output. Can be transferred to the TV / monitor 300 for display. Compared with the conventional method in which two frame buffers having different resolutions are prepared, this embodiment can generate two screen images having different resolutions and refresh rates without increasing the memory occupation amount and the CPU load. .

  FIG. 8A to FIG. 8C are diagrams for explaining another application example of the DISPC 60 with two output channels. In the first channel CH1, a screen A of the touch operation menu shown in FIG. 8A is generated as a screen for the mobile device 100 on the transmission side. On the other hand, on the second channel CH2, a game video screen B shown in FIG. 8B is generated as a screen for the TV / monitor 300 on the receiving side. In the first channel CH1, the screen is updated in accordance with the refresh rate of the panel of the mobile device 100, and in the second channel CH2, the screen is updated in accordance with the refresh rate of the TV / monitor 300.

  As shown in FIG. 8C, the screen A of the first channel CH1 of the DISPC 60 is output to the display panel of the mobile device 100 on the transmission side, while the screen B of the second channel CH2 is transferred from the transfer write-back path. The screen image is taken out and compressed and encoded, and then transferred to the transfer destination TV / monitor 300. The TV / monitor 300 receives the compression-encoded screen image, decodes it by the AVC decoder 400, and displays it on the display. Thereby, the screen A of the touch operation menu is displayed on the mobile device 100 on the transmission side, and the screen B of the game video is displayed on the TV / monitor 300 on the reception side.

  In this way, different screens can be generated by using the output 2-channel DISPC 60, so that the touch operation menu screen can be directly output to the mobile device 100 and operated at hand, while the game video screen is It can be transferred to the TV / monitor 300 and displayed on a large screen.

  FIG. 9A to FIG. 9C are diagrams for explaining still another application example of the DISPC 60 having two output channels. In the first channel CH1, a low-resolution screen A of 1280 pixels by 720 pixels shown in FIG. 9A is generated as a screen for the mobile device 100 on the transmission side. On the other hand, in the second channel CH2, as a screen for the TV / monitor 300 and personal computer 320 on the receiving side, a high-resolution screen B of 1980 horizontal by 1080 pixels shown in FIG. 9B is generated.

  As shown in FIG. 9C, the screen A of the first channel CH1 of the DISPC 60 is output to the display panel of the mobile device 100 on the transmission side, while the screen B of the second channel CH2 is transferred from the transfer write-back path. The screen image is taken out and compressed and encoded, and then transmitted to the transfer destination TV / monitor 300 and personal computer 320. The TV / monitor 300 and the personal computer 320 receive the compression-encoded screen image, decodes it by the AVC decoder 400, and displays it on the display. As a result, the low-resolution screen A is displayed on the mobile device 100 on the transmission side, and the high-resolution screen B is displayed on the TV / monitor 300 and the personal computer 320 on the reception side. However, when it is not necessary to display the screen A on the mobile device 100 on the transmission side, the screen A may not be displayed.

  In this way, if the output 2-channel DISPC 60 is used, a screen with the same image but different resolution and refresh rate can be generated. Therefore, a low-resolution screen is directly output to the mobile device 100, while a high-resolution screen is output. Can be transferred to the TV / monitor 300 and personal computer 320 for display. A plurality of users can share the same screen by transferring the high resolution screen to a plurality of receiving systems.

  FIG. 10 is a flowchart for explaining a software handshake procedure between the screen transmission system and the screen reception system.

  The screen transmission system transmits a screen connection request to the screen reception system (S10). This screen connection request includes, as attached information, screen content indicating the type of video, such as a game or movie, resolution, refresh rate, AVC compression format, presence / absence of audio, audio format, presence / absence of user input in the screen receiving system, etc. Is included.

  The screen transmission system initializes the DISPC 60, the AVC encoder 80, and the I / O 90 to prepare video transfer, prepare audio transfer, and prepare TCP / IP and UDP (S12).

  Upon receiving the screen connection request, the screen receiving system initializes the AVC decoder 400 and I / O, prepares audio processing, prepares transfer of user input, and prepares TCP / IP and UDP (S14). The screen reception system returns a screen connection acknowledgment to the screen transmission system (S16). This screen connection acknowledge includes presence / absence of user input as attached information. Here, a case where there is a user input will be described. The screen transmission system prepares to receive user input from the screen reception system (S18).

  The screen reception system transmits a connection OK notification to the screen transmission system indicating that the initialization has been completed and the screen transfer is ready to be received (S20). The screen transmission system confirms the completion of preparation for screen transfer (S22), and the initialization is completed, and a connection start notification to start screen transfer is transmitted to the screen reception system (S24).

  The screen transmission system and the screen reception system perform data transmission / reception by UDP connectionless communication in TCP / IP (S26 to S29). The screen transmission system transmits the screen and audio to the screen reception system, and receives user input from the screen reception system (S30). The screen reception system receives the screen and audio from the screen transmission system, and transmits user input to the screen transmission system (S32).

  According to the screen transfer system of the present embodiment, it is possible to transfer a screen image from one device to another device in real time. In addition, since the user input in the transfer destination device is transferred to the transfer source device, it is possible to generate a screen image reflecting the user input in the transfer source device and transfer the updated screen image to the transfer destination device. it can. Thereby, the user input is reflected on the transfer screen in real time, and the response does not deteriorate even when the user uses the transfer destination device.

  By providing a path for taking out the transfer screen image from the display controller and writing it back to the dedicated memory, the screen image can be transferred to the transfer destination device with low latency. In addition, by providing a dedicated data path for screen transfer processing separately from the system interconnect, screen transfer processing and normal processing compete to compete for system resources, causing changes in application behavior and screen transfer. Can be avoided.

  In particular, when the system load at the time of screen transfer increases, unpredictable changes occur in the software behavior, and the development cost for stabilizing the software behavior increases. In addition, the user does not desire that the operation of the application becomes heavy or the frequency of screen update decreases by transferring the screen. In this embodiment, such a situation is avoided by separating the screen transfer process and the normal process by a dedicated path or QoS control.

  Further, since the transfer screen image is encoded and transferred using standard compression encoding, the screen image can be decoded and displayed only by installing a normal decoder in the screen reception system. In particular, by using compression encoding that does not perform inter-frame prediction, it is possible to reduce the processing time required for encoding and transfer a screen image without delay.

  The present invention has been described based on the embodiments. The embodiments are exemplifications, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are within the scope of the present invention. .

  10 CPU, 20 GPU20, 30 peripheral, 40 system interconnect, 50 main memory, 52 frame buffer, 60 DISPC, 70 dedicated memory, 80 AVC encoder, 90 I / O, 100 mobile device, 110 mobile device, 150 format conversion / bit Expansion unit, 152 Scaling processing unit, 154 Color space conversion unit, 156 Alpha blend unit, 158 MIPI interface unit, 200 screen transfer function, 210 Old generation machine compatible hardware, 230 LCD / OLED panel, 240 touchpad, 250 buttons Keyboard sensor, 260 HDMI / MHL connector, 270 system LSI, 280 touch panel, 300 TV / moni , 310 stationary device, 320 a personal computer, 400 AVC decoder, 500 cloud server, 502 distribution platform 510 cloud service client, 600 video hosting service.

Claims (11)

A main memory with a frame buffer;
A display controller that reads frame data from the frame buffer and generates a screen output image for display on the display;
A dedicated memory provided separately from the main memory;
A system interconnect interconnecting the main memory, the display controller and the dedicated memory;
A path for taking out the screen output image from the display controller as a transfer screen image for transferring to another device and writing it back to the dedicated memory without passing through the system interconnect is provided. Screen transfer device. An encoder that compresses and encodes the transfer screen image;
A path in which the encoder reads the transfer screen image from the dedicated memory without going through the system interconnect, and writes the compressed and encoded transfer screen image to the dedicated memory without going through the system interconnect. The screen transfer apparatus according to claim 1, further comprising: An input / output unit that outputs the compressed and encoded transfer screen image to the outside;
The screen transfer apparatus according to claim 2, wherein the input / output unit includes a path for reading the transfer-encoded screen image from the dedicated memory without passing through the system interconnect. The display controller reads frame data from a plurality of frame buffers, generates a screen image in a plurality of passes, and synthesizes it into a final screen output image,
A path for taking out the screen image generated in each pass and the synthesized final screen output image as the transfer screen image from the display controller and writing them back to the dedicated memory without going through the system interconnect. The screen transfer device according to claim 1, wherein the screen transfer device is provided.   The display controller has two channels of screen output and generates two types of screen output images. One screen output image is output to a display panel connected to the display controller, and the other screen output image is 5. The screen transfer apparatus according to claim 1, wherein the screen transfer apparatus is taken out as a transfer screen image.   6. The screen transfer apparatus according to claim 5, wherein the two types of screen output images are screen output images having different resolutions or refresh rates.   6. The screen transfer apparatus according to claim 5, wherein one of the two types of screen output images is an operation screen.   8. The screen transfer apparatus according to claim 1, wherein the dedicated memory is shared between the screen transfer process and the backward compatibility process. A main memory with a frame buffer;
A display controller that reads frame data from the frame buffer and generates a screen output image for display on the display;
A dedicated memory for storing the screen output image generated by the display controller as a transfer screen image for transferring to another device;
An encoder that compresses and encodes the transfer screen image;
The screen transfer apparatus, wherein the encoder compresses and encodes the transfer screen image without performing inter-frame prediction.   The screen transfer apparatus according to claim 9, wherein the dedicated memory is shared between the screen transfer process and the backward compatibility process. A method of transferring a screen image in an apparatus in which a main memory, a display controller, and a dedicated memory are interconnected by a system interconnect,
A display controller reading frame data from a frame buffer provided in a main memory and generating a screen output image for display on a display;
Extracting the screen output image from the display controller as a transfer screen image for transferring to another device, and writing back to the dedicated memory without going through the system interconnect;
And a step of compressing and encoding the transfer screen image written back to the dedicated memory and transferring it to the other device.

Images