TWI615804B - Apparatus and method for combining video frame and graphics frame - Google Patents

Abstract

The signal processor of the present invention comprises a host processor, a command queue, a video decoding circuit, an image decoding circuit, a synthesis engine and two display buffers. The host processor generates a plurality of image commands based on image encoded data, video encoded data, and mask encoded data from a network, and sets a video flag to an active state. The command queue is configured to set a control command to be valid according to the image commands. The image decoding circuit generates the image frame and two drawing page masks, and the video decoding circuit generates the video frame and a video mask. When the control command is set to be valid or the video flag is set to the active state, according to the video mask and the two drawing page masks, the composition engine views the image frame and the two display buffers. The content of the video or the video frame is moved to another display buffer of the two display buffers.

Description

Device and method for mixing video frame and image frame

The present invention relates to image processing, and more particularly to an apparatus and method for mixing a video frame and an image frame.

The Microsoft MSDN file library, such as the remote desktop protocol (RDP), includes an image pipeline extension (MS-RDPEGFX), an image component interface accelerated extension (MS-RDPEGDI), a basic connection, and image remote processing. Specification (MS-RDPBCGR) to provide an illustration of the image remote system. The image remote system can send data over the line and receive, decode, and render the data using a compatible client. In the image remote system, a bitmap is transmitted from a server to a compatible client's surface, transferred between a drawing page and a drawing page, or transmitted from a drawing page to a bitmap. Take the memory (cache).

Figure 1 shows a schematic diagram of a conventional image remote system 100. In one embodiment, the image remote system 100 includes a server 101, a network 102, a client device 103, at least one peripheral device 104, and a display device 105. The network 102 supports the Microsoft Remote Desktop Protocol (RDP) specification, and the client device 103 can be a thin user box (thin-client). Box). The server 101 includes at least one virtual PC 101a as a host computer and serves at least one client (such as the client device 103). Based on the keyboard/mouse event from the peripheral device 104, the server 101 can transmit a group of image encoded data including a destination area (to generate an image view frame) to the client device 103 through the network 102. . In addition, the server 101 can transmit a set of video encoded data with a time stamp and a set of mask encoded data to the client device 103 through the network 102. Finally, the client device 103 can reconstruct a complete view frame of the hybrid video frame and the image view frame for display on the display device 105.

According to the Microsoft RDP specification, a video frame must be displayed on the display device 105 with reference to a corresponding time stamp, and after the image frame is decoded, the client device 103 decodes an image command "FrameEnd". ". Figure 2 shows a schematic diagram of an image mixing method used by a conventional client device. The video image is usually operated at a rate of 30 frames per second (fps), but its frame rate can be adjusted according to different network bandwidths.

Referring to FIG. 2, in the client device 103, the decoded plurality of video frames are stored in different video decoding buffers 201a-201c, and the decoded plurality of image frames are stored in different images. Image decoding buffers (also known as drawing page buffers) 202a-202b. No matter which frame (video frame or image frame) is updated, a BitBlit/Blit function is driven to move the image data of the entire frame from a rear buffer 203 (also known as a shadow buffer or a behind-the-scenes buffer). Move to a main buffer (also known as the front buffer) 204 to enter The line is displayed. Figure 3 shows an example of a different viewbox update triggered by two timestamp events and three FrameEnd events in a conventional client device 103. In the example of Fig. 3, since there is a total of five frame updates, the data of the entire frame in the rear buffer 203 has to be moved to the main buffer 204 a total of five times, so that not only wasted time but also bandwidth was wasted.

The architecture of Figure 2 is called a single buffer architecture. Since a single buffer architecture requires a large amount of memory access, system performance is significantly reduced. Another problem with a single buffer architecture is screen tearing. Screen tearing is a visual artifact phenomenon that occurs when two or more different frames are displayed in a single screen draw of a display device during vertical playback of the display device (vertical Within the retrace interval, there is not enough time for the high resolution image to move the viewbox image from one buffer to another. The most common solution to prevent screen tearing is to use a multi-viewbox buffer architecture, such as double-buffering. However, multi-viewbox buffers need to store a complete frame of image data, which may result in increased memory costs and processing time.

In view of the above problems, one of the objects of the present invention is to provide a signal processor for mixing a video frame and an image frame to accelerate image reconstruction.

An embodiment of the present invention provides a signal processor for mixing a video frame and an image frame in an image remote system. The signal The processor comprises: a host processor, a command queue, a video decoding circuit, an image decoding circuit, a synthesis engine, and two display buffers. The host processor generates a plurality of image commands based on the image encoded data from a network, and sets a video flag to an active state based on the video encoded data and the mask encoded data from the network. The command queue is configured to sequentially receive and transmit the image commands, and set a control command to be valid according to the image commands. The image decoding circuit decodes the image encoded material according to the image commands to generate the image view frame, a current drawing page mask, and a previous drawing page mask. The video decoding circuit is coupled to the host processor for determining whether to decode the video encoded data and the mask encoded data to generate the video frame and a video mask according to the video flag. When the control command is set to be valid or the video flag is set to the active state, according to the video mask and the two drawing page masks, the composition engine views the image frame and the two display buffers. The content of one and at least one of the video frames are moved to another display buffer of the two display buffers. The two display buffers form a reconstructed image for output to a display device according to the image frame, the content of one of the two display buffers, and at least one of the video frames.

Another embodiment of the present invention provides a method for mixing a video frame and an image frame, suitable for an image remote system, the method comprising: generating a plurality of images based on image encoded data from a network An image command, and setting a video flag to an active state according to the video encoded data and the mask encoded data from the network; according to the image commands, a control is performed The command is set to be valid; according to the image commands, the image encoded data is decoded to obtain the image frame, a current drawing page mask, and a previous drawing page mask; according to the video flag, whether to decode the image Video encoding data and the mask encoded data to obtain the video frame and a video mask; when the control command is set to be valid or the video flag is set to an active state, according to the video mask and the two Drawing a picture mask, moving the image frame, the content of one of the two display buffers, and at least one of the video frames to another display buffer of the two display buffers; and, according to the figure The reconstructed image is formed for output to a display device such as the view frame, the content of one of the two display buffers, and at least one of the video frames.

Another embodiment of the present invention provides an image remote system comprising: a network, a server, a client device, and a display device. The network supports the Microsoft Remote Desktop Protocol Specification, which is used by the server to transmit image encoded data, video encoded data, and mask encoded data. The client device includes a signal processor for receiving the image encoded data, the video encoded data, and the mask encoded data to form a reconstructed image. The display device is configured to display the reconstructed image. The signal processor is configured to mix a video frame and an image frame, and includes: a host processor, a command queue, a video decoding circuit, an image decoding circuit, a synthesis engine, and two display buffers. Device. The host processor generates a plurality of image commands according to the image encoded data from the network, and sets a video flag to be valid according to the video encoded data and the mask encoded data from the network. state. The command queue is configured to sequentially receive and transmit the image commands, and set a control command to be valid according to the image commands. The image decoding circuit decodes the image encoded material according to the image commands to generate the image view frame, a current drawing page mask, and a previous drawing page mask. The video decoding circuit is coupled to the host processor for determining whether to decode the video encoded data and the mask encoded data to generate the video frame and a video mask according to the video flag. When the control command is set to be valid or the video flag is set to the active state, according to the video mask and the two drawing page masks, the composition engine views the image frame and the two display buffers. The content of one and at least one of the video frames are moved to another display buffer of the two display buffers. The two display buffers form a reconstructed image for output to a display device according to the image frame, the content of one of the two display buffers, and at least one of the video frames.

12

12

100, 40‧‧‧ image remote system

101‧‧‧Server

101a‧‧‧Virtual PC

102‧‧‧Network

103, 41‧‧‧ client devices

104‧‧‧ Peripheral devices

105‧‧‧Display device

201a, 201b, 201c‧‧‧ video decoding buffer

202a, 202b‧‧‧ image decoding buffer

203‧‧‧Back buffer

204‧‧‧ main buffer

400‧‧‧Signal Processor

410‧‧‧Host processor

420‧‧‧ Order queue

430‧‧‧Synthesis Engine

450‧‧‧Image decoding circuit

470‧‧‧Video Decoding Circuit

48A, 48B‧‧‧ display buffer

551, 571, 574‧ ‧ buffer

552‧‧‧Image Engine

553‧‧‧Drawing Page Mask Generator

556‧‧‧Plot page buffer

55A, 55B‧‧‧ drawing page mask buffer

572‧‧‧Video Decoder

573‧‧‧Video Decoding Buffer

575‧‧‧Video Mask Generator

576‧‧‧Video Mask Buffer

The invention may be fully understood from the following detailed description and the appended claims. A schematic of the image remote system 100.

Figure 2 shows a schematic diagram of an image mixing method used by a conventional client device.

Figure 3 shows an example of a different viewbox update triggered by two timestamp events and three FrameEnd events in a conventional client device.

4A is a schematic diagram showing a remote image system in accordance with an embodiment of the present invention.

Figure 4B is a functional block diagram showing a signal processor in accordance with an embodiment of the present invention.

Figure 5A is a functional block diagram showing an image decoding circuit in accordance with an embodiment of the present invention.

Figure 5B is a functional block diagram showing a video decoding circuit in accordance with an embodiment of the present invention.

Figure 6 shows an example of a video mask Vm(n) and two drawing page masks Sm(n), Sm(n-1).

In the non-overlapping case, FIG. 7A shows a relationship diagram between the video decoding frame and the related parameters of the image decoding frame, video/image update, and data transmission path.

Figure 7B shows an example of the first view frame reconstruction sequence performed by the composition engine in the non-overlapping case.

Figure 7C shows a variation of the relevant parameters and associated masks according to Figures 7A and 7B.

7D is a flow chart showing a method of mixing a video frame and an image frame in a non-overlapping situation according to the first embodiment of the present invention.

Figure 8A shows the video decoding frame and the image solution in the case of overlap The relationship between the parameters of the code frame, video/image update, and data transfer path.

Figure 8B shows an example of a second view frame reconstruction sequence performed by the composition engine in the case of overlap.

Figure 8C shows a variation of the relevant parameters and associated masks according to Figures 8A and 8B.

8D is a flow chart showing a method of mixing a video frame and an image frame in an overlapping situation according to a second embodiment of the present invention.

The term "and/or" as used throughout the specification and subsequent claims includes a related item or a plurality of arbitrary and all combinations, and the singular forms such as "a" and "the" are used. It also includes the singular and plural meanings unless otherwise specified in this specification.

The present invention employs a composition mechanism to mix video frames and image frames in a non-overlapping manner or in an overlapping manner. One of the features of the present invention is to reduce the buffer from a video decoding buffer and a drawing page by using a double buffer structure, a video frame mask, two image frame masks, and two parameters (Vcnt and Gcnt). Data is moved by a buffer or a front buffer to a back buffer to speed up image reconstruction.

4A is a schematic diagram showing an image remote system 40, in accordance with an embodiment of the present invention. In one embodiment, the image remote system 40 includes a server 101, a network 102, a client device 41, and at least one peripheral device. 104, and a display device 105.

In one embodiment, the network 102 supports the Microsoft Remote Desktop Protocol Specification. In one embodiment, the client device 41 can be a compact user box. The client device 41 includes a signal processor 400.

The server 101 includes at least one virtual PC 101a as a host computer and serves at least one client (such as the client device 41). Based on the keyboard/mouse event from the peripheral device 104, the server 101 can transmit a group of image encoded data including a destination area (to generate an image view frame) to the client device 103 through the network 102. . In addition, the server 101 can transmit a set of time-coded video encoded data (to generate a video frame) and a set of mask encoded data to the client device 41 through the network 102. Finally, the client device 41 reconstructs a complete view frame according to at least one of the video view frame, the content of a display buffer, and the image view frame for display on the display device 105.

Figure 4B is a functional block diagram showing a signal processor in accordance with an embodiment of the present invention. Referring to FIG. 4B, the signal processor 400 of the present invention includes a host processor 410, a command queue 420, a composition engine 430, a video decoding circuit 470, an image decoding circuit 450, and Two display buffers 48A and 48B.

The host processor 410 receives a plurality of sets of image encoded data, a plurality of sets of video encoded data, and a plurality of sets of masked encoded data from the network 102, wherein each set of encoded data has a header. In one embodiment, the host processor 410 issues a plurality of phases based on a header of the input set of image encoded material. A closed image command, such as FrameEnd, to the command queue 420, after which the set of image encoded data is transmitted to the image decoding circuit 450. The command queue 420 includes a plurality of first in first out (FIFO) buffers and stores a plurality of image commands to provide an image engine 552 for execution (described in FIG. 5A). In particular, please note that an image command "FrameEnd" indicates that the image decoding circuit 450 has completed decoding of a corresponding image frame. Thus, when the image command "FrameEnd" is queued into the command queue 420, the command queue 420 asserts a control signal Ctl to trigger the composition engine 430.

In an embodiment, the host processor 410 sets a timer interrupt according to the time stamp included in the header of the input video encoding data, and then transmits the group of video encoded data and a corresponding one thereof. The group mask encodes the data to the video decoding circuit 470. Wherein, the time stamp indicates when the video decoding data needs to be transmitted to the display device 105 for display. When the timer interrupt occurs, the host processor 410 sets a video flag to an active state, that is, the host processor 410 transmits the three registers 43r through its address bus. The values of 51r, 52r (for example, the addresses of the registers 43r, 51r, and 52r are respectively located at A1, A2, and A3) (the registers 51r and 52r will be described in FIG. 5B) are set to preset values. For example: 1, causing the composition engine 430, the video decoder 572, and the video mask generator 575 to begin operation. When the system power is turned on, the registers 43r, 51r, 52r are reset to their initial values, and the initial value of the itself is different from the above preset value. Please note that the preset values of the above-mentioned registers 43r, 51r, 52r are equal to 1 It is merely an example and is not a limitation of the present invention.

The two display buffers 48A and 48B are used to avoid the occurrence of screen tearing. At any one time, one of the two display buffers 48A and 48B (as a main buffer or front buffer) is scanned for display, and the other of the two display buffers 48A and 48B is displayed. The buffer (as a back buffer or shadow buffer) will be drawn.

In accordance with the present invention, a timestamp event or a FrameEnd event triggers the composition engine 430 to deliver or synthesize image material. Specifically, when the composition engine 430 is triggered by a FrameEnd event (or a valid control signal Ctl, or an image update event), the composition engine 430 masks the Sm(n) according to the two drawing pages. Sm(n-1) performs image data transfer from a drawing page buffer 556 to a back buffer (one of the two display buffers 48A and 48B). In addition, when the composition engine 430 is triggered by a timestamp event (or the preset value of the register 43r, or a video update event, or a video flag of a valid state), the composition engine 430 masks according to a video. The cover Vm(n) performs video data transfer from a video decoding buffer 573 to the rear buffer. When the image data is delivered, the control signal Ctl is de-asserted; when the video data is delivered, the video flag is set to an inactive state, that is, the These registers 43r, 51r, 52r are reset to their own default values.

Figure 5A is a functional block diagram showing an image decoding circuit in accordance with an embodiment of the present invention. Please refer to FIG. 5A, the image decoding circuit 450 A buffer 551, an image engine 552, a page mask generator 553, a drawing page buffer 556, and two drawing page mask buffers 55A and 55B are included.

The buffer 551 stores image encoded material from the host processor 410, and the image engine 552 receives the image command Cmd from the command queue 420 and the image encoded material from the buffer 551, and includes A destination area (or image area) DE in the image command Cmd is imaged to decode the picture page buffer 556. The drawing page mask generator 553 generates a current drawing page mask Sm(n) of the current image frame n according to the target area DE, and writes the current drawing page mask Sm(n) to the two The drawing page masks one of the buffers 55A and 55B. At the same time, the drawing page mask generator 553 has stored a previous drawing page mask Sm(n-1) of the previous image frame n-1 to the other of the two drawing page mask buffers 55A and 55B. One. According to the present invention, the drawing page masks Sm(n) and Sm(n-1) and the video mask Vm(n) are both bitmap masks, and each mask Sm(n), Sm(n-1) And the number of mask values of Vm(n) is equal to the number of pixels of any one of the image frames n, n-1 and the video frame n. In the drawing page mask Sm(n), each pixel position is marked with one of two symbols (1 or 0) for indicating the current image frame n and the previous image frame n Whether the pixel value of the corresponding position of -1 is changed; in the drawing page mask Sm(n-1), each pixel position is marked with one of two symbols (1 or 0) for indicating Whether there is a change in the pixel value of the corresponding position of the first image frame n-1 and the immediately preceding second image frame n-2 immediately before. with Similarly, in the video mask Vm(n), each pixel position is marked with one of two symbols (1 or 0) for indicating the current video frame n and the previous video frame n. Whether the pixel value of the corresponding position of -1 has changed. For example, in the drawing page mask Sm(n), the mask value 1 indicates that the corresponding pixel position between the current image frame n and the previous image frame n-1 has a changed pixel value, The mask value 0 indicates that the pixel position is not changed in the corresponding pixel position between the current image frame n and the previous image frame n-1. The detailed operation of the image engine 552, the drawing page mask generator 553, the drawing page buffer 556, and the two drawing page mask buffers 55A and 55B is disclosed in U.S. Patent Nos. 13/669,762 and 14/473,607. In the patent application (the content of the above-mentioned patent is hereby incorporated by reference in its entirety).

Figure 5B is a functional block diagram showing a video decoding circuit in accordance with an embodiment of the present invention. Referring to FIG. 5A, the video decoding circuit 470 includes two buffers 571, 574, a video decoder 572, a video decoding buffer 573, a video mask generator 575, and a video mask buffer 576.

The two buffers 571, 574 are configured to store video encoded data and mask encoded data from the host processor 410. When the host processor 410 sets the value of the register 51r to 1 (indicating that a timer interrupt, or a video update, or a video flag of a valid state) is generated, the video decoder 572 decodes the video encoded data. And storing a video decoded image Vd in the video decoding buffer 573. When the host processor 410 sets the value of the register 52r to 1 (indicating When a timer interrupt, or a video update, or a valid state video flag is generated, the video mask generator 575 receives the mask encoded data, generates a video mask Vm according to the Microsoft RDP specification, and then stores the The video mask Vm is in the video mask buffer 576. In one embodiment, the video mask Vm is a Microsoft defined EGT mask.

According to the Microsoft RDP specification, in a reconstructed view frame (to be displayed), the display area (or active area) of a video view frame and an image view frame is complementary. Figure 6 shows an example of a video mask Vm(n) and two drawing page masks Sm(n), Sm(n-1). Please note that for the sake of clarity, the number of mask values for the video mask Vm(n) and the sheet masks Sm(n), Sm(n-1) of Fig. 6 is reduced to 12. Suppose that the display area of a video frame is in the lower left corner of the reconstructed view frame (indicated by a dashed rectangle), and the other areas of the reconstructed view frame are the display area of an image view frame; in the video mask Vm ( The mask values 1 and 0 of n) respectively indicate that the corresponding pixel position between the current video frame n and the previous video frame n-1 has a changed pixel value and an unchanged pixel value; the Sm is masked on the drawing page. The mask values 1 and 0 of (n) respectively indicate that the corresponding pixel position between the current image frame n and the previous image frame n-1 has a changed pixel value and an unchanged pixel value; The mask values 1 and 0 of the mask Sm(n-1) respectively indicate the corresponding pixels between the immediately preceding first image frame n-1 and the immediately preceding first image frame n-2. The position has a change in pixel value and an unchanged pixel value. Therefore, the mask values of the lower left corners of the drawing page masks Sm(n) and Sm(n-1) (indicated by a dotted rectangle) are all 0, and the area other than the lower left corner of the video mask Vm(n) Cover The mask value is also 0. The combination of the two drawing page masks Sm(n), Sm(n-1) determines which image area needs to be moved from the drawing page buffer 556 to the back buffer, and the single video mask Vm(n) Then, it is determined which image area needs to be moved from the video decoding buffer 573 to the back buffer.

In order to save bandwidth, the present invention has the following two methods for mixing the video frame and the image frame to form the reconstructed frame in the rear buffer: (1) non-overlapping method: the video frame and the The image frames do not overlap to form the reconstructed view frame in the back buffer; (2) the overlap method: the video view frame and the image view frame overlap to form the reconstructed view frame in the rear buffer Specifically, after all the data of the image frame is moved to the back buffer, the video frame is placed on the image frame.

Throughout the specification and drawings, the following nouns define the relevant meanings unless specifically stated otherwise in this specification. The term "S2O" means a data transfer from the draw page buffer to the back buffer (one of the two display buffers 48A and 48B). The term "O2O" means a data transfer from the front buffer (one of the two display buffers 48A and 48B) to the back buffer (the other of the two display buffers 48A and 48B). The term "V2O" indicates a data transfer from the video decoding buffer 573 to the subsequent buffer.

In operation, the composition engine 430 utilizes two parameters Gcnt and Vcnt to control video/image frame updates. Specifically, when the composition engine 430 is triggered by an image update but is not triggered by any video update, the composition engine 430 resets the parameter Gcnt to 0, and the parameter Vcnt The value of the value is incremented by one (i.e., the increment value is 1); when the composition engine 430 is triggered by a video update but is not triggered by any image update, the composition engine 430 resets the parameter Vcnt to 0, and the parameter is The value of Gcnt is increased by 1 (ie, the increment value is 1).

FIG. 7A shows the related parameters, video/image update of the video decoding frame and the image decoding frame in a non-overlapping situation (ie, using a non-overlapping method to mix a video frame and an image frame). Relationship diagram of data transmission channels. Please refer to FIG. 7A, when the composition engine 430 is simultaneously subjected to an image update event (or a valid control signal Ctl, or a FrameEnd event) and a video update event (or a preset value of the register 43r). The synthesis engine 430 resets the parameters Gcnt and Vcnt to 0 and performs V2O and S2O operations, triggered by a timestamp event or a valid state video flag. Thereafter, if the composition engine 430 is only triggered by a second video update, the composition engine 430 resets the parameter Vcnt to 0, sets the value of the parameter Gcnt to 1, and performs V2O and O2O operations. Then, if the composition engine 430 is only triggered by a third video update, the synthesis engine 430 resets the parameter Vcnt to 0, adds the value of the parameter Gcnt to 1, and performs only V2O operation. On the other hand, after the parameters Gcnt and Vcnt are reset to 0, if the composition engine 430 is only triggered by a second image update, the synthesis engine 430 resets the parameter Gcnt to 0, and the parameter Vcnt The value is set to 1, and V2O and O2O operations are performed. Then, if the composition engine 430 is only triggered by a third image update, the composition engine 430 resets the parameter Gcnt to 0, increments the value of the parameter Vcnt by 1, and performs only S2O operation.

Figure 7B shows that in the case of non-overlapping, the synthesis engine An example of the first frame reconstruction order of rows. Figure 7C shows a variation of the relevant parameters and associated masks according to Figures 7A and 7B. The first view frame reconstruction sequence will be described in detail below with reference to FIGS. 7A and 7C. First, set the mask value of the drawing page mask Sm(0) to 0.

Referring to FIG. 7B, during the frame 1, since there is a video update and an image update from the beginning, the synthesizing engine 430 resets the parameters Gcnt and Vcnt to 0, and then performs S2O and V2O operations. In other words, during the frame 1, the composition engine 430 moves the image data of the circular area in the drawing page buffer 556 to the back buffer 48A according to the combination of the drawing page masks Sm(0) and Sm(1). And moving the image data of the oblique triangle area in the video decoding buffer 573 to the back buffer 48A according to the video mask Vm(1). After the image of the frame 1 is reconstructed, the roles of the two display buffers are swapped such that the display buffer 48A becomes the front buffer and the display buffer 48B becomes the back buffer.

During the frame reconstruction of view frame 2, the image engine 552 is more like a white triangle in the drawing page buffer 556. During the frame 2, since the composition engine 430 is only triggered by a second image update, the composition engine 430 resets the parameter Gcnt to 0, adds 1 to the value of the parameter Vcnt, and performs S2O and V2O operations. In other words, during the frame 2, the composition engine 430 moves the image data of the circular area and the white triangle in the drawing page buffer 556 to the back buffer according to the combination of the drawing page masks Sm(1) and Sm(2). 48B, and according to the video mask Vm(2) (=Vm(1)), the shadow triangle in the video decoding buffer 573 The image data of the shaped area is moved to the rear buffer 48B. After the image of the frame 2 is reconstructed, the roles of the two display buffers are swapped such that the display buffer 48B becomes the front buffer and the display buffer 48A becomes the back buffer.

During the frame reconstruction of view frame 3, the image engine 552 is more like a slash in the drawing page buffer 556. During view frame 3, because the composition engine 430 is only triggered by a third image update, the composition engine 430 resets the parameter Gcnt to 0, increments the value of the parameter Vcnt, and then performs only S2O operation. In other words, during the frame 3, the composition engine 430 moves the image data of the oblique line moon region and the white triangle in the drawing page buffer 556 to the rear according to the combination of the drawing page masks Sm(2) and Sm(3). Buffer 48A. Since both buffers 48A, 48B already contain a diagonal triangle area, there is no need to perform V2O operation. After the image of the frame 3 is reconstructed, the roles of the two display buffers are exchanged again.

During the frame reconstruction of frame 4, the video decoder 573 is imaged in a diagonally elliptical region in the video decoding buffer 573. During view frame 4, because the composition engine 430 is only triggered by a video update, the composition engine 430 resets the parameter Vcnt to 0, increments the value of the parameter Gcnt, and then performs V2O and O2O operations. In addition, the mask values of the drawing page mask Sm(4) are both set to zero. In other words, during the frame 4, the composition engine 430 moves the image data of the oblique elliptical area in the video decoding buffer 573 to the back buffer 48B according to the video mask Vm(4), and masks the Sm according to the drawing page ( 3) The image data of the oblique elliptical area in the front buffer 48A is moved to the rear buffer 48B. Image in frame 4 After being reconstructed, the roles of the two display buffers are exchanged again.

During frame reconstruction of view frame 5, video decoder 573 is imaged in a diagonal heart shaped area in video decoding buffer 573. During the frame 5, since the composition engine 430 is only triggered by a video update, the composition engine 430 resets the parameter Vcnt to 0, adds 1 to the value of the parameter Gcnt, and then performs V2O operation. In other words, during the frame 5, the composition engine 430 moves the image data of the oblique heart-shaped area in the video decoding buffer 573 to the back buffer 48A according to the video mask Vm(5). The mask value of the drawing page mask Sm(5) is reset to 0 because there is no image update. After the image of the frame 5 is reconstructed, the roles of the two display buffers are exchanged again.

During the frame reconstruction of view frame 6, the image engine 552 is more like a white cloud region in the drawing page buffer 556. During view frame 6, because the composition engine 430 is only triggered by an image update, the composition engine 430 resets the parameter Gcnt to 0, increments the value of the parameter Vcnt, and then performs S2O and V2O operations. In other words, during view frame 6, the composition engine 430 moves the image data of the white cloud region in the drawing page buffer 556 to the back buffer 48B according to the drawing page mask Sm(6), and according to the video mask Vm ( 6) (= Vm (5)), the image data of the oblique heart-shaped area in the video decoding buffer 573 is moved to the rear buffer 48B.

7D is a flow chart showing a method of mixing a video frame and an image frame in a non-overlapping situation according to the first embodiment of the present invention. Hereinafter, please refer to FIGS. 4B, 5A-5B and 7A-7C to illustrate the hybrid video of the present invention. The method of looking at the frame and the image frame.

Step S702: reset the values of the parameters Gcnt and Vcnt to 0.

Step S704: Is the composition engine 430 triggered by an image update event (or a valid control signal Ctl, or a FrameEnd event)? If yes, go to step S706; if no, go to step S718.

Step S706: Is the composition engine 430 triggered by a video update event (or a preset value of the register 43r, or a timestamp event, or a video flag of a valid state)? If yes, go to step S708; otherwise, go to step S722.

Step S708: reset the values of the parameters Gcnt and Vcnt to 0.

Step S710: Is the value of the parameter Gcnt less than 2? If yes, go to step S712; otherwise, go to step S714.

Step S712: Perform S2O operation. That is, the composition engine 430 moves the image material in the drawing page buffer 556 to the back buffer according to the association of the drawing page masks Sm(n) and Sm(n-1).

Step S714: Is the value of the parameter Vcnt less than 2? If yes, go to step S716; otherwise, go to step S704.

Step S716: Perform V2O operation. In other words, the composition engine 430 moves the image data in the video decoding buffer 573 to the back buffer according to the mask Vm(n).

Step S718: Is the composition engine 430 triggered by a video update event (or a preset value of the register 43r or a timestamp event)? If Yes, go to step S720; otherwise, go to step S704.

Step S720: reset the value of the parameter Vcnt to 0, and add 1 to the value of the parameter Gcnt.

Step S722: reset the value of the parameter Gcnt to 0, and add 1 to the value of the parameter Vcnt.

FIG. 8A shows related parameters, video/image update, and data of the video decoding frame and the image decoding frame in an overlapping situation (ie, mixing a video frame and an image frame by using an overlapping method). Diagram of the route of delivery. Comparing Figures 7A and 8A, there is only one difference: that is, Vcnt>=2 and Gcnt=0. As described above, in the case of overlap, after all the data of the image frame is moved to the back buffer, the video frame is placed on the image frame. Accordingly, regardless of whether a video update or/and an image update occurs, the overlay engine 430 must perform V2O operation in the final stage of the frame reconstruction in the case of overlap.

Figure 8B shows an example of a second view frame reconstruction sequence performed by the composition engine in the case of overlap. Figure 8C shows a variation of the relevant parameters and associated masks according to Figures 8A and 8B. Compared with the first view frame reconstruction sequence of the 7B-7C picture, the second view frame reconstruction order of the 8B-8C picture has only one difference, that is, in the case of overlap, at Vcnt=2 and Gcnt=0. In the state (during frame 3), the composition engine 430 performs V2O operation, which is consistent with Figure 8A.

Figure 8D is in accordance with a second embodiment of the present invention, in the case of overlap Next, a flow chart of a method of mixing a video frame and an image frame is displayed. Hereinafter, please refer to FIGS. 4B, 5A-5B and 8A-8C to illustrate a method of mixing a video frame and an image frame according to the present invention.

Step S802: reset the value of the parameter Gcnt to 0.

Step S804: Is the composition engine 430 triggered by an image update event (or a valid control signal Ctl, or a FrameEnd event)? If yes, go to step S806; if no, go to step S814.

Step S806: reset the value of the parameter Gcnt to 0.

Step S808: Is the value of the parameter Gcnt less than 2? If yes, go to step S810; otherwise, go to step S812.

Step S810: Perform S2O operation. That is, the composition engine 430 moves the image material in the drawing page buffer 556 to the back buffer according to the association of the drawing page masks Sm(n) and Sm(n-1).

Step S812: Perform V2O operation. In other words, the composition engine 430 moves the image data in the video decoding buffer 573 to the back buffer according to the mask Vm(n).

Step S814: Is the composition engine 430 triggered by a video update event (or a preset value of the register 43r, or a timestamp event, or a valid flag)? If yes, go to step S816; otherwise, go to step S804.

Step S816: The value of the parameter Gcnt is incremented by one.

The preferred embodiments and additional figures are provided to illustrate the invention and are not intended to limit the scope of the invention; Any changes or modifications that may be made by the industry are included in the scope of the following patent application.

400‧‧‧Signal Processor

410‧‧‧Host processor

420‧‧‧ Order queue

430‧‧‧Synthesis Engine

450‧‧‧Image decoding circuit

470‧‧‧Video Decoding Circuit

48A, 48B‧‧‧ display buffer

Claims (39)

A signal processor for mixing a video frame and an image frame in an image remote system, the signal processor comprising: a host processor, generating a plurality of images according to image coded data from a network Like a command, and according to the video encoded data and mask encoded data from the network, a video flag is set to a valid state; a command queue, sequentially receiving and transmitting the image commands, and according to the maps Actuating a control command as a command; an image decoding circuit decoding the image encoded data according to the image commands to generate the image frame, a current drawing page mask, and a previous drawing page a video decoding circuit coupled to the host processor, and determining, according to the video flag, whether to decode the video encoded data and the mask encoded data to generate the video frame and a video mask; a synthesis engine, When the control command is set to be valid or the video flag is set to an active state, the synthesis engine images the image frame and the two display buffers according to the video mask and the two drawing page masks. One of the contents and at least one of the video frames being moved to another display buffer of the two display buffers; and the two display buffers, according to the image frame, the two display buffers And a content of the video frame and at least one of the video frames forming a reconstructed image for output to a display device; The host processor sets, by the address bus, the first register of the synthesis engine, the second register of the video decoding circuit, and the third register to a first set value, a first The second set value and a third set value are used to set the video flag to an active state. The signal processor as recited in claim 1, wherein the video frame and the image frame do not overlap. The signal processor as recited in claim 2, wherein when the control command is set to be valid and the video flag is set to an active state, or when the control command is set to be valid twice and the video flag is When the target is set to the idle state, the synthesis engine moves the image frame to the other display buffer according to the union of the two drawing page masks, and according to the video mask, the video view The box is moved to the other display buffer. The signal processor as recited in claim 2, wherein the synthesizing engine masks the two drawing pages when the control command is set to be invalid once and the video flag is continuously set to the active state twice. And a combination of moving the contents of one of the two display buffers to the other display buffer, and moving the video frame to the other display buffer according to the video mask. The signal processor as recited in claim 2, wherein the synthesis engine is based on the two drawing pages when the control command is continuously set to three times and the video flag is continuously set to an idle state at least twice. Masked The union moves the image frame to the other display buffer. The signal processor as recited in claim 1, wherein the synthesizing engine is based on the video mask when the control command is continuously set to be invalid at least twice and the video flag is continuously set to an active state at least three times. And moving the video frame to the other display buffer. The signal processor as recited in claim 1, wherein the video frame is placed above the image frame. The signal processor as recited in claim 7, wherein when the control command is set to be valid and the video flag is set to an active state, or when the control command is continuously set to be valid at least twice and When the video flag is continuously set to the idle state at least once, the synthesizing engine first moves the image frame to the other display buffer according to the union of the two drawing page masks, and then according to the video mask. And moving the video frame to the other display buffer. The signal processor of claim 7, wherein when the control command is set to be invalid once and the video flag is continuously set to a valid state, the synthesis engine first covers the two drawing pages. The collection of the cover moves the content of one of the two display buffers to the other display buffer, and the video frame is moved to the other display buffer according to the video mask. The signal processor as recited in claim 1, wherein the current drawing page mask, the previous drawing page mask, and the video mask are points An array mask, wherein the current drawing page mask indicates a plurality of changed pixels and unchanged pixels in a current image view frame compared to a immediately preceding first image view frame, wherein the previous drawing The page mask indicates that in the immediately preceding first image frame, a plurality of changed pixels and unchanged pixels are compared to a immediately preceding second image frame, and wherein the video mask is indicated in a In the current video frame, a plurality of pixels are changed and unchanged pixels compared to the first video frame immediately before. The signal processor of claim 1, wherein the video decoding circuit comprises: a video decoding buffer connected to the synthesizing engine for storing the video frame; a video mask buffer connected to The video engine is configured to store the video mask; the video decoder has a second register for responding to the second set value, and decoding the video encoded data to generate the video frame for storage a video decoding buffer; and a video mask generator having the third register for responding to the third set value, decoding the mask encoded data to generate the video mask for storage to the video mask buffer. The signal processor as recited in claim 1, wherein the image decoding circuit comprises: a drawing page buffer connected to the synthesis engine for storing the image a picture frame buffer; two drawing page mask buffers coupled to the composition engine for storing the current drawing page mask and the previous drawing page mask; an image engine based on the images including a destination area Actuating the image encoding data, generating the image frame and storing the image frame into the drawing page buffer, and a drawing page mask generator, generating the current drawing according to the target area The page mask and the current drawing page mask are stored in one of the two drawing page mask buffers. The signal processor as recited in claim 1, wherein when the image commands include a FrameEnd command, the command queue sets the control command to be valid. The signal processor as recited in claim 1, wherein the host processor sets the video flag to an active state based on a time stamp included in the video encoded material. A method for mixing a video frame and an image frame, suitable for an image remote system, the method comprising: generating a plurality of image commands according to image encoded data from a network, and a video encoding data and a mask encoding data of the network, setting a video flag to a valid state; according to the image commands, setting a control command to be valid; and decoding the image encoded data according to the image commands To get the image a view frame, a current drawing page mask, and a previous drawing page mask; determining, according to the video flag, whether to decode the video encoded data and the mask encoded data to obtain the video frame and a video mask; When the control command is set to be valid or the video flag is set to the active state, according to the video mask and the two drawing page masks, the image frame, the contents of one of the two display buffers, and the Forming at least one of the video frames to another display buffer of the two display buffers; and forming according to the image frame, the content of one of the two display buffers, and at least one of the video frames Reconstructing the image for output to a display device; wherein the moving step further comprises: when the control command is continuously set to be invalid at least twice and the video flag is continuously set to an active state at least three times, according to the video mask, The video frame is moved to the other display buffer. The method of claim 15, wherein the video frame and the image frame do not overlap. The method of claim 16, wherein the moving step further comprises: when the control command is set to be valid and the video flag is set to a valid state, or when the control command is continuously set to be effective And when the video flag is set to the idle state once, according to the joint of the two drawing pages The set moves the image view frame to the other display buffer, and moves the video view frame to the other display buffer according to the video mask. The method of claim 16, wherein the moving step further comprises: when the control command is set to be invalid once and the video flag is continuously set to a valid state twice, according to the two drawing pages The collection of the covers moves the contents of one of the two display buffers to the other display buffer, and the video frame is moved to the other display buffer based on the video mask. The method of claim 16, wherein the moving step further comprises: when the control command is continuously set to be valid three times and the video flag is continuously set to an idle state at least twice, according to the two drawing A union of page masks that move the image frame to the other display buffer. The method of claim 15, wherein the video frame is placed above the image frame. The method of claim 20, wherein the moving step further comprises: when the control command is set to be valid and the video flag is set to an active state, or when the control command is continuously set to be valid at least When the video flag is continuously set to the idle state at least once, the image frame is first moved to the other display buffer according to the union of the two drawing page masks. And moving the video frame to the other display buffer according to the video mask. The method of claim 20, wherein the moving step further comprises: when the control command is set to be invalid once and the video flag is continuously set to a valid state twice, according to the two drawing pages. The combination of the masks moves the contents of one of the two display buffers to the other display buffer, and the video frame is moved to the other display buffer according to the video mask. The method of claim 15, wherein the current drawing page mask, the previous drawing page mask, and the video mask are both bitmap masks, wherein the current drawing page mask indicates a current In the image view frame, a plurality of changed pixels and unchanged pixels are compared to a first image frame immediately before, wherein the previous drawing page mask indicates the first image frame immediately before the immediately adjacent image frame a plurality of changed pixels and unchanged pixels compared to a second image frame immediately before, and wherein the video mask indicates a first video in a current video frame compared to a immediately preceding video frame Depending on the frame, most of the pixels change pixels and unchanged pixels. The method of claim 15, wherein the decoding the image decoding data comprises: generating the image frame according to the image encoding data and the image commands including a destination area; Based on the destination area, the current drawing page mask is generated. The method of claim 15, wherein the step of setting the control command to be valid comprises: setting the control command to be valid when the image commands include a FrameEnd command. The method of claim 15, wherein the step of setting the video flag to an active state comprises: setting the video flag to an active state according to a time stamp included in the video encoded material. An image remote system comprising: a network supporting a Microsoft Remote Desktop Protocol specification; a server for transmitting image encoded data, video encoded data, and mask encoded data through the network; a client device comprising a The signal processor receives the image encoded data, the video encoded data and the mask encoded data to form a reconstructed image, and a display device for displaying the reconstructed image; wherein the signal processor is configured to mix a video frame And an image view frame, comprising: a host processor, generating a plurality of image commands according to the image coded data, and setting a video flag to an effective state according to the video coded data and the mask coded data; a command queue for sequentially receiving and transmitting the image commands, and setting a control command to be valid according to the image commands; an image decoding circuit for decoding the image encoded data according to the image commands a video frame, a current drawing page mask, and a previous drawing page mask; a video decoding circuit coupled to the host processor, and determining whether to decode the video encoded data according to the video flag The mask encodes data to generate the video frame and a video mask; a synthesis engine, when the control command is set to be valid and the video flag is set to an active state, according to the video mask and the two a drawing page mask, the composition engine moving the image frame, the content of one of the two display buffers, and at least one of the video frames to another display buffer of the two display buffers; Two display buffers, according to the image frame, the content of one of the two display buffers, and at least one of the video frames, forming a reconstructed image for output to the display device; wherein the host The first register, the second register of the video decoding circuit, and the third register are respectively set to a first set value, a second set value, and A third set value is used to set the video flag to an active state. The system of claim 27, wherein the video frame and the image frame do not overlap. The system of claim 28, wherein when the control command is set to be valid and the video flag is set to an active state, or when the control command is continuously set to be valid twice and the video flag is When set to the idle state once, the composition engine moves the image frame to the other display buffer according to the union of the two drawing page masks, and according to the video mask, the video frame Move to the other display buffer. The system of claim 28, wherein when the control command is set to be invalid once and the video flag is continuously set to the active state twice, the synthesizing engine is based on the joint of the two drawing pages. And moving the content of one of the two display buffers to the other display buffer, and moving the video frame to the other display buffer according to the video mask. The system of claim 28, wherein the synthesizing engine masks the two drawing pages when the control command is continuously set to three times and the video flag is continuously set to an idle state at least twice. The union of the image is moved to the other display buffer. The system of claim 27, wherein when the control command is continuously set to be invalid at least twice and the video flag is continuously set to an active state at least three times, the synthesizing engine according to the video mask The video view frame is moved to the other display buffer. The system described in claim 27, wherein the video frame It is placed above the image frame. The system of claim 33, wherein when the control command is set to be valid and the video flag is set to an active state, or when the control command is continuously set to be valid at least twice and the video flag is When the label is continuously set to the idle state at least once, the composition engine first moves the image frame to the other display buffer according to the union of the two drawing page masks, and according to the video mask, The video view frame is moved to the other display buffer. The system of claim 33, wherein when the control command is set to be invalid once and the video flag is continuously set to the active state twice, the synthesis engine first masks according to the two drawing pages. In association, the content of one of the two display buffers is moved to the other display buffer, and the video frame is moved to the other display buffer according to the video mask. The system of claim 27, wherein the current drawing page mask, the previous drawing page mask, and the video mask are both bitmap masks, wherein the current drawing page mask indicates a current In the image view frame, a plurality of changed pixels and unchanged pixels are compared to a first image frame immediately before, wherein the previous drawing page mask indicates the first image frame immediately before the immediately adjacent image frame a plurality of changed pixels and unchanged pixels compared to a second image frame immediately before, and wherein the video mask indicates a first video in a current video frame compared to a immediately preceding video frame Depending on the frame, most of them Change pixels and unchanged pixels. The system as claimed in claim 27, wherein the video decoding circuit comprises: a video decoding buffer connected to the synthesis engine for storing the video frame; a video mask buffer connected to the composite An audio decoder for storing the video mask; a video decoder having the second temporary register for responding to the second set value, decoding the video encoded data to generate the video view frame for storage to the video decoding buffer And a video mask generator having the third register for responding to the third set value, decoding the mask encoded data to generate the video mask for storage to the video mask buffer. The system of claim 27, wherein when the image commands include a FrameEnd command, the command queue sets the control command to be valid. The system of claim 27, wherein the host processor sets the video flag to an active state based on a time stamp included in the video encoded material.