JP2021125801A - Image encoding device, image decoding device, and image processing system - Google Patents

Abstract

To suppress a code amount to be generated, in encoding processing in which an image format is selectable for each block in an image.SOLUTION: On the basis of an intra-prediction mode to be applied to an encoding-object block in an image, a selection unit selects a third image format to be applied to encoding processing of the encoding-object block, out of a first image format and a second image format. The second image format has an amount of information smaller than an amount of information of the first image format. An encoding unit encodes the encoding-object block, by intra-prediction encoding using the third image format.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image coding device, an image decoding device, and an image processing system.

Virtual desktops that use servers or virtual machines on the cloud have become widespread in order to control information leaks or improve security. In the virtual desktop, a remote desktop system is used to remotely control the server from the client terminal.

FIG. 1 shows an example of a remote desktop system. The remote desktop system of FIG. 1 includes a client terminal 101 and a server 102, and the client terminal 101 and the server 102 communicate with each other via a communication network 103. The server 102 is provided in a remote environment such as a cloud, and an OS (Operating System) and an application program operate in the server 102.

The server 102 transmits the desktop image to the client terminal 101, and the client terminal 101 displays the received desktop image on the screen. When the user performs an operation using the keyboard or the mouse, the client terminal 101 transmits the operation information of the operation performed by the user to the server 102. The server 102 changes the desktop image according to the received operation information, and transmits the changed desktop image to the client terminal 101.

In relation to the remote desktop system, an information processing device capable of improving the operation response while maintaining the versatility of the thin client is known (see, for example, Patent Document 1). An image decoding device capable of efficiently decoding an image signal by intra-predicting a luminance signal and a color difference signal according to a color difference format is also known (see, for example, Patent Document 2). HEVC (High Efficiency Video Coding) is also known in relation to video coding (see, for example, Non-Patent Document 1).

Japanese Unexamined Patent Publication No. 2012-15868 Japanese Unexamined Patent Publication No. 2016-154353

ITU-T H.265 (11/2019), “High efficiency video coding”, Recommendation ITU-T H.265, ISO / IEC 23008-2

In virtual desktops, users often operate the screen. The desktop image transferred from the server includes various images from high-definition images such as CAD (Computer-Aided Design) to Internet moving images and the like. In this case, it is desirable to perform appropriate coding processing according to the difference in the video.

However, when the desktop image is divided into a plurality of blocks and the image format is switched for each block to perform the coding process, the code amount of the identification information indicating the image format of each block increases.

It should be noted that such a problem occurs not only in the desktop image coding process in the remote desktop system but also in various image coding processes.

In one aspect, it is an object of the present invention to suppress the amount of code generated in the coding process in which the image format can be selected for each block in the image.

In one proposal, the image coding apparatus includes a selection unit and a coding unit. The selection unit selects the third image format applied to the coding process of the coded target block from the first image format and the second image format based on the intra prediction mode applied to the coded target block in the image. select. The second image format has a smaller amount of information than the amount of information of the first image format. The coding unit encodes the coded block by intra-predictive coding using the third image format.

According to one aspect, it is possible to suppress the amount of code generated in the coding process in which the image format can be selected for each block in the image.

It is a figure which shows the remote desktop system. It is a figure which shows YUV4: 4: 4 and YUV4: 2: 0. It is a figure which shows the desktop image. It is a functional block diagram of an image coding apparatus. It is a functional block diagram of an image decoding apparatus. It is a functional block diagram of an image processing system. It is a figure which shows the adjacent pixel in the adjacent block. It is a flowchart (No. 1) of a video coding process. It is a flowchart (2) of a video coding process. It is a flowchart (1) of a video decoding process. It is a flowchart (2) of a video decoding process. It is a hardware block diagram of an information processing apparatus.

Hereinafter, embodiments will be described in detail with reference to the drawings.

The server device of Patent Document 1 divides the screen into a plurality of areas and monitors the frequency of change for each area. Then, the server device compresses the data of the high-frequency change area whose change frequency exceeds the threshold value into the data of the compression method for moving images, and transmits the data together with the attribute information for specifying the high-frequency change area to the client terminal. Further, the server device generates an image of the update rectangle for the area other than the frequently changed area, and transmits the image to the client terminal together with the attribute information for specifying the update rectangle.

The client terminal decodes the received compression method data for the moving image and displays the frequently changed area in the area specified by the received attribute information. Further, the client terminal displays the received update rectangular image in the area specified by the received attribute information in the area other than the frequently changed area.

According to the server device of Patent Document 1, among the images transmitted to the client terminal, the load of the compression process and the decoding process can be minimized while focusing on the image that deteriorates the operation response and reducing the amount of data. .. Therefore, it is possible to improve the operation response while maintaining the versatility of the thin client.

By the way, by applying the technique of Patent Document 1, the desktop image is divided into a high-frequency change area and a low-frequency change area, moving image coding is applied to the high-frequency change area, and a still image code is applied to the low-frequency change area. It is also possible to apply the conversion. The high frequency change area is an area where the frequency of change exceeds the threshold value, for example, a moving image area. The low-frequency change area is an area where the change frequency is equal to or less than the threshold value, such as a still image area.

In the high frequency change area, the image quality deterioration is less likely to be recognized as compared with the low frequency change area, so that the server device encodes the compressed signal of the image. On the other hand, in the low-frequency change region, the image quality deterioration is more likely to be recognized as compared with the high-frequency change region, so the uncompressed signal of the image is encoded. As a result, it is possible to reduce the amount of code in the high frequency change area and maintain the image quality in the low frequency change area.

In the moving image coding, each of the images at a plurality of times included in the video is divided into a plurality of blocks, and the coding process is performed in block units. As the coding target block, a block in the YUV color space is usually used. The component Y in the YUV color space represents a luminance signal, and the component U and the component V represent a color difference signal.

As the uncompressed signal of the image, for example, a signal of the image format of YUV4: 4: 4 can be used, and as the compressed signal of the image, for example, a signal of the image format of YUV4: 2: 0 can be used. By encoding the blocks in the low frequency change area with YUV 4: 4: 4 and the blocks in the high frequency change area with YUV 4: 2: 0, the entire desktop image is encoded with YUV 4: 4: 4. In comparison, it is possible to reduce the network bandwidth while suppressing the deterioration of the subjective image quality.

FIG. 2 shows an example of YUV 4: 4: 4 and YUV 4: 2: 0. YUV4: 4: 4 is an image format in which blocks of each component of component Y, component U, and component V have an equal size. On the other hand, YUV4: 2: 0 is an image format in which the horizontal and vertical sizes of the blocks of the component U and the component V are half the size of the blocks of the component Y. In video coding, a YUV 4: 2: 0 block may be used by taking advantage of the human visual characteristic that deterioration of a color difference signal is difficult to be recognized.

In the following, the coding using YUV4: 4: 4 may be described as YUV4: 4: 4 coding, and the coding using YUV4: 2: 0 may be described as YUV4: 2: 0 coding.

FIG. 3 shows an example of a desktop image displayed on the screen of the client terminal 101 in the remote desktop system of FIG. The desktop image of FIG. 3 represents, for example, an operation screen of a video editing application program, and includes a still image area 301, a moving image area 302, and a moving image area 303. Text and the like are displayed in the still image area 301, and the video to be edited is displayed in the moving image area 302 and the moving image area 303. In this case, YUV4: 4: 4 coding is applied to the still image area 301, and YUV4: 2: 0 coding is applied to the moving image area 302 and the moving image area 303.

Compared with YUV4: 2: 0 coding, YUV4: 4: 4 coding improves the image quality after coding, but increases the amount of generated code. Therefore, it is preferable to adaptively switch between YUV 4: 4: 4 coding and YUV 4: 2: 0 coding according to the region to be coded from the viewpoint of both subjective image quality and generated code amount. I can say.

However, when a flag indicating which coding of YUV4: 4: 4 coding or YUV4: 2: 0 coding is applied is added for each small area such as a block, it is coded. The amount of code of the flag included in the bitstream of the desktop image increases.

FIG. 4 shows an example of a functional configuration of the image coding apparatus of the embodiment. The image coding device 401 of FIG. 4 includes a selection unit 411 and a coding unit 412.

The selection unit 411 has a third image format applied to the coding process of the coded target block from the first image format and the second image format based on the intra prediction mode applied to the coded target block in the image. Select. The second image format has a smaller amount of information than the amount of information of the first image format. The coding unit 412 encodes the coded block by intra-predictive coding using the third image format.

According to the image coding device 401 of FIG. 4, it is possible to suppress the amount of code generated in the coding process in which the image format can be selected for each block in the image.

FIG. 5 shows an example of a functional configuration of the image decoding device of the embodiment. The image decoding device 501 of FIG. 5 includes a selection unit 511 and a decoding unit 512.

The selection unit 511 selects a third image format applied to the decoding process of the decoding target block from the first image format and the second image format based on the intra prediction mode of the decoding target block in the image. The second image format has a smaller amount of information than the amount of information of the first image format. The decoding unit 512 decodes the code information of the decoding target block by intra-predictive decoding using the third image format.

According to the image decoding device 501 of FIG. 5, it is possible to suppress the amount of code generated in the coding process in which the image format can be selected for each block in the image.

FIG. 6 shows a functional configuration example of an image processing system including the image coding device 401 of FIG. 4 and the image decoding device 501 of FIG. The image processing system of FIG. 6 is, for example, a remote desktop system and includes a client terminal 601 and a server 602. The client terminal 601 corresponds to the image decoding device 501 of FIG. 5, and the server 602 corresponds to the image coding device 401 of FIG.

The client terminal 601 and the server 602 communicate with each other via the communication network 603. The communication network 603 is, for example, a WAN (Wide Area Network) or a LAN (Local Area Network). The client terminal 601 is, for example, a personal computer or a mobile terminal device. As the mobile terminal device, for example, a smartphone, a tablet, or a notebook personal computer is used.

The client terminal 601 includes an input unit 611, a selection unit 612, a decoding unit 613, a frame memory 614, a display unit 615, and a communication unit 616. The selection unit 612 and the decoding unit 613 correspond to the selection unit 511 and the decoding unit 512 of FIG. 5, respectively. The selection unit 612 is an example of the second selection unit.

The server 602 includes a video generation unit 621, a selection unit 622, an encoding unit 623, a frame memory 624, and a communication unit 625. The selection unit 622 and the coding unit 623 correspond to the selection unit 411 and the coding unit 412 of FIG. 4, respectively. The selection unit 622 is an example of the first selection unit.

The input unit 611 of the client terminal 601 receives an operation input from the user, and the communication unit 616 transmits the operation information of the received operation input to the server 602.

The communication unit 625 of the server 602 receives the operation information from the client terminal 601. The video generation unit 621 generates a video to be displayed on the screen of the client terminal 601 according to the received operation information, and outputs the generated video to the coding unit 623. The video includes a plurality of images in time series. Each image corresponds to an image to be encoded and is sometimes called a picture or a frame.

The coding unit 623 divides each image included in the video into a plurality of blocks, and performs coding processing in block units. Each block corresponds to a block to be coded and is coded by inter-predictive coding or intra-predictive coding. Inter-predictive coding is a coding method that encodes a coded image using the information of the coded image, and intra-predictive coding is a coded image that uses only the information that the coded image has. It is a coding method that encodes. As the block, for example, a CU (Coding Unit) can be used.

Details of intra-predictive coding are described in, for example, Chapter 8.4 of Non-Patent Document 1. Intra-prediction itself is MPEG-AVC (Moving Picture Experts Group-Advanced Video Coding) / H.A. It has been adopted since 264.

The coding unit 623 generates a reference image from the coded image, stores it in the frame memory 624, and performs inter-predictive coding using the reference image. Further, the coding unit 623 generates reference pixels from the coded blocks in the coded target image, stores them in the frame memory 624, and performs intra-predictive coding using the reference pixels. The coding unit 623 generates a reference image or a reference pixel by performing a local decoding process on the code information of the coded image or the coded block.

The coding unit 623 determines the prediction mode applied to each block and encodes the blocks based on the determined prediction mode. At this time, the selection unit 622 selects an image format to be applied to the block coding process from the image format F1 and the image format F2 according to the determined prediction mode.

The image format F1 and the image format F2 correspond to the first image format and the second image format, respectively. The number of pixels of the image format F2 is smaller than the number of pixels of the image format F1. Therefore, the image format F1 is a high-resolution image format, and the image format F2 is a low-resolution image format having a lower resolution than the image format F1. YUV4: 4: 4 is an example of the image format F1, and YUV4: 2: 0 is an example of the image format F2.

For example, when the image format of the image to be encoded is the image format F1, the coding unit 623 can generate a block of the image format F2 by compressing the image of the block to be encoded by downsampling.

When the determined prediction mode is the intra prediction mode, the selection unit 622 selects the image format F1 or the image format F2 based on the determined intra prediction mode. Then, the coding unit 623 encodes the block by intra-predictive coding using the selected image format.

On the other hand, when the determined prediction mode is the inter-prediction mode, the selection unit 622 selects the image format F2, and the coding unit 623 encodes the block by inter-prediction coding using the image format F2. .. By selecting the image format F2 for the block to which the inter-prediction mode is applied, the amount of code of the block encoded by the inter-prediction coding can be reduced.

An image containing only strong edges is more likely to recognize image quality deterioration due to compression than an image containing only weak edges. Therefore, when a strong edge exists in the block, the image format F1 is applied to suppress the deterioration of the subjective image quality, and when the strong edge does not exist in the block, the image format F2 is applied. It is desirable to suppress the increase in the amount of generated code.

The intra prediction mode represents either Planar prediction, DC (Direct Current) prediction, or angle prediction. Of these, Planar prediction and DC prediction correspond to intra prediction other than angle prediction.

When the intra prediction mode represents Planar prediction or DC prediction, it is considered that there is no strong edge in the coded block, so the selection unit 622 selects the image format F2. On the other hand, when the intra prediction mode represents angle prediction, there is a possibility that a strong edge exists in the coded target block, so the selection unit 622 selects the image format F1.

By selecting the image format according to whether the intra prediction mode represents Planar prediction, DC prediction, or angle prediction, it is possible to easily determine the image format simply by referring to the determined intra prediction mode. can.

By the way, when the intra prediction mode represents the angle prediction, the selection unit 622 selects either the image format F1 or the image format F2 based on the pixel values of the pixels in the adjacent block adjacent to the coded target block. Is also possible. By selecting the image format based on the pixel values of the pixels in the adjacent block, the pixel values of the reference pixels of the encoded block can be used as the selection criteria of the image format.

For example, the selection unit 622 calculates the edge strength using the pixel values of the adjacent pixels adjacent to the coded target block in the adjacent block. When the edge strength of the adjacent pixel is larger than the threshold value, it can be estimated that a similar strong edge exists in the coded block.

FIG. 7 shows an example of adjacent pixels in an adjacent block. The left adjacent block 702 is adjacent to the left side of the coding target block 701, and the upper adjacent block 703 is adjacent to the upper side of the coding target block 701. The left adjacent pixel 712 in the left adjacent block 702 is adjacent to the left side of the coding target block 701, and the upper adjacent pixel 713 in the upper adjacent block 703 is adjacent to the upper side of the coding target block 701.

For example, when the size of the coded block 701 is N × N, in the intra prediction, 2N + 1 pixels in the left adjacent block 702 and the blocks above and below it (upper left block and lower left block) are reference pixels adjacent to the left. Used as. Further, 2N + 1 pixels in the upper adjacent block 703 and its left and right blocks (upper left block and upper right block) are used as upper adjacent reference pixels. At this time, as in the lower left block, the pixels at the end of the coded block are copied and used for the unprocessed block positions in the raster scan order.

As shown in FIG. 7, the coordinates in the horizontal direction are x-coordinates, the coordinates in the vertical direction are y-coordinates, and the coordinates of the upper left pixel 711 of the coded block 701 are (x, y) = (0,0). .. In this case, in the intra prediction, not only the left adjacent pixel 712 and the upper adjacent pixel 713, but also the pixels of the coordinates (-1, -1) in the upper left block and the coordinates (N + 1, -1) to the coordinates (N + 1, -1) in the upper right block ( The pixels of 2N, -1) can also be used as reference pixels. However, in order to calculate the edge strength, it is sufficient to use the left adjacent pixel 712 and the upper adjacent pixel 713.

When the intra prediction mode represents the angle prediction, the mode number of the intra prediction mode determines which of the left adjacent pixel 712 and the upper adjacent pixel 713 is referred to in the intra prediction. When the left adjacent pixel 712 is referred to, the selection unit 622 calculates the edge strength using the pixel value of the left adjacent pixel 712 stored as the reference pixel in the frame memory 624. When the upper adjacent pixel 713 is referred to, the selection unit 622 calculates the edge strength using the pixel value of the upper adjacent pixel 713 stored as the reference pixel in the frame memory 624.

When the edge strength is larger than the threshold value, the selection unit 622 estimates that a strong edge exists in the coded block 701 and selects the image format F1. On the other hand, when the edge strength is smaller than the threshold value, the selection unit 622 estimates that there is no strong edge in the coded block 701 and selects the image format F2.

In this way, by comparing the edge strength calculated using the pixel values of adjacent pixels with the threshold value, the presence or absence of strong edges in the coded block is estimated from the pixel values of the reference pixels of the coded block. be able to. Therefore, it becomes possible to select an image format suitable for the coded block from the image format F1 or the image format F2.

Since the edge of the image appears most prominently in the component Y among the component Y, the component U, and the component V, it is effective to calculate the edge intensity using the pixel value of the component Y. In this case, the selection unit 622 uses the edge strength EL (y) of the left adjacent pixel 712 indicated by the y coordinate as the pixel value Y (x, y) of the component Y in the coordinate (x, y) according to the following equation. Can be calculated.

EL (y) = | Y (-1, y-1) -Y (-1, y) |
+ | Y (-1, y) -Y (-1, y + 1) | (1)

EL (y) is calculated based on the difference between the pixel values of two vertically adjacent left adjacent pixels 712. When calculating EL (y) for the purpose of estimating whether or not a strong edge exists in the coded block 701, it is sufficient to use the pixel value of the left adjacent pixel 712, and it is sufficient to use the pixel value in the left adjacent block 702. It is not always necessary to use other pixels. EL (y) is calculated for all y values corresponding to the height of the coded block 701.

The selection unit 622 selects the image format F1 when any of the ELs (y) is equal to or greater than the threshold value THL, and selects the image format F2 when all the ELs (y) are smaller than the threshold value THL. The selection unit 622 may compare the statistical values of all EL (y) with the threshold THL instead of comparing the EL (y) with the threshold THL. As the statistical value, the average value, the median value, the mode value, the maximum value and the like are used.

Similarly, the selection unit 622 can calculate the edge strength EU (x) of the upper adjacent pixel 713 indicated by the x coordinate by the following equation.

EU (x) = | Y (x-1, -1) -Y (x, -1) |
+ | Y (x, -1) -Y (x + 1, -1) | (2)

EU (x) is calculated based on the difference between the pixel values of two horizontally adjacent upper adjacent pixels 713. When calculating EU (x) for the purpose of estimating whether or not there is a strong edge in the coded block 701, it is sufficient to use the pixel value of the upper adjacent pixel 713, and it is sufficient to use the pixel value in the upper adjacent block 703. It is not always necessary to use other pixels. EU (x) is calculated for all x values corresponding to the width of the coded block 701.

The selection unit 622 selects the image format F1 when any EU (x) is equal to or greater than the threshold value THU, and selects the image format F2 when all EU (x) are smaller than the threshold value THU. The selection unit 622 may compare all EU (x) statistics with the threshold THU instead of comparing the EU (x) with the threshold THU.

When the intra prediction mode represents the angle prediction, the selection unit 622 can estimate whether or not a strong edge exists in the coded target block 701 based on an index other than the edge strength. As an index other than the edge strength, for example, an image format applied to the coding process of the adjacent block referred to by the angle prediction can be used.

When the image format F1 is applied to the coding process of the adjacent block, the selection unit 622 estimates that a strong edge exists in the coded block 701 and selects the image format F1. On the other hand, when the image format F2 is applied to the coding process of the adjacent block, the selection unit 622 estimates that there is no strong edge in the coded block 701 and selects the image format F2.

When the coded block is the first block to be encoded in the image, the selection unit 622 can select the image format F1 or the image format F2 by using, for example, the technique of Patent Document 1. The selection unit 622 selects the image format F2 when the coded block corresponds to the high frequency change area, and selects the image format F1 when the coded block corresponds to the low frequency change area.

The coding unit 623 encodes each block of each image included in the video to generate the code information of the coded block, and also generates the prediction mode information indicating the prediction mode of the coded block. The communication unit 625 transmits a bit stream including the code information and the prediction mode information of each block to the client terminal 601.

The communication unit 616 of the client terminal 601 receives the bit stream from the server 602 and outputs the received bit stream to the decoding unit 613.

The decoding unit 613 decodes the code information included in the bit stream in block units. Each block corresponds to a block to be decoded, and the code information of each block is decoded by inter-predictive decoding or intra-predictive decoding. The decoding unit 613 stores the decoded image in the frame memory 614, and uses the decoded image as a reference image for inter-predictive decoding. Further, the decoding unit 613 stores the pixels of the decoded block in the image to be decoded in the frame memory 614, and uses the pixels of the decoded block as reference pixels to perform intra-predictive decoding.

The decoding unit 613 acquires the prediction mode by decoding the prediction mode information of each block, and decodes the code information of the block based on the acquired prediction mode. At this time, the selection unit 612 selects an image format applied to the block decoding process from the image format F1 and the image format F2 by the same selection method as the server 602 according to the acquired prediction mode.

By selecting the image format by the same selection method as that of the server 602, it is possible to select the same image format as the coding process for each block. Therefore, it is not necessary to include a flag indicating which of the image format F1 and the image format F2 is applied in the code information of each block, and it is possible to suppress an increase in the code amount of the bit stream.

When the acquired prediction mode is the intra prediction mode, the selection unit 612 selects the image format F1 or the image format F2 based on the acquired intra prediction mode. Then, the decoding unit 613 decodes the code information of the block by intra-predictive decoding using the selected image format.

For example, the selection unit 612 selects the image format F2 when the intra prediction mode represents Planar prediction or DC prediction, and selects the image format F1 when the intra prediction mode represents angle prediction.

When the intra prediction mode represents angle prediction, the selection unit 612 can also select either the image format F1 or the image format F2 based on the pixel values of the pixels in the adjacent block adjacent to the decoding target block. be. In this case, the selection unit 612 calculates the edge strength using the pixel values of the adjacent pixels adjacent to the decoding target block in the adjacent block, similarly to the server 602. Then, the selection unit 612 selects the image format F1 when the edge strength is larger than the threshold value, and selects the image format F2 when the edge strength is smaller than the threshold value.

On the other hand, when the acquired prediction mode is the inter-prediction mode, the selection unit 612 selects the image format F2, and the decoding unit 613 decodes the code information of the block by inter-prediction decoding using the image format F2. ..

The display unit 615 reproduces the video by displaying the decoded image in the frame memory 614 on the screen.

8A and 8B are flowcharts showing an example of the video coding process performed by the server 602 of FIG. First, the coding unit 623 divides the image included in the video into a plurality of blocks (step 801), and determines the coding target block and the prediction mode of the coding target block (step 802).

Next, the selection unit 622 checks the prediction mode of the coded block (step 803). When the prediction mode is the inter-prediction mode (step 803, NO), the selection unit 622 selects the image format F2 (step 810). On the other hand, when the prediction mode is the intra prediction mode (step 803, YES), the selection unit 622 checks whether the intra prediction mode represents Planar prediction, DC prediction, or angle prediction (step 804).

When the intra prediction mode represents Planar prediction or DC prediction (step 804, NO), the selection unit 622 selects the image format F2 (step 810). On the other hand, when the intra prediction mode represents angle prediction (step 804, YES), the selection unit 622 calculates the edge strength using the pixel values of the adjacent pixels adjacent to the coded block (step 805). .. Then, the selection unit 622 compares the edge strength with the threshold value (step 806).

When the edge strength is equal to or higher than the threshold value (step 806, YES), the selection unit 622 selects the image format F1 (step 807). When the edge intensity is smaller than the threshold value (step 806, NO), the selection unit 622 selects the image format F2 (step 810).

For example, when the EL (y) of the equation (1) is used as the edge strength, the selection unit 622 calculates the EL (y) for all the values of y corresponding to the height of the block to be encoded. Each EL (y) is compared to the threshold THL. Then, the selection unit 622 selects the image format F1 when any of the EL (y) is equal to or higher than the threshold value THL, and selects the image format F2 when all the EL (y) are smaller than the threshold value THL.

Further, when the EU (x) of the equation (2) is used as the edge strength, the selection unit 622 calculates the EU (x) for all the values of x corresponding to the width of the coded block, and each of them The EU (x) is compared to the threshold THU. Then, the selection unit 622 selects the image format F1 when any EU (x) is equal to or higher than the threshold value THU, and selects the image format F2 when all the EUs (x) are smaller than the threshold value THU.

When the block to be encoded is the first block in the image, the selection unit 622 selects the image format F1 or the image format F2 by using the technique of Patent Document 1.

Next, the coding unit 623 encodes the coded block by the predictive coding indicated by the determined prediction mode using the selected image format, and generates the code information of the coded block (step). 808). Then, the communication unit 625 transmits the code information and the prediction mode information of the coded block to the client terminal 601 (step 809).

Next, the coding unit 623 checks whether the coded block is the last block in the image (step 811). If the encoded block is not the last block (steps 811, NO), the server 602 repeats the processes from step 802 onward for the next block.

If the coded block is the last block (steps 811, YES), the coding unit 623 checks whether the coded image is the last image included in the video (step 812). .. If the encoded image is not the last image (steps 812, NO), the server 602 repeats the processes from step 801 onwards for the next image. If the encoded image is the last image (steps 812, YES), the server 602 ends the process.

9A and 9B are flowcharts showing an example of the video decoding process performed by the client terminal 601 of FIG. First, the communication unit 616 receives the code information and the prediction mode information of the decoding target block from the server 602 (step 901).

Next, the decoding unit 613 acquires the prediction mode of the decoding target block by decoding the prediction mode information, and checks the acquired prediction mode (step 902). When the prediction mode is the inter-prediction mode (step 902, NO), the selection unit 612 selects the image format F2 (step 908). On the other hand, when the prediction mode is the intra prediction mode (step 902, YES), the selection unit 612 checks whether the intra prediction mode represents Planar prediction, DC prediction, or angle prediction (step 903).

When the intra prediction mode represents Planar prediction or DC prediction (step 903, NO), the selection unit 612 selects the image format F2 (step 908). On the other hand, when the intra prediction mode represents angle prediction (step 903, YES), the selection unit 612 calculates the edge strength using the pixel values of the adjacent pixels adjacent to the coded block (step 904). .. Then, the selection unit 612 compares the edge strength with the threshold value (step 905).

When the edge strength is equal to or higher than the threshold value (step 905, YES), the selection unit 612 selects the image format F1 (step 906). When the edge intensity is smaller than the threshold value (step 905, NO), the selection unit 612 selects the image format F2 (step 908).

When the block to be decoded is the first block in the image, the selection unit 612 selects the image format F1 or the image format F2 by using the technique of Patent Document 1.

Next, the decoding unit 613 decodes the code information of the decoding target block by the prediction decoding indicated by the acquired prediction mode using the selected image format (step 907). Then, the decoding unit 613 stores the decoding pixels of the decoding target block in the frame memory 614.

Next, the decoding unit 613 checks whether or not the decoded code information is the code information of the last block in the image (step 909). When the decoded block is not the code information of the last block (step 909, NO), the client terminal 601 repeats the processing after step 901 for the next code information.

When the decoded code information is the code information of the last block (step 909, YES), the display unit 615 displays the decoded image in the frame memory 614 on the screen (step 910). Then, the decoding unit 613 checks whether or not the decoded code information is the last code information included in the bit stream (step 911).

When the decoded code information is not the last code information (steps 911, NO), the client terminal 601 repeats the processing after step 901 for the next code information. When the decoded code information is the last code information (steps 911, YES), the client terminal 601 ends the process.

The configuration of the image coding device 401 in FIG. 4 is only an example, and some components may be omitted or changed depending on the application or conditions of the image coding device 401. The configuration of the image decoding device 501 of FIG. 5 is only an example, and some components may be omitted or changed depending on the use or conditions of the image decoding device 501.

The configuration of the image processing system of FIG. 6 is only an example, and some components may be omitted or changed depending on the use or conditions of the image processing system. For example, when the client terminal 601 and the server 602 perform bidirectional video communication, the video generation unit 621, the selection unit 622, and the coding unit 623 are added to the client terminal 601 and the selection unit 612 and the decoding unit are added to the server 602. 613 can be added. The image coding device 401 of FIG. 4 and the image decoding device 501 of FIG. 5 can also be used for video communication other than the remote desktop system.

The flowcharts shown in FIGS. 8A to 9B are merely examples, and some processing may be omitted or changed depending on the configuration or conditions of the image processing system. For example, when the intra prediction mode represents angle prediction and the image format F1 is selected without calculating the edge intensity, the processes of steps 805 and 806 in FIG. 8A can be omitted. In this case, the processes of steps 904 and 905 of FIG. 9A can also be omitted.

The remote desktop system shown in FIG. 1 is only an example, and some components may be omitted or changed depending on the use or conditions of the remote desktop system. YUV 4: 4: 4 and YUV 4: 2: 0 shown in FIG. 2 are merely examples, and different image formats may be used depending on the configuration or conditions of the image processing system. For example, YUV4: 2: 2 or YUV4: 1: 1 may be used instead of YUV4: 2: 0.

The still image area and the moving image area shown in FIG. 3 are merely examples, and the still image area and the moving image area change according to the desktop image. The coded block and the adjacent block shown in FIG. 7 are merely examples, and the shape of each block changes according to the image division method.

Equations (1) and (2) are merely examples, and different calculation equations may be used depending on the configuration or conditions of the image processing system.

FIG. 10 shows a hardware configuration example of an information processing device (computer) used as the image coding device 401 of FIG. 4, the image decoding device 501 of FIG. 5, the client terminal 601 of FIG. 6, and the server 602. The information processing device of FIG. 10 includes a CPU (Central Processing Unit) 1001, a memory 1002, an input device 1003, an output device 1004, an auxiliary storage device 1005, a medium drive device 1006, and a network connection device 1007. These components are hardware and are connected to each other by bus 1008.

The memory 1002 is, for example, a semiconductor memory such as a ROM (Read Only Memory), a RAM (Random Access Memory), or a flash memory, and stores a program and data used for processing. The memory 1002 can be used as the frame memory 614 or the frame memory 624 of FIG.

The CPU 1001 (processor) operates as the selection unit 411 and the coding unit 412 in FIG. 4 by executing a program using, for example, the memory 1002. The CPU 1001 also operates as the selection unit 511 and the decoding unit 512 in FIG. 5 by executing the program using the memory 1002.

The CPU 1001 also operates as the selection unit 612 and the decoding unit 613 in FIG. 6 by executing the program using the memory 1002. The CPU 1001 also operates as the video generation unit 621, the selection unit 622, and the coding unit 623 of FIG. 6 by executing the program using the memory 1002.

The input device 1003 is, for example, a keyboard, a pointing device, or the like, and is used for inputting instructions or information from a user or an operator. The output device 1004 is, for example, a display device, a printer, a speaker, or the like, and is used for making an inquiry to a user or an operator and outputting a processing result. The input device 1003 and the output device 1004 can be used as the input unit 611 and the display unit 615 of FIG. 6, respectively. The processing result may be a decoded image displayed on the screen.

The auxiliary storage device 1005 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The auxiliary storage device 1005 may be a hard disk drive or a flash memory. The information processing device can store programs and data in the auxiliary storage device 1005 and load them into the memory 1002 for use.

The medium driving device 1006 drives the portable recording medium 1009 and accesses the recorded contents. The portable recording medium 1009 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, or the like. The portable recording medium 1009 may be a CD-ROM (Compact Disk Read Only Memory), a DVD (Digital Versatile Disk), or a USB (Universal Serial Bus) memory. The user or operator can store the programs and data in the portable recording medium 1009 and load them into the memory 1002 for use.

In this way, the computer-readable recording medium that stores the programs and data used for processing is physical (non-temporary) recording such as memory 1002, auxiliary storage device 1005, and portable recording medium 1009. It is a medium.

The network connection device 1007 is a communication interface circuit that is connected to the communication network 603 and performs data conversion associated with communication. The network connection device 1007 can be used as the communication unit 616 or the communication unit 625 of FIG. The information processing device can receive programs and data from an external device via the network connection device 1007, load them into the memory 1002, and use them.

The information processing device does not have to include all the components of FIG. 10, and some components may be omitted depending on the application or conditions. For example, when the information processing device is the server 602 and the interface with the user or the operator is unnecessary, the input device 1003 and the output device 1004 may be omitted. Further, when the portable recording medium 1009 is not used, the medium driving device 1006 may be omitted.

Having described in detail the embodiments of the disclosure and its advantages, those skilled in the art will be able to make various changes, additions and omissions without departing from the scope of the invention as expressly stated in the claims. Let's do it.

The following additional notes will be further disclosed with respect to the embodiments described with reference to FIGS. 2 to 10.
(Appendix 1)
Based on the intra-prediction mode applied to the coded block in the image, the code is selected from the first image format and the second image format having an amount of information smaller than the amount of information of the first image format. A selection unit that selects the third image format applied to the coding process of the block to be converted, and
A coding unit that encodes the coded block by intra-predictive coding using the third image format, and a coding unit.
An image coding apparatus comprising.
(Appendix 2)
The selection unit selects the first image format as the third image format when the intra prediction mode represents an angle prediction, and the second image format when the intra prediction mode represents an intra prediction other than the angle prediction. The image coding apparatus according to Appendix 1, wherein the image format is selected as the third image format.
(Appendix 3)
When the intra prediction mode represents angle prediction, the selection unit uses the first image format or the second image based on the pixel values of pixels in adjacent blocks adjacent to the coded block in the image. The appendix 1 is characterized in that when the format is selected as the third image format and the intra prediction mode represents an intra prediction other than the angle prediction, the second image format is selected as the third image format. Image encoding device.
(Appendix 4)
The selection unit calculates the edge strength using the pixel values of the pixels in the adjacent block, and when the edge strength is larger than the threshold value, the first image format is selected as the third image format, and the edge The image coding apparatus according to Appendix 3, wherein when the intensity is smaller than the threshold value, the second image format is selected as the third image format.
(Appendix 5)
When the intra prediction mode is applied to the coded block, the selection unit selects the first image format or the second image format as the third image format, and the coding unit selects the first image format or the second image format as the third image format. The block to be encoded is encoded by intra-predictive coding using the three image formats.
When the inter-prediction mode is applied to the coded block, the selection unit selects the second image format as the third image format, and the coding unit uses the third image format. The image coding apparatus according to any one of Supplementary note 1 to 4, wherein the coded block is encoded by predictive coding.
(Appendix 6)
Decoding of the decoding target block from the first image format and the second image format having an information amount smaller than the information amount of the first image format based on the intra prediction mode of the decoding target block in the image. A selection section that selects the third image format applied to the process,
A decoding unit that decodes the code information of the decoding target block by intra-predictive decoding using the third image format, and a decoding unit.
An image decoding device comprising.
(Appendix 7)
The selection unit selects the first image format as the third image format when the intra prediction mode represents an angle prediction, and the second image format when the intra prediction mode represents an intra prediction other than the angle prediction. The image decoding apparatus according to Appendix 6, wherein the image format is selected as the third image format.
(Appendix 8)
When the intra prediction mode represents angle prediction, the selection unit determines the first image format or the second image format based on the pixel values of the pixels in the adjacent block adjacent to the decoding target block in the image. 6 is selected as the third image format, and when the intra prediction mode represents an intra prediction other than the angle prediction, the second image format is selected as the third image format. Decryptor.
(Appendix 9)
The selection unit calculates the edge strength using the pixel values of the pixels in the adjacent block, and when the edge strength is larger than the threshold value, the first image format is selected as the third image format, and the edge The image decoding apparatus according to Appendix 8, wherein when the intensity is smaller than the threshold value, the second image format is selected as the third image format.
(Appendix 10)
When the intra prediction mode is applied to the decoding target block, the selection unit selects the first image format or the second image format as the third image format, and the decoding unit selects the third image format. The code information of the decoding target block is decoded by intra-predictive decoding using the format.
When the inter-prediction mode is applied to the decoding target block, the selection unit selects the second image format as the third image format, and the decoding unit uses the third image format for inter-prediction decoding. The image decoding apparatus according to any one of Supplementary note 6 to 9, wherein the code information of the decoding target block is decoded by the image decoding device.
(Appendix 11)
Based on the intra-prediction mode applied to the coded block in the image, the code is selected from the first image format and the second image format having an amount of information smaller than the amount of information of the first image format. A first selection unit that selects a third image format applied to the coding process of the block to be converted, and
A coding unit that encodes the coded block by intra-predictive coding using the third image format and generates code information of the coded block.
A second selection unit that selects the third image format from the first image format and the second image format based on the intra-prediction mode of the encoded block.
A decoding unit that decodes the code information of the encoded block by intra-predictive decoding using the third image format.
An image processing system characterized by being equipped with.

101, 601 Client terminal 102, 602 Server 103, 603 Communication network 301 Still image area 302, 303 Video area 401 Image encoding device 411, 511, 612, 622 Selection unit 412, 623 Coding unit 501 Image decoding device 512, 613 Decoding unit 611 Input unit 614, 624 Frame memory 615 Display unit 616, 625 Communication unit 621 Image generation unit 701 Coded block 702 Left adjacent block 703 Upper adjacent block 711 Upper left pixel 712 Left adjacent pixel 713 Upper adjacent pixel 1001 CPU
1002 Memory 1003 Input device 1004 Output device 1005 Auxiliary storage device 1006 Media drive device 1007 Network connection device 1008 Bus 1009 Portable recording medium

Claims (7)

Based on the intra-prediction mode applied to the coded block in the image, the code is selected from the first image format and the second image format having an amount of information smaller than the amount of information of the first image format. A selection unit that selects the third image format applied to the coding process of the block to be converted, and
A coding unit that encodes the coded block by intra-predictive coding using the third image format, and a coding unit.
An image coding apparatus comprising. The selection unit selects the first image format as the third image format when the intra prediction mode represents an angle prediction, and the second image format when the intra prediction mode represents an intra prediction other than the angle prediction. The image coding apparatus according to claim 1, wherein the image format is selected as the third image format. When the intra prediction mode represents angle prediction, the selection unit uses the first image format or the second image based on the pixel values of pixels in adjacent blocks adjacent to the coded block in the image. The first aspect of claim 1, wherein when the format is selected as the third image format and the intra prediction mode represents an intra prediction other than the angle prediction, the second image format is selected as the third image format. Image encoding device. The selection unit calculates the edge strength using the pixel values of the pixels in the adjacent block, and when the edge strength is larger than the threshold value, the first image format is selected as the third image format, and the edge The image coding apparatus according to claim 3, wherein when the intensity is smaller than the threshold value, the second image format is selected as the third image format. When the intra prediction mode is applied to the coded block, the selection unit selects the first image format or the second image format as the third image format, and the coding unit selects the first image format or the second image format as the third image format. The block to be encoded is encoded by intra-predictive coding using the three image formats.
When the inter-prediction mode is applied to the coded block, the selection unit selects the second image format as the third image format, and the coding unit uses the third image format. The image coding apparatus according to any one of claims 1 to 4, wherein the coded block is encoded by predictive coding. Decoding of the decoding target block from the first image format and the second image format having an information amount smaller than the information amount of the first image format based on the intra prediction mode of the decoding target block in the image. A selection section that selects the third image format applied to the process,
A decoding unit that decodes the code information of the decoding target block by intra-predictive decoding using the third image format, and a decoding unit.
An image decoding device comprising. Based on the intra-prediction mode applied to the coded block in the image, the code is selected from the first image format and the second image format having an amount of information smaller than the amount of information of the first image format. A first selection unit that selects a third image format applied to the coding process of the block to be converted, and
A coding unit that encodes the coded block by intra-predictive coding using the third image format and generates code information of the coded block.
A second selection unit that selects the third image format from the first image format and the second image format based on the intra-prediction mode of the encoded block.
A decoding unit that decodes the code information of the encoded block by intra-predictive decoding using the third image format.
An image processing system characterized by being equipped with.

Images