JP2007288761A - Image distribution system, image processing apparatus and method, distribution device and method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To distribute an image so as to be appropriately displayed even if the distribution image has a data amount beyond the bandwidth of a transmission line. <P>SOLUTION: In a display device, an image quality deteriorated region caused by a tap structure is excluded from a converted image D21'-1 obtained by applying image conversion processing due to arithmetic using a tap coefficient to a transmission image, a converted image D21"-1 in the remaining high image quality region is composed at a position corresponding to the position on the distribution image, and a display image is generated. In a distribution device, an image in which converted images D21" are composed adjacently to each other in the display device, is extracted from the distribution image as a transmission image D21 and transmitted to the display device. The present invention may be applicable to a server which distributes an image. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image distribution system, an image processing apparatus and method, a distribution apparatus and method, and a program, and in particular, when a distribution image transmitted via a network has a data amount exceeding the transmission band of the network. The present invention relates to a suitable image distribution system, image processing apparatus and method, distribution apparatus and method, and program.

  FIG. 1 shows a configuration example of a conventional image distribution system (see Patent Document 1). In this image distribution system, a distribution image D1 that can be displayed on the display device 2 and exceeds the transmission band of the network 3 is divided into transmission images D2 that can be transmitted in the transmission band of the network 3, It is transmitted to the display device 2 via the network 3.

  For example, an image signal supplied from a camera or a hard disk recorder (not shown) is input to the distribution device 1 as an image signal of the distribution image D1.

  Here, the distribution image D1 is an SHD (Super High Definition) image having an image size of 3840 × 2160 pixels and higher image quality than an HD (High Definition) image as shown in FIG.

  The distribution apparatus 1 divides each frame of the distribution image D1 corresponding to the input image signal into an image having a data amount that can be transmitted by the network 3.

  Specifically, in this example, since the network 3 has a transmission band capable of transmitting HD images, each frame of the SHD image D1 as a distribution image is converted into an HD image (1920) as shown in FIG. Divided into four equal parts into transmission images D2-1 to D2-4 corresponding to (× 1080 pixels) (hereinafter simply referred to as image D2 when there is no need to distinguish them individually). Is done.

  The distribution apparatus 1 encodes an image obtained as a result of dividing the distribution image D1 as a transmission image D2 based on the MPEG2 standard, packetizes it, and transmits it to the display apparatus 2 via the network 3.

  In the example of FIG. 3, the four transmission images D2-1 to D2-4 divided from one frame are encoded and transmitted to the display device 2.

  The display device 2 performs image conversion processing by calculation using tap coefficients obtained by learning performed on each transmission image D2 (FIG. 4A) transmitted from the distribution device 1 via the network 3. 4B, the image is converted into a high-quality image (hereinafter referred to as a converted image) D2 ′ having the same image size.

  FIG. 5 shows a configuration example of an image conversion unit that performs an image conversion process of the display device 2 for converting the transmission image D2 into the converted image D2 '.

  The tap extraction unit 11 converts the transmission image D2 received by the display device 2 (in this example, the transmission image D2 that is a 1/4 HD image of the SHD image D1) to a higher level than that obtained. A pixel constituting a high-definition HD image (this high-quality image is an image to be obtained from now on and is virtually assumed since it does not exist at this stage) is sequentially set as a pixel of interest, and further, Prediction taps that are pixel values of a plurality of pixels used to predict the pixel value of the target pixel are extracted from the transmission image D2 and supplied to the prediction unit 15.

  Specifically, the tap extraction unit 11 corresponds to the pixel on the transmission image D2 corresponding to the pixel of interest (for example, on the transmission image D2 on which the same subject portion as that shown in the position of the pixel of interest is reflected). For example, pixel values of a plurality of pixels having a spatial positional relationship as shown in FIG. 6 are extracted as prediction taps.

  In the example of FIG. 6, the pixel value of the pixel on the transmission image D <b> 2 corresponding to the target pixel W, the pixel located at a position away from the pixel by M pixels in the vertical direction, and only N pixels in the horizontal direction. A pixel value of a pixel at a distant position is extracted as a prediction tap.

  The tap extraction unit 12 corresponds to a pixel on the transmission image D2 corresponding to the pixel of interest (for example, a pixel on the transmission image D2 in which the same subject portion is reflected at the position of the pixel of interest). The pixel values of a plurality of pixels having a spatial positional relationship as shown in FIG. 6 are extracted as class taps and supplied to the class classification unit 13.

  In order to simplify the description, the prediction tap and the class tap have a plurality of pixels having the same tap structure, that is, a plurality of pixels having the same positional relationship corresponding to the position corresponding to the target pixel. The pixel value of the pixel is adopted. However, the prediction tap and the class tap can have different tap structures.

  The class classification unit 13 classifies the target pixel based on the class tap from the tap extraction unit 12 and supplies the class code corresponding to the class obtained as a result to the coefficient memory 14.

  Here, as a method of classifying, for example, ADRC (Adaptive Dynamic Range Coding) can be adopted. In the method using ADRC, the pixel values of the pixels constituting the class tap are subjected to ADRC processing, and the class of the target pixel is determined according to the ADRC code obtained as a result.

  The coefficient memory 14 stores tap coefficients for each class that are obtained in advance by learning between a high-quality image and a corresponding low-quality image. That is, the coefficient memory 14 stores a tap coefficient for each of a plurality of classes in which the target pixel can be classified by the class classification unit 13.

  The coefficient memory 14 supplies a class representing the class code supplied from the class classification unit 13 among the tap coefficients for each class, that is, the tap coefficient of the class of the target pixel to the prediction unit 15.

  The prediction unit 15 uses the prediction tap supplied from the tap extraction unit 11 and the tap coefficient supplied from the coefficient memory 14 to perform a predetermined prediction calculation for obtaining a true prediction value of the target pixel. Thereby, the prediction unit 15 obtains the pixel value (predicted value) of the pixel of interest, that is, the pixel value of the pixel of the HD image (converted image D2 ') with higher image quality.

  The transmission image D2 is converted into a high-quality converted image D2 'by such an image conversion unit.

  Returning to FIG. 1, the display device 2 combines the high-quality converted image D2 ′ obtained as a result of the above-described image conversion processing on the transmission image D2 with a position corresponding to the position on the distribution image D1 of the corresponding transmission image D2. Then, a display image is generated and displayed on a display screen of 4096 × 2160 pixels (ie, a screen for 4K2K), for example.

  In the case of this example, as shown in FIG. 7, converted images D2′-1 to four transmission images D2-1 to D2-4 (FIG. 3) divided from one frame of an SHD image D1 as a distribution image. D2′-4 is synthesized at a position corresponding to the position on the SHD image D1 of the corresponding transmission image D2 to generate a display image, which is displayed on the display screen.

JP 2002-162953 A

  By the way, as described above, the image conversion process in the display device 2 is performed using a prediction tap and a class tap having a predetermined spatial tap structure. In this case, depending on the position of the pixel of interest, a necessary prediction tap may be used. Alternatively, there is a case where the class tap cannot be obtained and the pixel value of the target pixel cannot be appropriately predicted.

  When, for example, the transmission image D2-1 shown in FIG. 3 is converted with the prediction tap and the class tap having the tap structure shown in FIG. 6, the pixels on the transmission image D2-1 corresponding to the target pixel W are as shown in FIG. In addition, when N pixels are not separated from the right end on the transmission image D2-1, there is no pixel at the position on the transmission image D2-1 corresponding to the black circle in the figure. Since the pixel value of the target pixel is generated without obtaining the class tap, the joint portion of the image synthesis near the center of the screen may be qualitatively noticeable.

  That is, when the prediction tap and the class tap having the tap structure shown in FIG. 6 are used, the pixels on the transmission image D2 corresponding to the target pixel are not separated from the left and right ends of the transmission image D2 by N pixels, or from both the upper and lower ends. When M pixels are not separated, a class tap sufficient for the pixel value of the target pixel cannot be obtained unless special processing such as complementing the class tap that has not been obtained is performed. The image quality of the hatched area in FIG. 4B of the converted image D2 ′ is deteriorated.

  As a result, the display image displayed on the display device 2 includes a region where the image quality is deteriorated (a region indicated by hatching in FIG. 7).

  Although it is possible to delete the area where the image quality has deteriorated so as not to be displayed, as shown in FIG. 9, even if the area near the image frame is deleted and not displayed, as shown in FIG. A portion that passes through the center of the display image in the vertical and horizontal directions remains. Also, as shown in FIG. 10, if the area is further deleted so as not to be displayed, the image of that portion of the distribution image D1 becomes a discontinuous display image, resulting in an unnatural image.

  The present invention has been made in view of such a situation, and even when a distribution image has a data amount exceeding the transmission band, it can be distributed so that it is appropriately displayed. It is what you want to do.

  An image distribution system according to a first aspect of the present invention includes a distribution device that distributes a distribution image by transmitting a transmission image constituting the distribution image via a network, and transmits the distribution image from the distribution device via the network. In an image distribution system comprising an image processing device that generates a display image corresponding to the distribution image from the transmitted image that has been transmitted, the image processing device receives the transmission image transmitted from the distribution device. A conversion means for converting the transmission image received by the reception means into a high-quality image by performing an image conversion process by a calculation using a tap coefficient obtained by learning in advance, and the conversion means From the image obtained as a result of the image conversion process, depending on the tap structure of the tap used in the image conversion process Generation means for generating a display image by excluding an area where quality deteriorates and combining the remaining high-quality area at a position corresponding to a position on the distribution image of a corresponding area on the transmission image The distribution apparatus is configured to input the distribution image and to transmit the image in the transmission band of the network, and to perform the image conversion processing by the conversion means of the image processing apparatus. An image that is synthesized adjacently by the image quality region as the transmission image, an extraction unit that extracts the image from the distribution image, and the transmission image extracted by the extraction unit via the network. Transmission means for transmitting to the image processing apparatus.

  In the image distribution system according to the first aspect of the present invention, the transmission image transmitted from the distribution device is received by the image processing device, and the received transmission image is obtained by learning in advance. The image is converted into a high-quality image by performing image conversion processing by calculation using a tap coefficient, and the image quality varies depending on the tap structure of the tap used in the image conversion processing from the image obtained as a result of the image conversion processing. The deteriorated area is eliminated, and the remaining high-quality area is synthesized with a position corresponding to the position on the distribution image of the corresponding area on the transmission image, and the display image is generated. The distribution image is input and can be transmitted in the transmission band of the network, and the image conversion process of the image processing apparatus is performed, so that the high-quality area is adjacent. Image that becomes synthesis Te is, as the transmission image, the extracted from the distribution image, extracted the transmission image is transmitted to the image processing apparatus via the network.

  The conversion means of the image processing apparatus extracts a feature amount of the entire distribution image constituted by the transmission image from the transmission image, and executes the image conversion process using the extracted whole feature amount can do.

  An image processing apparatus according to a second aspect of the present invention includes: a receiving unit that receives a transmission image that constitutes a distribution image transmitted from a distribution device; and a learning that is performed in advance on the transmission image received by the receiving unit. A conversion unit that converts the image into a high-quality image by performing an image conversion process based on an operation using the obtained tap coefficient, and is used in the image conversion process from an image obtained as a result of the image conversion process by the conversion unit. The area where the image quality is deteriorated according to the tap structure of the tap to be removed is excluded, and the remaining high-quality area is synthesized with the position corresponding to the position on the distribution image of the corresponding area on the transmission image. Generating means for generating.

  The conversion means can extract a feature amount of the entire distribution image constituted by the transmission image from the transmission image, and execute the image conversion process using the extracted whole feature amount.

  An image processing method or program according to a second aspect of the present invention includes a reception step of receiving a transmission image constituting a distribution image transmitted from a distribution device, and the transmission image received in the processing of the reception step. A conversion step for converting the image into a high-quality image by performing an image conversion process using a tap coefficient obtained by learning in advance, and using the image obtained as a result of the image conversion process in the image conversion process The area where the image quality is deteriorated according to the tap structure of the tap to be removed is excluded, and the remaining high-quality area is synthesized with the position corresponding to the position on the distribution image of the corresponding area on the transmission image. Generating step.

  In the image processing apparatus, the image processing method, or the program according to the second aspect of the present invention, a transmission image constituting the distribution image transmitted from the distribution apparatus is received, and the received transmission image is learned beforehand. The image is converted into a high-quality image by performing an image conversion process by calculation using the obtained tap coefficients, and the image obtained as a result of the image conversion process is used in accordance with the tap structure of the tap used in the image conversion process. Thus, the area where the image quality deteriorates is eliminated, and the remaining high-quality area is combined with the corresponding area on the transmission image at a position corresponding to the position on the distribution image to generate a display image.

  The third distribution device of the present invention converts an image constituting the distribution image into a high-quality image by performing image conversion processing by calculation using a tap coefficient obtained by learning in advance, and obtains the result. An area in which image quality deteriorates according to the tap structure of the tap used in the image conversion process is excluded from the obtained image, and the remaining high-quality area is replaced with the distribution image of the image area constituting the corresponding distribution image. In the distribution apparatus that distributes the distribution image by transmitting the image constituting the distribution image to the image processing apparatus that generates the display image by combining with a position corresponding to the upper position, An input means for inputting a distribution image, and the image quality can be transmitted in the transmission band of the network, and the image conversion process is performed so that the high-quality region is adjacent to the distribution unit. The image to be formed is extracted from the distribution image as an image constituting the distribution image, and the image extracted by the extraction unit is transmitted to the image processing apparatus via the network. Transmission means.

  The third distribution method or program according to the present invention converts an image constituting the distribution image into a high-quality image by performing image conversion processing by calculation using a tap coefficient obtained by learning in advance. From the image obtained as a result, an area where the image quality deteriorates in accordance with the tap structure of the tap used in the image conversion process is excluded, and the remaining high image quality area is replaced with the image area constituting the corresponding distribution image. A distribution method for distributing the distribution image by transmitting an image constituting the distribution image via a network to an image processing apparatus that generates the display image by combining with a position corresponding to a position on the distribution image; In the program for causing the computer to execute the distribution process, an input step for inputting the distribution image and transmission in the transmission band of the network are possible. And an extraction step of extracting, from the distribution image, an image in which the high-quality area is adjacently synthesized by performing the image conversion process as an image constituting the distribution image; And a transmission step of transmitting the image extracted by the processing of the extraction step to the image processing apparatus via the network.

  In the third distribution apparatus, distribution method, or program of the present invention, an image conversion process is performed on an image constituting the distribution image by an operation using a tap coefficient obtained by learning in advance to obtain a high-quality image. The image obtained by converting and removing the area where the image quality deteriorates according to the tap structure of the tap used in the image conversion process from the image obtained as a result, and the remaining high-quality area is an image constituting the corresponding distribution image The image constituting the distribution image is transmitted via the network to an image processing apparatus that generates the display image by combining the position corresponding to the position on the distribution image in the region of the distribution image to distribute the distribution image. So that the image can be transmitted in the transmission band of the network and the image conversion process is performed so that the high-quality area is synthesized adjacently. That image is an image that constitutes the distribution image, the extracted from the distribution image, extracted the image is transmitted to the image processing apparatus via the network.

  According to the first to third aspects of the present invention, even when a distribution image has a data amount exceeding the transmission band, it can be distributed so that it is appropriately displayed.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments of the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

The image delivery system according to the first aspect of the present invention is:
The transmission image (for example, the transmission image D21 in FIG. 11) constituting the distribution image (for example, the distribution image D1 in FIG. 11) is transmitted via a network (for example, the network 3 in FIG. 11). A distribution device for distribution (for example, the distribution device 51 in FIG. 11) and an image processing device for generating a display image corresponding to the distribution image from the transmission image transmitted from the distribution device via the network (for example, In the image distribution system comprising the display device 52) of FIG.
The image processing apparatus includes:
Receiving means for receiving the transmission image transmitted from the distribution device (for example, the receiving unit 71 in FIG. 18);
Conversion means for converting the transmission image (for example, FIG. 19A) received by the reception means into a high-quality image by performing image conversion processing by calculation using tap coefficients obtained in advance by learning. , The high image quality generation unit 73) of FIG.
From the image (for example, FIG. 22A) obtained as a result of the image conversion process by the conversion unit, an area where the image quality deteriorates according to the tap structure of the tap used in the image conversion process (for example, hatched lines in FIG. 22A). And the remaining high-quality area (for example, FIG. 22B) is combined with a position corresponding to the position on the distribution image of the corresponding area on the transmission image to display the display image (for example, 24) and generating means (for example, the display control unit 74 in FIG. 18) for generating,
The distribution device includes:
Input means (for example, the input unit 61 in FIG. 12) for inputting the distribution image;
Transmission is possible in the transmission band of the network, and the image conversion process by the conversion unit of the image processing apparatus is performed, so that the high-quality area is synthesized adjacently by the generation unit. Extraction means (for example, the generation unit 62 in FIG. 12) that extracts an image (for example, the transmission image D21 in FIG. 17) from the distribution image as the transmission image;
Transmission means for transmitting the transmission image extracted by the extraction means to the image processing apparatus via the network (for example, the communication unit 64 in FIG. 12).

  The conversion means of the image processing device extracts the feature amount of the entire distribution image constituted by the transmission image from the transmission image (for example, the whole feature amount extraction unit 81 in FIG. 20), and extracts the whole The image conversion process can be executed using the feature amount (for example, the image conversion unit 82 in FIG. 20).

The second image processing apparatus of the present invention is
Receiving means (for example, the receiving unit 71 in FIG. 18) for receiving a transmission image constituting the distribution image transmitted from the distribution device;
Conversion means for converting the transmission image received by the receiving means into a high-quality image by performing image conversion processing by calculation using tap coefficients obtained in advance by learning (for example, the high-quality image in FIG. 18) Conversion unit 73);
From the image obtained as a result of the image conversion process by the conversion means, an area in which image quality deteriorates according to the tap structure of the tap used in the image conversion process is excluded, and the remaining high-quality area is transferred to the corresponding transmission. Generation means (for example, the display control unit 74 in FIG. 18) that generates a display image by combining the region on the image with a position corresponding to the position on the distribution image.

An image processing method or a program according to the second aspect of the present invention includes:
A receiving step (for example, processing of the receiving unit 71 in FIG. 18) for receiving a transmission image constituting the distribution image transmitted from the distribution device;
A conversion step for converting the transmission image received in the reception step processing into a high-quality image by performing image conversion processing by calculation using tap coefficients obtained in advance by learning (for example, in FIG. Processing of the high image quality generation unit 73),
From the image obtained as a result of the image conversion process, an area in which the image quality deteriorates according to the tap structure of the tap used in the image conversion process is excluded, and the remaining high-quality area is defined as a corresponding area on the transmission image. And a generating step (for example, processing of the display control unit 74 in FIG. 18) for generating a display image by combining with a position corresponding to the position on the distribution image.

The third distribution device of the present invention is:
The image constituting the distribution image is converted into a high-quality image by performing image conversion processing by calculation using tap coefficients obtained in advance by learning, and the resulting image is converted into the image conversion processing. The area where the image quality deteriorates according to the tap structure of the tap to be used is excluded, and the remaining high-quality area is combined with the position corresponding to the position on the distribution image of the image area constituting the corresponding distribution image. In the distribution apparatus that distributes the distribution image by transmitting the image constituting the distribution image to the image processing apparatus that generates the display image via a network,
Input means for inputting the distribution image (for example, the input unit 61 of FIG. 12);
As an image constituting the distribution image, an image that can be transmitted in the transmission band of the network and is subjected to the image conversion process so that the high-quality area is adjacently synthesized is the image Extraction means for extracting from the distribution image (for example, the generation unit 62 in FIG. 12);
Transmission means (for example, the communication unit 64 in FIG. 12) that transmits the image extracted by the extraction means to the image processing apparatus via the network.

The third delivery method or program of the present invention is:
The image constituting the distribution image is converted into a high-quality image by performing image conversion processing by calculation using a tap coefficient obtained by learning in advance, and the resulting image is converted into the image conversion processing. The area where the image quality deteriorates according to the tap structure of the tap to be used is excluded, and the remaining high-quality area is combined with the position corresponding to the position on the distribution image of the image area constituting the corresponding distribution image. The computer executes a distribution method for distributing the distribution image or a distribution process for distributing the distribution image by transmitting the image constituting the distribution image to the image processing apparatus that generates the display image via a network. In the program to let
An input step for inputting the distribution image (for example, processing of the input unit 61 in FIG. 12);
As an image constituting the distribution image, an image that can be transmitted in the transmission band of the network and is subjected to the image conversion process so that the high-quality region is adjacently synthesized is the image An extraction step (for example, processing of the generation unit 62 in FIG. 12) for extracting from the distribution image;
A transmission step (for example, processing of the communication unit 64 in FIG. 12) that transmits the image extracted in the processing of the extraction step to the image processing apparatus via the network.

  FIG. 11 shows a configuration example of an image distribution system to which the present invention is applied. In this image distribution system, a transmission image D21 that can be displayed on the distribution destination display device 52 and is generated as described later from a distribution image D1 having a data amount exceeding the transmission band of the network 3 is transmitted. The data is transmitted from the distribution device 51 to the display device 52 via the network 3.

  For example, an image signal supplied from a camera (not shown) or a hard disk record is input to the distribution device 51 as an image signal of the distribution image D1.

  Here, the distribution image D1 is an SHD image having an image size of 3840 × 2160 pixels and a higher image quality than the HD image as shown in FIG. The high image quality here means, for example, high resolution and low noise.

  The distribution device 51 extracts an image having a data amount that can be transmitted by the network 3 from each frame of the distribution image D1 corresponding to the input image signal, and uses the extracted image as a transmission image D21. The data is transmitted to the display device 52 via the network 3.

  The display device 52 performs an image conversion process on each transmission image D21 transmitted from the distribution device 51 via the network 3 by calculation using a tap coefficient obtained by learning performed in advance, and each transmission image D21 is converted into a higher quality image.

  The display device 52 generates a display image by combining the image obtained by converting the transmission image D21 with higher image quality into a position corresponding to the position on the distribution image D1 of the corresponding transmission image D21, and generates the display image, for example, 4096 × Display on a 2160-pixel display (that is, a screen for 4K2K).

  FIG. 12 shows a configuration example of the distribution apparatus 51.

  The input unit 61 inputs an image signal of a distribution image (SHD image in this example) D1 supplied from a camera or the like and supplies the image signal to the generation unit 62.

  The generation unit 62 has a data amount that can be transmitted via the network 3 from each frame of the SHD image D1 corresponding to the image signal supplied from the input unit 61, and a tap structure in image conversion processing in the display device 52. The image of the area corresponding to the is extracted. The image extracted here is a transmission image D21.

  Specifically, when image conversion is performed with the prediction tap and the class tap having the tap structure shown in FIG. 6 on the display device 52, the upper left corner of the frame shown in FIG. , 0), 1920 × 1080 pixels with (N, M), (1920 + N, M), (1920 + N, 1080 + M), and (N, 1080 + M) pixels as the four corners The image D21-1 of the area of (1920-N, M), (3840-N, M), (3840-N, 1080 + M), and (1920-N, 1080 + M) shown in FIG. Image D21-2 of a 1920 × 1080 pixel area with four corners as pixels, (1920-N, 1080-M), (3840-N, 1080-M), (3840-N, 2160-M) shown in FIG. ), And (1920-N, 2160-M) pixels D21-3 in an area of 1920 × 1080 pixels having four corners, and (N, 1080-M), (1920 + N, 1080) shown in FIG. -M), (1920 + N, 2160-M), and (N, 2160-M) image D in an area of 1920 × 1080 pixels having four corners 21-4 are respectively extracted and become transmission images D21.

  That is, in the distribution device 51, the transmission image D21- of the region having the same size as each region obtained by dividing the SHD image D1 and the position on the SHD image D1 translated in the center direction by the number of pixels corresponding to the tap structure. 1 to D21-4 are extracted.

  The transmission images D21-1 to D21-4 shown in FIGS. 13 to 16 are collectively shown in FIG. In FIG. 17, the transmission images D21-1 to D21-4 are shown with different oblique lines.

  Returning to FIG. 12, the generation unit 62 supplies the transmission image D <b> 21 extracted as described above to the data processing unit 63.

  The data processing unit 63 performs image compression encoding processing according to the MPEG2 standard on the transmission image D21 supplied from the generation unit 62, and transmits the image data of the transmission image D21 obtained as a result to the communication unit 64. Supply.

  The communication unit 64 transmits the image data of the transmission image D21 supplied from the data processing unit 63 to the display device 52 via the network 3 for each frame of the corresponding SHD image D1.

  FIG. 18 shows a configuration example of the display device 52.

  The receiving unit 71 receives the transmission image D <b> 21 transmitted from the distribution device 51 via the network 3 and supplies it to the data processing unit 72.

  The data processing unit 72 performs an image decompression decoding process based on the MPEG2 standard on each transmission image D21 supplied from the reception unit 71, restores the original transmission image D21, and supplies it to the high image quality generation unit 73. .

  The high image quality generation unit 73 performs an image conversion process on the transmission image D21 (FIG. 19A) supplied from the data processing unit 72 by an operation using a tap coefficient obtained by learning performed in advance, and thereby the image quality in FIG. As shown, a high-quality image (hereinafter referred to as a converted image) D21 ′ having the same image size is generated and supplied to the display control unit 74.

  FIG. 20 shows a configuration example of the high image quality generation unit 73.

  The image data of the transmission image D21 restored by the data processing unit 72 is supplied to the entire feature amount extraction unit 81 for each frame of the corresponding SHD image D1. The whole feature amount extraction unit 81 extracts a feature amount as a feature of the entire frame of the SHD image D1 from the supplied image data of a set of four transmission images D21 in this example.

  Specifically, for example, the average luminance value of each of a set of four transmission images D21-1 to D21-4 (FIG. 17) is calculated, and the average value of the average luminance values is used as the overall feature amount. The

  The total feature amount extraction unit 81 supplies the extracted total feature amount to the image conversion units 82-1 to 82-4, and converts each of the image data of the set of four transmission images D21 to the corresponding image conversion unit. 82-1 to 82-4.

  For example, the image data of the transmission image D21-1 is converted into the image conversion unit 82-1, the image data of the transmission image D21-2 is converted into the image conversion unit 82-2, and the image data of the transmission image D21-3 is converted into the image. The image data of the transmission image D21-4 is supplied to the unit 82-3 and the image conversion unit 82-4.

  The image conversion unit 82 performs an operation using the tap coefficient obtained by learning in advance on the image data of the input transmission image D21 (FIG. 19A) using the entire feature amount, thereby converting the high-quality converted image. D21 ′ (FIG. 19B) is generated.

  FIG. 21 shows a configuration example of the image conversion unit 82.

  The tap extraction unit 101 converts a transmission image D21 (HD image) to obtain a higher quality HD image (this high quality image is an image to be obtained from now on, and does not exist at this stage). Are assumed to be pixels of interest, and prediction taps that are pixel values of a plurality of pixels used for predicting pixel values of the pixels of interest are transmitted from the transmission image D21. Extracted and supplied to the prediction unit 105.

  Specifically, the tap extraction unit 101 corresponds to the pixel of interest on the transmission image D21 (for example, on the transmission image D21 in which the same subject portion that is reflected at the position of the pixel of interest is reflected). For example, pixel values of a plurality of pixels having a spatial positional relationship as shown in FIG. 6 are extracted as prediction taps.

  The tap extraction unit 102 corresponds to a pixel on the transmission image D21 corresponding to the pixel of interest (for example, a pixel on the transmission image D21 in which the same subject portion is reflected at the position of the pixel of interest). The pixel values of a plurality of pixels having a spatial positional relationship as shown in FIG. 6 are extracted as class taps and supplied to the class classification unit 103.

  Here, for the sake of simplicity of explanation, the prediction tap and the class tap are a plurality of pixels having the same positional relationship corresponding to the pixel value of the plurality of pixels having the same structure, that is, the position corresponding to the target pixel. The pixel value of this pixel is adopted. However, the prediction tap and the class tap can have different tap structures.

  The class classification unit 103 classifies the pixel of interest based on the class tap from the tap extraction unit 102 and the overall feature amount supplied from the overall feature amount extraction unit 81, and class codes corresponding to the resulting class are obtained. The coefficient memory 104 is supplied.

  Here, for example, ADRC can be adopted as a method of classifying.

  The coefficient memory 104 stores tap coefficients for each class that are obtained in advance by learning. That is, the coefficient memory 104 stores a tap coefficient for each of a plurality of classes in which the target pixel can be classified by the class classification unit 103.

  The coefficient memory 104 supplies the class representing the class code supplied from the class classification unit 103 among the tap coefficients for each class, that is, the tap coefficient of the class of the target pixel to the prediction unit 105.

  The prediction unit 105 calculates Equation (1) using the prediction tap supplied from the tap extraction unit 101 and the tap coefficient supplied from the coefficient memory 104, and obtains the true value prediction value of the target pixel. Thereby, the pixel value (predicted value) of the target pixel, that is, the pixel value of the pixel of the HD image with higher image quality is obtained.

  In equation (1), y represents the predicted value, xn represents the pixel value of the nth pixel constituting the prediction tap, and wn represents the nth tap coefficient multiplied by the nth pixel value. Represents.

  That is, since the image conversion unit 82 uses the prediction tap and the class tap having the tap structure shown in FIG. 6 in the image conversion process, similarly to the conventional image conversion unit shown in FIG. 5, the transmission image corresponding to the target pixel is used. When the pixel on D21 is a pixel that is not separated by N pixels from the left and right ends of the transmission image D21 or a pixel that is not separated by M pixels from the upper and lower ends, the class tap or the prediction tap that is not obtained is complemented. When special processing is not performed, the pixel value of the target pixel is not predicted appropriately, and the image quality of the hatched area in FIG. 19B of the converted image D21 ′ is deteriorated.

  Returning to FIG. 18, the display control unit 74 generates a display image by synthesizing the converted image D21 ′ supplied from the high image quality generation unit 73 and converted from the transmission image D21 to high image quality. An area where the image quality is deteriorated (area hatched in FIG. 22A) is deleted from D21 ′, and the converted image D21 ″ shown in FIG. 22B obtained as a result corresponds to the converted image D21 ″. A display image is generated by combining with a position corresponding to the position on the distribution image D1 of the region on the transmission image D21 to be displayed and displayed on the display 75.

  The transmission image D21-1 shown in FIG. 13 is a direction (rightward direction) adjacent to the transmission image D21-2 (FIG. 14) with respect to the upper left region obtained by equally dividing the SHD image D1 indicated by the broken line in FIG. The area for N pixels and the area for M pixels in the direction (downward) adjacent to the transmission image D21-4 (FIG. 16) are included as a margin area, and the area is subjected to image conversion processing. Since the image quality is deteriorated, as shown in FIGS. 23 and 24, a converted image D21′-1 obtained by image conversion of the transmission image D21-1 shown in FIG. 13 (indicated by a broken line in FIG. 23). The image D21 ″ -1 from which the area where the image quality is degraded is deleted from the SHD image D1 in the area on the transmission image D21-1 corresponding to the converted image D21 ″ -1, The upper left area of the SHD image D1 indicated by the broken line in FIG. It is synthesized in a position corresponding to the position.

  The transmission image D21-2 shown in FIG. 14 is a direction (left direction) adjacent to the transmission image D21-1 (FIG. 13) with respect to the upper right region obtained by equally dividing the SHD image D1 indicated by the broken line in FIG. The area for N pixels and the area for M pixels in the direction (downward) adjacent to the transmission image D21-3 (FIG. 15) are included as a margin area, and the area is subjected to image conversion processing. Since the image quality is deteriorated, as shown in FIG. 24, the area where the image quality is deteriorated is deleted from the converted image D21′-1 obtained as a result of image conversion of the transmission image D21-2 shown in FIG. The image D21 ″ -2 is obtained by dividing the SHD image D1 indicated by the broken line in FIG. 14 into four equal positions on the SHD image D1 in the region on the transmission image D21-2 corresponding to the converted image D21 ″ -2. Is synthesized at a position corresponding to the position of the upper right area.

  The transmission image D21-3 shown in FIG. 15 is a direction (left direction) adjacent to the transmission image D21-4 (FIG. 16) with respect to the lower right region obtained by equally dividing the SHD image D1 indicated by the broken line in FIG. ) Area for N pixels and an area for M pixels in the direction (upward direction) adjacent to the transmission image D21-2 (FIG. 14) as a margin area, and that area is an image conversion process. Therefore, as shown in FIG. 24, an area having a deteriorated image quality is deleted from the converted image D21′-3 obtained as a result of image conversion of the transmission image D21-3 shown in FIG. The image D21 ″ -3 obtained is the same as the position on the SHD image D1 in the region on the transmission image D21-3 corresponding to the converted image D21 ″ -3, that is, the SHD image D1 indicated by the broken line in FIG. It is synthesized at a position corresponding to the position of the divided lower right region.

  The transmission image D21-4 shown in FIG. 16 is a direction (right direction) adjacent to the transmission image D21-3 (FIG. 15) with respect to the lower left region obtained by equally dividing the SHD image D1 indicated by the broken line in FIG. The area for N pixels and the area for M pixels in the direction (upward direction) adjacent to the transmission image D21-1 (FIG. 13) are included as a margin area, and the area is subjected to image conversion processing. Since the image quality is deteriorated, as shown in FIG. 24, the area where the image quality is deteriorated is deleted from the converted image D21′-4 obtained as a result of image conversion of the transmission image D21-4 shown in FIG. The image D21 ″ -4 is obtained by dividing the position on the SHD image D1 in the region on the transmission image D21-4 corresponding to the converted image D21 ″ -4, that is, the SHD image D1 indicated by the broken line in FIG. Is synthesized at a position corresponding to the position of the lower right area.

  That is, in the image distribution system of FIG. 11, an image that can be transmitted through the network 3 is extracted from the distribution image D1 as the transmission image D21 by the distribution device 51 and distributed by transmitting the transmission image D21 via the network 3. The image D1 is distributed to the display device 52.

  Further, in the display device 52, an image quality degradation region (FIG. 22A) caused by the tap structure is obtained from the converted image D21 ′ (FIG. 22A) obtained by performing the image conversion process by the calculation using the tap coefficient on the transmission image D21. The converted image D21 ″ (FIG. 22B) of the high-quality area remaining after removing the hatched area is synthesized at a position corresponding to the position on the distribution image D1 to generate a display image (FIG. 23, FIG. 23). 24), the distribution device 51 divides the distribution image D1 into four by image conversion processing, that is, an image in which the converted images D21 ″ are adjacently synthesized (FIG. 24) on the display device 52. A transmission image D21 having a margin that can appropriately predict the pixel values of the pixels in the region (the region indicated by the broken line in FIGS. 13 to 16) is extracted from the distribution image D1 and displayed on the display device. 52 is transmitted to.

  As described above, the transmission image D21 constituting the distribution image D1 extracted from the distribution image D1 is transmitted via the network 3, so that the distribution image D1 is distributed. Even if the image exceeds the bandwidth, the distribution image D1 can be distributed via the network 3, and an appropriate display image D21 '' corresponding to the distribution image D1 (an image in which the continuity of the joint is maintained). ) Can be generated.

  Further, in the image conversion process for the transmission image D21 in the display device 52, the feature amount of the entire distribution image D1 is used from the transmission image D21, so that it is possible to prevent a qualitative discontinuous region from occurring.

  FIG. 25 illustrates a configuration example of a learning device that performs learning of tap coefficients used in image conversion processing.

  In this example, the learning data storage unit 151 stores an SHD image signal (SHD image) as learning data used for learning tap coefficients.

  Next, a learning device for creating (learning) prediction coefficients stored in the ROM table 158 will be described with reference to FIG. The learning device stores the prediction coefficient in the ROM table 158 by using image data with high image quality (high resolution and low noise) to be generated.

  The learning device, as shown in FIG. 25, learns by limiting the bandwidth of the high-quality image data and the learning data storage unit 151 that stores the high-quality image data to be generated (teaching image of the learning pair). A band limiting unit 152 that generates a pair of student images, an entire feature amount extracting unit 153 that extracts a feature amount of the entire band-limited image data, and pixel data (class tap) used for class code generation are A tap extraction unit 154 for extracting from the restricted image data, a class code generation unit 155 for generating a class code used for class classification based on the feature amount of the entire extracted image data and the waveform pattern of the class tap, and learning processing Tap extraction unit 156 for extracting pixel data (prediction taps) for use in image data from band-limited image data, and a normal equation for each class code Vertical, includes a learning unit 157 for determining the prediction coefficients for each of the class code, and a ROM table 158 for storing the determined predictive coefficients.

  In order to obtain the prediction coefficient by learning, first, it is necessary to generate band-limited image data from already known high-quality image data. As a band limiting method, for example, there is high-efficiency encoding such as MPEG encoding / decoding.

  The entire feature amount extraction unit 153 is supplied with the image data band-limited by the band limiting unit 152. The feature amount extraction unit 153 outputs a feature amount of the entire image data, for example, information indicating average luminance.

  The tap extracting unit 154 cuts out the image data from the band limiting unit 152 for each predetermined region, and acquires the waveform pattern of the class tap by performing ADRC processing or the like on the class tap that is the cut out image data.

  The tap extraction unit 156 has the same configuration as the tap extraction unit 154, and supplies image data cut out from a predetermined area to the learning unit 157.

  The class code generation unit 155 determines a class code based on the feature amount from the entire feature amount extraction unit 153 and the waveform pattern obtained by the ADRC process. As a result, the class code generation unit 155 generates a class code to which the block belongs, and supplies a class code indicating the class to the learning unit 157.

  The learning unit 157 includes each class code class supplied from the class code generation unit 155, image data x1, x2,..., Xn supplied from the tap extraction unit 156 for each class code class, and a learning data storage unit 151. A normal equation is established using the high-quality image data y supplied.

  Here, in order to explain the normal equation, learning of a conversion formula from a plurality of high-quality image data to normal image data and signal conversion using the prediction formula will be described.

  Hereinafter, for the sake of explanation, a case in which learning is more generalized and prediction using n samples is performed will be described. Assume that the levels of the image data are x1, x2,..., Xn, respectively, and the quantized data resulting from p-bit ADRC processing is q1,.

  At this time, as described above, when the image data levels are x1, x2,..., Xn and the image quality levels of high image quality are y, the prediction coefficients w1, w2,. Set an n-tap linear estimation formula. This is shown in equation (2). Before learning, wn is an undetermined coefficient.

  Learning is performed on a plurality of image data for each class. When the number of data samples is m, Equation (3) shown below is set according to Equation (2).

  When m> n, the prediction coefficients w1,..., wn are not uniquely determined, so that the element of the error vector e is defined by the following equation (4) and the prediction that minimizes the equation (5) Find the coefficient. This is a so-called least square method.

  Here, the partial differential coefficient according to wn in equation (5) is obtained. It is only necessary to determine each wn so that the following expression (6) is set to “0”.

  Hereinafter, when Xij and Yi are defined as in Expression (7) and Expression (8), Expression (6) is rewritten into Expression (9) using a matrix. This equation is generally called a normal equation.

  After the input of all the learning data is completed, the learning unit 157 establishes a normal equation shown in Expression (9) for each class code class.

The normal equation is solved for each wn by using a general matrix solving method such as a sweep-out method, and a prediction coefficient is calculated for each class. In other words, equation (9) is rewritten as equation (10), equation (11) is obtained by a matrix solution method such as a sweep-out method, and the determinant W of the prediction coefficient is calculated for each class code. Then, the learning unit 157 writes the calculated prediction coefficient in the ROM table 158.
X · W = Y (10)
W = X-1 Y (11)

  As a result of such learning, the ROM table 158 stores prediction coefficients that can be estimated statistically closest to the true value for estimating the high-quality image data y for each class. The table stored in the ROM table 158 is the coefficient memory 104 used in the high image quality generation unit 73 of the display device 52 to which the present invention is applied as described above. With this processing, the learning of the prediction coefficient for creating high-quality image data from the normal image data is completed by the linear estimation equation.

  As described above, the learning device is not band-limited by learning using the image data that has been band-limited in advance in consideration of the band-limiting by the data processing unit 63 occurring in the distribution device 51. A prediction coefficient for reproducing high-quality image data can be generated.

  In the description of the present embodiment, the ADRC process is adopted as the pattern generation means for generating the compressed data pattern. However, this is only an example, and the signal waveform pattern can be expressed by a small number of classes. Any information compression means may be provided, and for example, DPCM (predictive coding), VQ (vector quantization), or the like may be used.

  In this embodiment, the region is divided by the tap extraction circuit 154 and the tap extraction circuit 156. However, the present invention is not limited to this. For example, the region division may be performed by one circuit. Of course it is good.

  Further, n and m in the equations (1) to (8) and N and M in FIG. 4 indicate numbers used independently in each description.

  In the above description, the case where the distribution image D1 is an SHD image and the transmission image D21 is an HD image has been described as an example. However, the distribution image D1 is an image having a data amount exceeding the transmission band of the network 3, and the transmission image D21. Any other image can be used as the distribution image D1 or the transmission image D21 as long as the image is converted into a high-quality image and a display image is generated.

  FIG. 26 shows a configuration example of another image distribution system to which the present invention is applied. In this image distribution system, an image conversion device 201 is further provided before the distribution device 51 of the image distribution system in FIG.

  In the example of FIG. 11 described above, a distribution image D1 (in this example, an SHD image) that can be displayed on the display device 52 of the distribution destination is supplied to the distribution device 51, and the distribution device 51 uses the SHD image from the SHD image. The transmission image D21 (FIGS. 13 to 16) that can be transmitted in the transmission band of the network 3 and can be combined while maintaining the continuity is generated by the display device 52. The transmission image D21 is displayed on the distribution side by the display device 52. There may be a case where an image having an image size different from a possible SHD image, for example, an HD image is supplied.

  In the example of FIG. 26, an HD image having a different image size from the SHD image that can be displayed by the display device 52 is supplied to the distribution side as the distribution image D0.

  The image conversion apparatus 201 generates an SHD image (distribution image D1) having an image size that can be displayed on the display device 52 as a distribution destination from the HD image input as the distribution image D0, and supplies the SHD image to the distribution apparatus 51.

  As described above with reference to FIG. 13 to FIG. 16, the distribution device 51 can transmit in the transmission band of the network 3 for each frame of the SHD image D <b> 1 supplied from the image conversion device 201. Then, a transmission image D21 that can be synthesized while maintaining continuity is generated, and the generated transmission image D21 is transmitted to the display device 52 via the network 3.

  As described above, the display device 52 performs image conversion processing by calculation using tap coefficients obtained by learning performed in advance on each transmission image D21 transmitted from the distribution device 51 via the network 3. Each transmission image D21 is converted into a higher-quality image (for example, an image in which noise added at the time of transmission is removed).

  The display device 52 generates a display image by combining the image obtained by converting the transmission image D21 with higher image quality into a position corresponding to the position on the distribution image D1 of the corresponding transmission image D21, and generates the display image, for example, 4096 × Display on a 2160-pixel display (that is, a screen for 4K2K).

  FIG. 27 shows a configuration example of the image conversion apparatus 201.

  The tap extraction unit 211 extracts, as prediction taps, pixel values of a plurality of pixels having a spatial positional relationship as shown in FIG. 28 for a predetermined target pixel for each frame of the HD image supplied thereto. To do.

  In the example of FIG. 28, the pixel value of the pixel of interest W, the pixel value at a position away from the pixel by a pixel in the vertical direction, and the pixel value at a position away from the pixel by b pixels in the horizontal direction. Are extracted as prediction taps.

  The tap extraction unit 212 extracts, as class taps, pixel values of a plurality of pixels having a spatial positional relationship as shown in FIG. 28 for the target pixel for each frame of the HD image supplied thereto, It is output to the class classification unit 213.

  In order to simplify the description, the prediction tap and the class tap have a plurality of pixels having the same tap structure, that is, a plurality of pixels having the same positional relationship corresponding to the position corresponding to the target pixel. The pixel value of the pixel is adopted. However, the prediction tap and the class tap may have different tap structures.

  The class classification unit 213 performs class classification using the class tap supplied from the tap extraction unit 212, and supplies a class code representing the resulting class to the coefficient memory 214.

  The coefficient memory 214 stores tap coefficients for each class that are obtained in advance by learning between a high-quality image (in this example, an SHD image) and a corresponding low-quality image (in this example, an HD image). I remember it. That is, the coefficient memory 214 stores tap coefficients for each of a plurality of classes in which the target pixel can be classified by the class classification unit 213.

  The coefficient memory 214 supplies, to the prediction unit 215, the tap coefficient of the class representing the class code supplied from the class classification unit 213 among the tap coefficients for each class.

  The prediction unit 215 obtains a predicted value of the pixel value of the SHD pixel by linear combination of the pixel value of the HD pixel constituting the prediction tap supplied from the tap extraction unit 211 and the tap coefficient supplied from the coefficient memory 214. Perform adaptive processing.

  In other words, the prediction unit 215 changes the predetermined HD pixel including the target pixel W of the HD image shown in FIG. 28 to the target pixel W of the HD image on the currently predicted SHD image as shown in FIG. A predicted value of 2 × 2 SHD pixels centering on the corresponding position is obtained. In FIG. 29, the points indicated by · represent SHD pixels.

  In the prediction unit 215, for each frame of the HD image, HD pixels are sequentially performed as the target pixel, and when the predicted values of all the SHD pixels constituting the SHD image are obtained, the predicted value is , Supplied to the distribution device 51.

  As described above, even when an image having an image size different from the image size that can be displayed on the display device 52 is supplied as the distribution image D0, the image conversion device 201 converts the image size to an image size that can be displayed on the display device 52. Thus, the transmission image D21 that can be transmitted in the transmission band of the network 3 and can be synthesized while maintaining the continuity in the display device 52 is displayed via the network 3 in the display device 52. 52 can be transmitted.

  FIG. 30 shows a configuration example of another image distribution system to which the present invention is applied. In this image distribution system, a distribution device 251 is provided instead of the image conversion device 201 and the distribution device 51 of the image distribution system in FIG.

  As in the case of the example of FIG. 26, the distribution device 251 is supplied with an HD image different from the SHD image that can be displayed by the display device 52 as the distribution image D0.

  For each frame of the supplied HD image D0, the distribution device 251 converts an image of a predetermined area on the HD image D0 into an SHD image, so that the transmission band of the network 3 shown in FIGS. The display device 52 generates a transmission image D21 that can be transmitted and can be synthesized while maintaining continuity. That is, in the example of FIG. 26, an SHD image frame is once generated from each frame of the HD image supplied as the distribution image D0, and a transmission image D21 is generated from the generated frame of the SHD image. In the example, the transmission image D21 is generated directly from the HD image, not after the frame of the SHD image is generated.

  Specifically, (hs, vs), (1920/2 + hs, vs), (1920) when the upper left corner of the frame on the HD image D0 shown in FIG. 31 is (0,0). / 2 + hs, 1080/2 + vs) and (hs, 1080/2 + vs) pixels from the image D0-1 in a (1920/2) × (1080/2) pixel area with four corners 1920 with four corners on the (N, M), (1920 + N, M), (1920 + N, 1080 + M), and (N, 1080 + M) pixels on the SHD image D1 shown in FIG. An image D21-1 of a region of × 1080 pixels is generated.

  Here, in the display device 52, image conversion is performed using the prediction tap and class tap having the tap structure shown in FIG. 6, and the distribution device 251 uses the prediction tap and class tap shown in FIG. As shown in FIG. 29, it is assumed that 2 × 2, that is, double SHD pixels are generated from one HD pixel.

  Therefore, the distribution device 251 sets hs = N / 2 and vs = M / 2, thereby causing the image D0-1 in the area of (1920/2) × (1080/2) pixels on the HD image shown in FIG. Thus, the image D21-1 of the 1920 × 1080 pixel area shown in FIG. 13 can be generated. Note that hs> b, vs> a in order to secure the HD pixel having the tap structure of FIG. The same applies to FIGS. 32 to 34 described later.

  32, (1920 / 2-hs, vs), (1920-hs, vs), (1920-hs, 1080/2 + vs), and (1920 / 2-hs, 1080) on the HD image D0 shown in FIG. (1920-N, M), (3840-) on the SHD image D1 shown in FIG. 14 from the image D0-2 in the area of 1920/2 × 1080/2 pixels having four corners of / 2 + vs) pixels. N, M), (3840-N, 1080 + M), and (1920-N, 1080 + M) pixels are generated as an image D21-2 of an area of 1920 × 1080 pixels having four corners.

  33, (1920 / 2-hs, 1080 / 2-vs), (1920-hs, 1080 / 2-vs), (1920-hs, 1080-vs), and (1920) on the HD image D0 shown in FIG. / 2-hs, 1080-vs) from the image D0-3 in the area of 1920/2 × 1080/2 pixels with the four corners as (1920-N, 1080- on the SHD image D1 shown in FIG. M), (3840-N, 1080-M), (3840-N, 2160-M), and (1920-N, 2160-M) image D21-3 in a 1920 × 1080 pixel area having four corners Is generated.

  Then, (hs, 1080 / 2-vs), (1920/2 + hs, 1080 / 2-vs), (1920/2 + hs, 1080-vs), and (19) on the HD image D0 shown in FIG. (Ns, 1080-M), (N, 1080-M), (D) on the SHD image D1 shown in FIG. 1920 + N, 1080-M), (1920 + N, 2160-M), and (N, 2160-M) pixels D21-4 are generated in a 1920 × 1080 pixel area having four corners.

  31 to 34, the coordinate on the x-axis is smaller than b on the tap structure shown in FIG. 29 or larger than 1920-hs, or the y-axis coordinate is smaller than a or 1080-vs. In the larger region, in the tap structure shown in FIG. 29, the HD pixels necessary for generating the SHD pixels are missing, so the SHD pixels corresponding to the region cannot be accurately predicted. Since the SHD pixel does not constitute the transmission image D21 shown in FIGS. 13 to 16, there is no particular problem. Different tap structures can be used for this region.

  As described above, when the HD image different from the SHD image that can be displayed by the display device 52 is supplied as the distribution image D0, the transmission image D21 can be directly generated from the HD image.

  Next, the series of processes described above can be performed by hardware or software. When a series of processing is performed by software, a program constituting the software is installed in a general-purpose computer or the like.

  Therefore, FIG. 35 shows a configuration example of an embodiment of a computer in which a program for executing the series of processes described above is installed.

  The program can be recorded in advance in a hard disk 2005 or ROM 2003 as a recording medium built in the computer.

  Alternatively, the program is stored temporarily or on a removable recording medium 2011 such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), an MO (Magneto Optical) disk, a DVD (Digital Versatile Disc), a magnetic disk, or a semiconductor memory. It can be stored permanently (recorded). Such a removable recording medium 2011 can be provided as so-called package software.

  The program is installed on the computer from the removable recording medium 2011 as described above, or transferred from the download site to the computer wirelessly via a digital satellite broadcasting artificial satellite, or a LAN (Local Area Network), The program can be transferred to a computer via a network such as the Internet, and the computer can receive the program transferred in this way by the communication unit 2008 and install it in the built-in hard disk 2005.

  The computer includes a CPU (Central Processing Unit) 2002. An input / output interface 2010 is connected to the CPU 2002 via the bus 2001, and the CPU 2002 operates the input unit 2007 including a keyboard, a mouse, a microphone, and the like by the user via the input / output interface 2010. When a command is input by the equalization, a program stored in a ROM (Read Only Memory) 2003 is executed accordingly. Alternatively, the CPU 2002 may be a program stored in the hard disk 2005, a program transferred from a satellite or a network, received by the communication unit 2008 and installed in the hard disk 2005, or a removable recording medium 2011 mounted on the drive 2009. The program read and installed in the hard disk 2005 is loaded into a RAM (Random Access Memory) 2004 and executed. As a result, the CPU 2002 performs processing performed by the configuration of the block diagram described above. Then, the CPU 2002 outputs the processing result from the output unit 2006 including an LCD (Liquid Crystal Display), a speaker, or the like, for example, via the input / output interface 2010, or from the communication unit 2008 as necessary. Transmission, and further recording in the hard disk 2005 is performed.

  Here, the program may be processed by one computer, or may be distributedly processed by a plurality of computers. Furthermore, the program may be transferred to a remote computer and executed.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a figure which shows the structural example of the conventional image delivery system. It is a figure explaining a delivery picture. It is a figure explaining a transmission image. It is a figure explaining the image conversion process which converts a transmission image into a higher quality image. It is a block diagram which shows the structural example of the image conversion part which performs the image conversion process which converts a transmission image into a higher quality image. It is a figure explaining the tap structure utilized in the image conversion process which converts a transmission image into a higher quality image. It is a figure which shows the example of a display of the display apparatus 2 of FIG. It is another figure explaining the image conversion process which converts a transmission image into a higher quality image. It is a figure which shows the other example of a display of the display apparatus 2 of FIG. It is a figure which shows the other example of a display of the display apparatus 2 of FIG. It is a figure which shows the structural example of the image delivery system to which this invention is applied. It is a block diagram which shows the structural example of the delivery apparatus 51 of FIG. It is a figure explaining the process of the production | generation part 62 of FIG. It is another figure explaining the process of the production | generation part 62 of FIG. It is another figure explaining the process of the production | generation part 62 of FIG. It is another figure explaining the process of the production | generation part 62 of FIG. It is another figure explaining the process of the production | generation part 62 of FIG. FIG. 12 is a block diagram illustrating a configuration example of the display device 52 in FIG. 11. It is a figure explaining the process of the high quality production | generation part 73 of FIG. It is a block diagram which shows the structural example of the high quality production | generation part 73 of FIG. It is a figure which shows the structural example of the image quality conversion part 82 of FIG. It is a figure explaining the process of the display control part 74 of FIG. It is another figure explaining the process of the display control part 74 of FIG. It is another figure explaining the process of the display control part 74 of FIG. It is a block diagram which shows the structural example of the learning apparatus which learns a tap coefficient. It is a figure which shows the structural example of the other image delivery system to which this invention is applied. It is a block diagram which shows the structural example of the image conversion apparatus 201 of FIG. It is a figure which shows the tap structure utilized in the image conversion process in the image conversion apparatus 201 of FIG. It is a figure explaining the image conversion process in the image conversion apparatus 201 of FIG. It is a figure which shows the structural example of the other image delivery system to which this invention is applied. 6 is a diagram illustrating processing of a distribution device 251. FIG. It is another figure explaining the process of the delivery apparatus. It is another figure explaining the process of the delivery apparatus. It is another figure explaining the process of the delivery apparatus. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  3 network, 51 distribution device, 52 display device, 61 input unit, 62 generation unit, 63 data processing unit, 64 communication unit, 71 reception unit, 72 data processing unit, 73 high image quality generation unit, 74 display control unit, 75 display , 81 global feature value extraction unit, 82 image conversion unit, 101 tap extraction unit, 102 tap extraction unit, 103 class classification unit, 104 coefficient memory, 105 prediction unit, 151 learning data storage unit, 152 global feature value extraction unit, 153 Image conversion unit, 154 display control unit, 155 display, 156 input unit, 157 control unit, 201 image conversion device, 251 distribution device

Claims (9)

A distribution device that distributes the distribution image by transmitting a transmission image that constitutes the distribution image via a network, and that corresponds to the distribution image from the transmission image that has been transmitted from the distribution device via the network. In an image distribution system including an image processing device that generates a display image,
The image processing apparatus includes:
Receiving means for receiving the transmission image transmitted from the distribution device;
Conversion means for converting the transmission image received by the reception means into a high-quality image by performing image conversion processing by calculation using a tap coefficient obtained by learning in advance;
From the image obtained as a result of the image conversion process by the conversion means, an area in which image quality deteriorates according to the tap structure of the tap used in the image conversion process is excluded, and the remaining high-quality area is transferred to the corresponding transmission. Generating means for generating the display image by combining with a position corresponding to a position on the distribution image of a region on the image,
The distribution device includes:
Input means for inputting the distribution image;
Transmission is possible in the transmission band of the network, and the image conversion process by the conversion unit of the image processing apparatus is performed, so that the high-quality area is synthesized adjacently by the generation unit. Extraction means for extracting an image from the distribution image as the transmission image;
An image distribution system comprising: transmission means for transmitting the transmission image extracted by the extraction means to the image processing apparatus via the network. The conversion means of the image processing apparatus extracts a feature amount of the entire distribution image constituted by the transmission image from the transmission image, and executes the image conversion process using the extracted whole feature amount The image delivery system according to claim 1. Receiving means for receiving a transmission image constituting a distribution image transmitted from the distribution device;
Conversion means for converting the transmission image received by the reception means into a high-quality image by performing image conversion processing by calculation using a tap coefficient obtained by learning in advance;
From the image obtained as a result of the image conversion process by the conversion means, an area in which image quality deteriorates according to the tap structure of the tap used in the image conversion process is excluded, and the remaining high-quality area is transferred to the corresponding transmission. An image processing apparatus comprising: a generating unit configured to generate a display image by combining the region on the image with a position corresponding to the position on the distribution image. The conversion unit extracts a feature amount of the entire distribution image constituted by the transmission image from the transmission image, and executes the image conversion process using the extracted whole feature amount. The image processing apparatus described. A receiving step of receiving a transmission image constituting the distribution image transmitted from the distribution device;
A conversion step of converting the transmission image received in the processing of the reception step into a high-quality image by performing an image conversion process by a calculation using a tap coefficient obtained by learning in advance;
From the image obtained as a result of the image conversion process, an area in which the image quality deteriorates in accordance with the tap structure of the tap used in the image conversion process is excluded, and the remaining high-quality area is an area on the corresponding transmission image. A generating step of generating a display image by combining with a position corresponding to the position on the distribution image. In a program for causing a computer to execute display image generation processing for generating a display image,
A receiving step of receiving a transmission image constituting the distribution image transmitted from the distribution device;
A conversion step of converting the transmission image received in the processing of the reception step into a high-quality image by performing an image conversion process by a calculation using a tap coefficient obtained by learning in advance;
From the image obtained as a result of the image conversion process, an area in which the image quality deteriorates in accordance with the tap structure of the tap used in the image conversion process is excluded, and the remaining high-quality area is an area on the corresponding transmission image. A program for causing a computer to execute a display image generation process including: a generation step of generating a display image by combining with a position corresponding to a position on the distribution image. The image constituting the distribution image is converted into a high-quality image by performing image conversion processing by calculation using a tap coefficient obtained by learning in advance, and the resulting image is converted into the image conversion processing. The area where the image quality deteriorates according to the tap structure of the tap to be used is excluded, and the remaining high-quality area is combined with the position corresponding to the position on the distribution image of the image area constituting the corresponding distribution image. In the distribution apparatus that distributes the distribution image by transmitting the image constituting the distribution image to the image processing apparatus that generates the display image via a network,
Input means for inputting the distribution image;
As an image constituting the distribution image, an image that can be transmitted in the transmission band of the network and is subjected to the image conversion process so that the high-quality region is adjacently synthesized is the image Extraction means for extracting from the distribution image;
A distribution device comprising: a transmission unit configured to transmit the image extracted by the extraction unit to the image processing device via the network. The image constituting the distribution image is converted into a high-quality image by performing image conversion processing by calculation using a tap coefficient obtained by learning in advance, and the resulting image is converted into the image conversion processing. The area where the image quality deteriorates according to the tap structure of the tap to be used is excluded, and the remaining high-quality area is combined with the position corresponding to the position on the distribution image of the image area constituting the corresponding distribution image. In the distribution method of distributing the distribution image by transmitting the image constituting the distribution image to the image processing apparatus that generates the display image via a network,
An input step for inputting the distribution image;
As an image constituting the distribution image, an image that can be transmitted in the transmission band of the network and is subjected to the image conversion process so that the high-quality region is adjacently synthesized is the image An extraction step for extracting from the distribution image;
A transmission method including: a transmission step of transmitting the image extracted in the processing of the extraction step to the image processing device via the network. The image constituting the distribution image is converted into a high-quality image by performing image conversion processing by calculation using a tap coefficient obtained by learning in advance, and the resulting image is converted into the image conversion processing. The area where the image quality deteriorates according to the tap structure of the tap to be used is excluded, and the remaining high-quality area is combined with the position corresponding to the position on the distribution image of the image area constituting the corresponding distribution image. In a program for causing a computer to execute a distribution process for distributing the distribution image by transmitting the image constituting the distribution image to the image processing apparatus that generates the display image via a network,
An input step for inputting the distribution image;
As an image constituting the distribution image, an image that can be transmitted in the transmission band of the network and is subjected to the image conversion process so that the high-quality region is adjacently synthesized is the image An extraction step for extracting from the distribution image;
A program that causes a computer to execute a distribution process including: a transmission step of transmitting the image extracted in the extraction step process to the image processing apparatus via the network.

Images