JP2012195880A - Image processing apparatus, image processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an apparatus which when performing improvement of resolution on an input image signal, is capable of sufficiently extracting performance for improvement of resolution by increasing reference information needed for improvement of resolution.SOLUTION: According to the embodiment, an image conversion apparatus generates a converted image signal and transmits the converted image signal to a receiving apparatus via a repeating apparatus. The receiving apparatus performs restoration processing and super-resolution processing on the received converted image signal, thereby obtaining a high resolution image signal having improved resolution. In order to reduce the capacity of the repeating apparatus, the image conversion apparatus obtains the converted image signal by processing an original image signal. In order to improve the performance of the super-resolution processing by the receiving apparatus, a conversion property extraction apparatus generates image conversion associated information to support the super-resolution processing. Then, a connection apparatus transmits the converted image signal and the image conversion associated information to the receiving apparatus.

Description

  Embodiments described herein relate generally to an image processing apparatus, an image processing method, and a program.

  In recent years, television receivers and displays that employ super-resolution technology are commercially available. The super-resolution technique is a technique for reproducing a detailed portion lost during image pixel sampling (during photoelectric conversion or reduction processing by an image sensor) by image processing. For example, when playing back and displaying low-resolution image information recorded on a Digital Versatile Disc (DVD), using super-resolution technology reproduces the detailed part of the image, resulting in a clear image with high resolution. can get.

  In order to further improve the resolution, there is a technique for improving the processing performance for high resolution by analyzing an image by preprocessing and removing noise. There is also a technique for analyzing an image by preprocessing and changing a method for high resolution according to the analysis content.

JP 2009-65535 A JP 2010-130069 A

  When preprocessing is performed on an input image signal, information necessary for high resolution processing is removed together with noise in the preprocessing, and high resolution may fail. Further, when the accuracy of analysis information obtained by image analysis of an input image signal is low, the analysis information may not be sufficiently utilized as reference information for high resolution.

  Current high-resolution processing devices are used in environments where the input image type is unpredictable. The image type is a type that is classified based on, for example, a difference in shooting target, a difference in shooting environment, a difference in camera or lens, a difference in characteristics of a transmission method, a difference in characteristics of a recording method, and the like. These types are unpredictable from broadcast signals and playback signals from DVDs.

  Thus, the current high-resolution processing apparatus is not sufficiently optimized specifically for a specific video type. For this reason, there is a limit to the improvement in resolution without maximizing the performance of the super-resolution technology.

  Accordingly, an object of the present invention is to provide an image that can sufficiently bring out the performance for increasing the resolution by making the reference information necessary for increasing the resolution rich when the resolution increasing process is performed on the input image signal. It is an object to provide a processing device, an image processing method, and a program.

  According to the embodiment, the image conversion device generates a converted image signal and transmits the converted image signal to the reception device via the relay device. The receiving device performs a restoration process and a super-resolution process on the received converted image signal to obtain a high-resolution image signal with improved resolution. In this case, the image conversion apparatus obtains the converted image signal by processing the original image signal in order to reduce the capacity of the relay apparatus. On the other hand, the conversion characteristic extraction device generates image conversion related information for assisting the super-resolution processing in order to improve the performance of the super-resolution processing in the receiving device. Then, the converted image signal and the image conversion related information are transmitted to the receiving device.

It is a block diagram which shows an example of a structure of embodiment. It is the figure shown in order to demonstrate an example of operation | movement of the conversion characteristic information reflector and correction circuit shown in FIG. FIG. 10 is a diagram for explaining another example of the operation of the conversion characteristic information reflector and the correction circuit shown in FIG. 1. FIG. 10 is a diagram for explaining yet another example of the operation of the conversion characteristic information reflector and the correction circuit shown in FIG. 1. It is a figure which shows an example of the parameter production | generation apparatus corresponding to the image conversion relevant information demonstrated in FIG. It is a figure which shows an example of the other parameter production | generation apparatus corresponding to the image conversion relevant information demonstrated in FIG. It is a figure which shows an example of the further parameter production | generation apparatus corresponding to the image conversion relevant information demonstrated in FIG. It is a block diagram which shows the structural example of other embodiment. Furthermore, it is a block diagram which shows the structural example of other embodiment.

  Hereinafter, embodiments will be described with reference to the drawings. FIG. 1 is a block diagram illustrating an example of the configuration of the embodiment. Reference numeral 100 denotes a transmission device, and reference numeral 200 denotes a reception device. Note that the terms “transmission device” and “reception device” used in the present disclosure are not limited to transmission and reception, and may have a relationship of “recording” and “reproduction”. Various terms such as “imaging unit” and “display unit”, “reading unit”, and “output unit” can be applied, but for the sake of convenience, these will be described as “transmitting device” and “receiving device”.

  Therefore, the apparatus, method, program, and program product described in this disclosure include various communication systems, information recording / reproducing apparatuses, monitoring systems in companies or public institutions, local area networks (LAN) in companies, and homes. Applicable to the field.

  The transmission device 100 includes an input device 101 for an original image signal, and the original image signal is input to the image conversion device 102 and the conversion characteristic extraction device 103. The image conversion apparatus 102 reduces the original image signal, for example, and outputs a converted image signal. In this case, the conversion characteristic extraction device 103 generates “reduction conversion characteristic” as the conversion characteristic, for example, as the image conversion related information.

  Here, it has been described that the image conversion apparatus 102 performs the reduction process using the original image signal as an example. However, the present disclosure is not limited to the reduction process, and various conversion processes described later may be included in the embodiments. Please keep in mind. Therefore, the conversion characteristic extraction device 103 extracts conversion characteristics corresponding to various conversion processes.

  The converted image signal and the image conversion related information are output by the connection device 104 to the relay device 400 (including the communication line 401 or the storage medium 402). The connection device 104 may include a high frequency modulator, a high frequency receiver, and a mutual authentication device. The connection device 104 may also include an encoding / decoding processing function for the purpose of transmission / reception. Here, the communication line includes, for example, the Internet, a wired communication line, a wireless communication line, an infrared communication line, an optical communication line, and the like, and the communication area is not limited. Further, the storage device is not limited to a specific storage medium including a recording / playback device, a server, a magnetic disk, an optical disk, a semiconductor memory, and the like. The overall control block 111 comprehensively controls the operation sequence and operation content of each block in the transmission apparatus 100 using a software program.

  The converted image signal and the image conversion related information are input to the connection device 201 of the reception device 200 via the relay device 400. The connection device 201 may include a high frequency receiver, a high frequency modulator, and a mutual authentication device. The connection device 201 may also include an encoding / decoding processing function for the purpose of transmission / reception. The converted image signal and the image conversion related information captured via the connection device 201 are input to the super-resolution processing device 202. The super-resolution processing device 202 performs restoration processing and super-resolution processing on the converted image signal to obtain a high-resolution image signal with improved resolution. In the super-resolution processing device 202, when the super-resolution processing is performed, the image conversion related information is stored in the super-resolution so that the super-resolution processing is performed with high accuracy and the performance of the super-resolution processing is improved. Used to assist in processing. The high-resolution image signal that has been subjected to the super-resolution processing is input to the display device 203. The overall control block 211 comprehensively controls the operation sequence and operation content of each block in the receiving apparatus 200 using a software program.

  An example of the block configuration of the super-resolution processing apparatus 202 is shown in the lower part of FIG. A signal input from the connection device 201, for example, a packet sequence is separated by the separator 221, a packet including image conversion related information is input to the conversion characteristic information reflector 230, and a packet including image information is converted into an image restoration circuit. 223 is input. The image restoration circuit 223 restores the converted image signal converted by the transmission device 100 to the original image signal, and obtains a restored image signal. The image restoration circuit 223 extracts an image signal from, for example, a packet stream. The restored image signal is subjected to image processing, for example, by the initial image generator 229 and input to the high resolution image generator 225. The restored image signal is also input to the error detector 224. The separator 221 and the image restoration circuit 223 may be included in the connection device 201.

  The high-resolution image generator 225 uses a super-resolution technique to improve the resolution of the image signal, for example, by pixel correction processing. The image signal can have a higher resolution than the number of samples on the transmission side. The output of the high resolution image generator 225 is input to the predictive low resolution image generator 226. The predicted low resolution image generation circuit 226 generates an image signal (predicted low resolution image signal) having the same resolution as the image signal sent from the transmission device, and inputs the generated image signal to the error detector 224.

  The predicted low resolution image generation circuit 226 generates a predicted low resolution image signal by using a point spread function (PSF) when generating a predicted low resolution image signal. Here, when the PSF parameter is applied, the correction parameter from the conversion characteristic information reflector 230 is used.

  The predicted low resolution image signal should be the same as the originally transmitted image signal. However, there may be an error in the path of conversion processing, transmission processing, and super-resolution processing. Therefore, as described above, the prediction low-resolution image generation circuit 226 uses the correction parameter from the conversion characteristic information reflector 230. The conversion characteristic information is information for specifying a correction parameter. The instructed correction parameter is, for example, a correction parameter that can improve an error between the original image signal and the predicted low-resolution image signal.

  The predicted low resolution image generation circuit 226 performs processing (for example, filtering processing and level adjustment processing) to suppress distortion components generated in the predicted low resolution image signal using the correction parameter.

  Thereby, the error detector 224 outputs an accurate error signal in which the location where the error signal is generated in the transmitted converted image signal, the magnitude of the error signal, etc. are clarified. Using this error signal, the predicted original image signal in the high-resolution image generator 225 is corrected so as to approach the original image signal. Such processing is repeatedly executed as super-resolution processing. The high resolution image generator 225 supplies the super resolution image signal with improved resolution to the output circuit 227.

  As described above, the image conversion apparatus 102 processes the original image signal to obtain a converted image signal in order to reduce the capacity of the relay apparatus 400 (communication line or storage medium). On the other hand, the conversion characteristic extraction device 103 generates image conversion related information for assisting the super-resolution processing. Then, the connection device 104 outputs the converted image signal and the image conversion related information to a communication line or a storage medium.

  In the receiving apparatus 200, the conversion characteristic information is reflected in order to improve the performance of the super-resolution processing. In the above-described embodiment, since the conversion characteristic information is directly sent from the transmission side, it is not necessary to predict the conversion characteristic on the receiving side, and the conversion characteristic information is accurately recognized and used. Can do. That is, the receiving apparatus can directly and accurately grasp the conversion characteristic information. Then, the error included in the predicted original image signal is surely corrected. For this reason, the super-resolution processing apparatus 202 can sufficiently exhibit its performance.

  Next, a method and apparatus for creating conversion characteristic information on the transmission side and a method and apparatus for reflecting conversion characteristic information on the reception side will be described.

  The PSF information in the predictive low-resolution image generation circuit 226 describes from which peripheral high-resolution pixels, for example, one pixel with low resolution is generated. When a low-resolution pixel is created, it is preferable to perform a low-pass filter process on the high-resolution image to remove high-frequency components higher than the Nyquist frequency before reducing the image size (reducing the resolution). On the other hand, in order to perform super-resolution processing, it is theoretically convenient for super-resolution not to perform low-pass filtering.

  The example of FIG. 2 shows an example of image conversion and transmission, reception processing, and use state of conversion characteristic information. The original image signal SIG1 (pixels X11 to X33) is filtered F on the transmission side, and then reduced to 1/3 to obtain a converted image signal SIG2 (pixel Y). On the reception side, a three-fold enlargement process G is performed to obtain a predicted original image signal SIG3 (pixels X′11 to X′33). Temporary correction processing is performed on the predicted original image signal SIG3, and super-resolution processing is performed.

  Here, the super-resolution processing is performed so that the predicted original image signal SIG3 becomes a signal that more closely approximates the original image signal SIG1. That is, the predicted original image signal SIG3 is subjected to filter processing F with the same characteristics as the transmission side, and is reduced to 1/3. The predicted low resolution image signal Y ′ is compared with the transmitted low resolution image signal Y by the error detector 224. The error ERR = Y−Y ′ obtained as a result of this comparison is input to the correction circuit 228 and used to correct the predicted original image signal SIG3.

  By repeating the above correction process, the predicted original image signal SIG3 becomes more approximate to the original image signal SIG1. When the level of the error signal falls within an allowable range, it is determined that the signal is a useful high-resolution image signal.

  FIG. 2 shows two types of conversion characteristic information for performing the filter processing F, that is, here, examples of filter characteristics. One is an example of performing filtering by an average value filter, and the other is a Gaussian filter. This is an example of performing filtering, and any of them may be adopted.

  The conversion characteristic information (the characteristic of the filter processing F here) performed on the transmission side as described above is also directly used on the reception side. For this reason, the processing speed and processing load for approximating the predicted original image signal SIG3 to the original image signal SIG1 are reduced.

  FIG. 3 shows image conversion and transmission, reception processing, use state of conversion characteristic information, and other examples. The method for obtaining the converted image signal by processing the original image signal in order to reduce the capacity of the relay device is not limited to the method shown in FIG.

  In FIG. 3, the filter processing F is performed on the original image signal SIG1 (pixels X11 to X33) on the transmission side to obtain the filtered image signal SIG2 (pixels Y11 to Y33). The filter process F is, for example, a two-dimensional filter process using a horizontal filter HF and a vertical filter VF1. In this filtering process, a filtering process in the spatial direction (time direction) may be performed. The filtered converted image signal SIG2 is transmitted via the relay device.

  In this case, the converted image signal may be encoded and transmitted, or may be transmitted as it is.

  On the reception side, a predicted original image signal SIG3 (pixels X′11 to X′33) is obtained by performing inverse filtering G on the filtered image signal SIG2. As shown in comparison with (b1) and (b2) in FIG. 5B, the inverse filter processing G has a characteristic that suppresses the high frequency, and the inverse filter processing G It is a characteristic that emphasizes frequency. Temporary correction processing is performed on the predicted original image signal SIG3 obtained in this way, and super-resolution processing is performed.

  Here, the super-resolution processing is performed so that the predicted original image signal SIG3 becomes a signal that more closely approximates the original image signal SIG1. That is, the predicted original image signal SIG3 is subjected to filter processing F with the same characteristics as those on the transmission side, and is reproduced as a predicted filtered image signal SIG4 (pixels Y′11 to Y′33). The prediction filtered image signal SIG4 (pixels Y'11-Y'33) is compared with the transmitted filtered image signal SIG2 (pixels Y11-Y33) in the error detector 224. The error ERR = Y−Y ′ obtained as a result of this comparison is input to the correction circuit 228 and used to correct the predicted original image signal SIG3.

  By repeating the above correction process, the predicted original image signal SIG3 becomes more approximate to the original image signal SIG1. When the level of the error signal falls within an allowable range, it is determined that the signal is a useful high-resolution image signal.

  FIG. 4 shows still another example of image conversion and transmission, reception processing, and use state of conversion characteristic information. Here, an encoding process is shown as another method for processing the original image signal SG1 to obtain the converted image signal SG2 in order to reduce the capacity of the relay apparatus. The pixel data of the original image signal SG1 is encoded and decoded on the receiving side via the relay device. In the pixel data of the decoded image signal SG3 (pixels X′11 to X′33) obtained by decoding, error data is generated in the vicinity. The error data is, for example, block noise.

  After the decoded image signal including the block noise is subjected to super-resolution processing and is directly used as the predicted original image signal SIG4 (pixels Z11 to Z33), whether the predicted original image signal includes block noise, Contains components caused by block noise.

  Therefore, in the present embodiment, in order to remove the noise of the predicted original image signal SIG4 (pixels Z11 to Z33), the predicted low-resolution image signal is filtered F by the filter Fi2. The filtered predicted decoded image signal (pixels R11-R33) and the decoded image signal SG3 (pixels X′11-X′33) are compared, and prediction is performed according to the error signal ERR = X′-R. The original image signal SIG4 (pixels Z11 to Z33) is corrected (variation of filter characteristics, level adjustment, etc.) to reduce the level of the error signal. A useful high resolution image signal is when the level of the error signal falls within an acceptable range. The filter process G between the decoded image signal SG3 and the predicted original image signal SIG4 is a filter process used when generating the first predicted original image signal, and is, for example, a random noise filter process.

  Here, the filter processing F can apply characteristics that have been found in advance by simulation or the like. Therefore, according to the type of encoding / decoding process, the corresponding filter processing characteristic parameter can be transmitted from the transmission side to the reception side.

  The above processing has shown an embodiment in which the predicted low resolution image signal is corrected. However, preprocessing may be performed on the restored image signal before performing the super-resolution processing of the decoded image signal.

  For this purpose, for example, information for determining a filter characteristic acting on the decoded image signal may be set in the receiving apparatus in advance.

  FIG. 5 shows an example in which conversion characteristic information and correction parameters to be set in the conversion characteristic information reflector 230 are created in advance. An encoding process is shown as an example in which the original image signal SG1 is processed to obtain the converted image signal SG2 in order to reduce the capacity of the relay device. The pixel data of the original image signal SG1 is encoded on the transmission side and decoded on the reception side via the relay device, but error data is generated when decoded. Therefore, a filter characteristic that can eliminate or reduce this error data is investigated in advance.

  That is, the decoded image signal SIG3 is input to the error detector RrD1 through the filter Fi3. The original image signal SIG1 is input to the error detector RrD1. The error detector RrD1 compares the pixel of the original image signal SIG1 with the corresponding pixel of the decoded image signal SIG3, and detects an error. Thereby, error data is detected. The filter coefficient of the filter Fi3 is varied according to the error data. Then, the filter coefficient when the level of the error data is equal to or less than the allowable value is determined as a correction parameter (a parameter for correcting the filter coefficient) in the conversion characteristic information reflector 230.

  The identification information of the encoding / decoding method and the corresponding correction parameter at this time are stored in a memory, for example, and set in the conversion characteristic information reflector 230 of the receiving apparatus. When receiving the identification information of the encoding / decoding method, the receiving apparatus 200 outputs the correction parameter set in the conversion characteristic information reflector 230 and based on the correction parameter for the filter processing F described above. Set the filter coefficient.

  With the above processing, noise of the predicted original image signal is reduced, and high-quality super-resolution processing can be realized.

  FIG. 6 shows another example in which conversion characteristic information and correction parameters to be set in the conversion characteristic information reflector 230 are created in advance.

  This example is an example of an apparatus for creating conversion characteristic information and correction parameters to be applied to a system in which an image is reduced and transmitted on the transmission side and a reduced image is enlarged on the reception side. For example, a test image signal is input to the input terminal 501. The test image signal (original image signal) is input to the reducer 502 and reduced. The reduced image is enlarged by the enlarger 503 and restored to the original image signal (predicted original image signal).

  The error information detector 504 compares the original image signal and the predicted original image signal, for example, in units of preset blocks. The error information detector 504 detects noise and distortion generated in the image reduction / enlargement path. Noise and distortion detected by the error information detector 504 are extracted by the error extractor 505 and input to the error reduction information generator 506.

  The error reduction information generator 506 creates a correction parameter in a direction in which the error is reduced, and supplies the correction parameter to the correction circuit 511. The correction parameters include horizontal / vertical filter coefficients and level control information. The correction circuit 511 changes the filter processing characteristics or level control characteristics that are converted into the predicted original image signal from the enlarger 503 in accordance with the correction parameters. When the error is reduced within the allowable range, the error reduction information generator 506 determines the correction parameter at that time.

  The fixed correction parameter is input to the correction information table generator 510 and registered in the correction information table. When the correction parameter is determined, the image conversion characteristic information is also determined. As the image conversion characteristic information, for example, conversion method information indicating the value of the enlargement / reduction ratio, still image / moving image information indicating a still image or moving image of the image, and in the case of a moving image, the image moving speed is any of F1, F2, and F3 Image motion speed information indicating whether the average luminance level is A1, A2, or A3, and luminance dispersion information, and the specified image conversion characteristic information is also the correction information. Registered in the table.

  Such registration information is created by an image characteristic determination unit 507 and a processing information recognition device 508. The image characteristic determiner 507 determines whether the original image signal as the test signal is a still image or a moving image, or in the case of a moving image, the image motion speed is F1, F2, or F3, and the average luminance level is A1, A2, or A2. It is determined which of A3. Information such as luminance dispersion and luminance histogram can also be acquired. The processing information recognition apparatus 508 determines the value of the enlargement / reduction ratio using the control information from the control block.

  The created correction information table is stored in advance in the memory of the conversion characteristic information reflector 230 of the receiving device 200. When the conversion characteristic information reflector 230 receives the image conversion characteristic information from the transmission device 100, the conversion characteristic information reflector 230 may designate a correction parameter (also referred to as error erasure information) corresponding to the image conversion characteristic information on the table. it can. Then, the conversion characteristic information reflector 230 reads the designated correction parameter from the memory and gives it to the correction circuit 228. For this reason, a fixed error caused by the characteristics of the conversion path is suppressed from the predicted original image signal and input to the high-resolution image generator 225.

  As described above, in particular, this system measures the error that occurs when the original image signal is converted and the converted image is restored, and is prepared in advance by transmitting it directly to the receiving apparatus 200 as error erasure information. For this reason, the effect for suppressing the error in the receiving apparatus 200 is remarkable. Therefore, the predicted original image signal can be matched with the original image signal as much as possible, and the performance of the super-resolution processing device 202 can be maximized.

  In the above example, the image conversion characteristics show conversion method information regarding image reduction / enlargement, still image / moving image information, image motion speed information, luminance level information, luminance dispersion information, etc. It is not necessary. The effect of the present apparatus is exhibited even at least with the conversion method information alone. The above conversion method information is not limited to information relating to reduction / enlargement.

  In the above description, the converted image signal from the reducer 502 is directly input to the enlarger 503. However, characteristics (attenuation characteristics, frequency characteristics, etc.) corresponding to the relay apparatus 400 may be given by the pseudo relay apparatus 511 on the way. Further, the above apparatus may be configured independently of the transmission apparatus 100 of FIG. 1 or may be integrated into the transmission apparatus 100 of FIG.

  FIG. 7 shows still another example in which conversion characteristic information and correction parameters to be set in the conversion characteristic information reflector 230 are created in advance. In this example, the transmission side encodes (compresses) the original image signal to generate an encoded image signal, and transmits the encoded image signal. The reception side receives and decodes the encoded image signal, It is an example of the apparatus which produces | generates a decoded image signal, and also raises the resolution of a decoded image signal. Blocks having functions similar to those of the apparatus of FIG. 2 are denoted by the same reference numerals as the blocks of FIG. 2 is largely different from the block of FIG. 2 in that the encoder 502 encodes the test signal (original image signal) to generate an encoded image signal, and the decoder 503 decodes the encoded image signal. This is the point that generates

  In the correction information table in the correction information table generator 510, correction parameters for reducing conversion and restoration errors are registered according to the encoding / decoding method.

  There are various encoding / decoding methods such as Joint Photographic Experts Group (JPEG), Moving Picture Experts Group 1/2 (MPEG 1/2), H 264 / Advanced Video Coding (AVC), etc. is there. A unique restoration error is caused by each method and by the conversion characteristics on the transmission side. By reducing this restoration error, the function of super-resolution processing on the receiving side is sufficiently exhibited.

  In the above figure, the converted image signal from the encoder 522 is directly input to the decoder 523. However, characteristics (attenuation characteristics, frequency characteristics, etc.) corresponding to the relay apparatus 400 may be given by the pseudo relay apparatus 511 on the way. Further, the above apparatus may be configured independently of the transmission apparatus 100 of FIG. 1 or may be integrated into the transmission apparatus 100 of FIG.

  In the above-described embodiment, the conversion characteristic information is not limited to the above-described example, and may be color format conversion characteristic information and a correction parameter associated with the color format conversion characteristic.

  FIG. 8 shows still another embodiment. In this embodiment, an image characteristic detection device 106 is provided in the transmission device 100. For example, as described with reference to FIGS. 6 and 7, the image characteristic detection device 106 detects a moving image / still image, an average luminance level, synchronization disturbance information, and the like. Furthermore, the image conversion apparatus 102 can be controlled so that two or more areas including minute image movements are selected and the image conversion apparatus 102 thins out frames in which such an image does not exist. Such a process means, for example, the object extraction process, and the background-only image signal frame is thinned out. It also means the extraction and separation of object displacement. For example, a human face may be an object. In this case, there is a possibility that the two areas including minute image motion are human eye images. Such a function is important in a monitoring device.

  Further, when synchronization is disturbed or the average luminance level is abnormally large, this may be detected by the image characteristic detection device 106 and the image conversion device 102 may be controlled so that the frame is thinned out. In the monitoring system, there is a case where the monitoring video is unnecessary when the camera is installed in a region where there are many abnormal radio waves and there are many synchronization disturbances. In such a case, frames in a period in which synchronization is disturbed may be thinned out due to synchronization disturbance. In addition, sometimes in a region where abnormal light is incident on the camera (for example, a car lighting light), the luminance level becomes abnormally high, and the monitoring image may be unnecessary. In such a case, the frame may be thinned out or the level may be greatly suppressed.

  For example, a time stamp included in each packet stream for transmitting the conversion characteristic information and the image signal can be used as the synchronization information so that the image signal corresponding to the conversion characteristic information is synchronized with each other.

  FIG. 9 shows still another embodiment. The transmission apparatus 200 of this embodiment can respond to request data transmitted upstream from the reception apparatus 200. The request data may be generated, for example, in the overall control block 211 of the receiving device 200, or may be automatically generated by the super-resolution processing device 202. Further, in response to the user operating the operation device 213, the overall control block 211 may generate request data.

  Various contents can be requested by the request data. For example, there is request data for controlling the conversion characteristic extraction function, such as transmission stop of transmission of conversion image signals and conversion characteristic information, transmission restart, and the like. There is also request data for switching image conversion characteristics and switching or changing image feature extraction items. For example, it is possible to request that only a converted image signal obtained by converting a moving image is transmitted, or to request that only a converted image signal obtained by converting only a still image is transmitted. Further, image conversion with the luminance level suppressed and image conversion with the luminance level enhanced may be performed according to the time zone (for example, nighttime or daytime).

  Further, it may be request data for giving a command to compress a plurality of image frames obtained by the image conversion apparatus 102 by multi-view video encoding included in the communication means or storage means in the relay apparatus. Further, the request data may specify the area for image conversion from within the frame.

  Although the above embodiments have been described as apparatuses and methods, an IC chip for realizing such apparatuses and methods, a program for operating a computer, and a storage medium storing the programs are also within the scope of the present invention.

  Further, in the present disclosure, a low-resolution image signal for transmission with a reduced data amount of the original image signal is created on the transmission side, but there are various techniques for obtaining an image signal for transmission. Therefore, in the present disclosure, image signals to be transmitted may be collectively referred to as converted image signals. In addition, on the receiving side, the image signal that approximates the original image signal is referred to as a predicted original image signal, a prediction filtered image signal, and the like, but is not limited to these terms, and may be collectively referred to as a restored image signal. Good.

  That is, the present disclosure discloses that a converted image signal received via a relay device is subjected to restoration processing and super-resolution processing to obtain a high-resolution image signal with improved resolution. It can be used as a program for realizing an image processing method for an apparatus that generates and transmits. And this program processes the original image signal to reduce the capacity in the relay device to generate the converted image signal, and in order to improve the performance of the super-resolution processing in the receiving device, The method includes generating image conversion related information for assisting super-resolution processing, and outputting the converted image signal and the image conversion related information to the communication line or the storage medium.

  The present disclosure can also be applied to a storage medium that stores a program that implements the image processing method. That is, this storage medium is a storage medium that stores an image processing program that causes a computer to execute the above steps.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 100 ... Transmission device 101 ... Input device 102 ... Image conversion device 103 ... Conversion characteristic extraction device 104 ... Connection device 200 ... Reception device 201 ... Connection 202, super-resolution processing device, 203 ... display device, 221 ... separator, 223 ... image restoration circuit, 224 ... error detector, 225 ... high-resolution image generation 226... Predictive low-resolution image generation device, 227... Output circuit, 228... Correction circuit, 230.

Claims (12)

An apparatus that generates and transmits the converted image signal to a receiving apparatus that performs a restoration process and a super-resolution process on the converted image signal received through the relay apparatus to obtain a high-resolution image signal with improved resolution. There,
An image conversion device for processing the original image signal to obtain the converted image signal in order to reduce the capacity of the relay device;
A conversion characteristic extraction device that generates image conversion related information for assisting the super-resolution processing in order to enhance the performance of the super-resolution processing in the receiving device;
An image processing apparatus having a connection device that outputs the converted image signal and the image conversion related information to the communication line or a storage medium. The image conversion device obtains and outputs the converted image signal by reducing the screen size of the original image signal,
The image processing apparatus according to claim 1, wherein the conversion characteristic extraction apparatus generates the image conversion related information including reduction ratio information of the screen size. The image processing apparatus according to claim 1, wherein the conversion characteristic extraction apparatus adds filter processing characteristic information applied to the original image signal to the image conversion related information. The image conversion device obtains and outputs the converted image signal by encoding the original image signal,
The image processing apparatus according to claim 1, wherein the conversion characteristic extracting apparatus adds method information of the encoding process to the image conversion related information. The image conversion device generates the converted image signal by irreversibly compressing the original image signal,
The said conversion characteristic extraction apparatus adds the error information of the decoded image signal and the said original image signal when decoding the said conversion image signal by which the said irreversible compression process was decoded to the said image conversion relevant information. Image processing device. The image conversion device generates the converted image signal by converting a color format of the original image signal,
The image processing apparatus according to claim 1, wherein the conversion characteristic extracting apparatus adds conversion information of the color format to the image conversion related information. The image according to claim 1, further comprising an image characteristic detection device that detects an image characteristic of the original image signal, wherein the image conversion device adaptively performs image conversion according to a detection result of the image characteristic detection device. Processing equipment. The image processing device according to claim 7, wherein the image characteristic detection device detects a change in a specific object of the original image signal, and the image conversion device performs image conversion for thinning out a frame without a change in the object. The image processing apparatus according to claim 1, wherein the conversion characteristic extraction function is controlled in accordance with request data from the receiving apparatus. An apparatus for generating and transmitting the converted image signal to a receiving apparatus that performs restoration processing and super-resolution processing on the converted image signal received via the relay apparatus and obtains a high-resolution image signal with improved resolution. An image processing method comprising:
In order to reduce the capacity of the relay device by the image conversion device, the original image signal is processed to generate the converted image signal,
In order to enhance the performance of the super-resolution processing in the receiving device by the conversion characteristic extraction device, generate image conversion related information for assisting the super-resolution processing,
An image processing method for outputting the converted image signal and the image conversion related information to the communication line or the storage medium by a connection device. To a device that generates and transmits the converted image signal to a receiving device that performs restoration processing and super-resolution processing on the converted image signal received via the relay device and obtains a high-resolution image signal with improved resolution A program for realizing an image processing method for
Processing the original image signal to reduce the capacity at the relay device to generate the converted image signal;
Generating image conversion related information for assisting the super-resolution processing in order to enhance the performance of the super-resolution processing in the receiving device;
An image processing program comprising a step of outputting the converted image signal and the image conversion related information to the communication line or a storage medium. To a device that generates and transmits the converted image signal to a receiving device that performs restoration processing and super-resolution processing on the converted image signal received via the relay device and obtains a high-resolution image signal with improved resolution A storage medium storing a program for realizing an image processing method;
Processing the original image signal to reduce the capacity at the relay device to generate the converted image signal, and performing the super-resolution processing to improve the performance of the super-resolution processing at the receiving device. A storage medium storing an image processing program for executing a step of generating image conversion related information for assisting and a step of outputting the converted image signal and the image conversion related information to the communication line or a storage medium.

Images