JP2009026026A - Image processor, method for processing image, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To effectively use a plurality of image processing units. <P>SOLUTION: In an image comparison mode, the image processor 200 uses image processing units 213-1 to 213-3 to apply a plurality of processes each having different parameters to received input images, and then displays the processed images on a single screen concurrently. A user selects a most suitable processed image among the plurality of processed images. In a full screen mode, the image processor 200 applies an image process using a most suitable parameter selected by the user in the image comparison mode, to the input images, wherein a higher-quality image process is executed by using a broader tap range than the image comparison mode. This invention is, for example, applicable to an image processor. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image processing device, an image processing method, and a program, and more particularly, to an image processing device, an image processing method, and a program that are suitable when a plurality of image processing units are provided.

  When performing a plurality of different image processing on an image and comparing the results of the image processing, it is easy to compare the processed images after the plurality of image processing by alternately displaying them on one monitor. Conceivable. In addition, there is a method in which a plurality of processed images are simultaneously displayed on a monitor and compared (for example, see Patent Documents 1 and 2).

  When a plurality of different image processes are performed on a plurality of still images and the image processing results are compared and examined, the still images to be compared are sequentially supplied to one image processing unit, and the result obtained is obtained. A plurality of processed images can be arranged on the monitor at the same time.

JP 2000-305193 A JP 2006-67155 A

  However, when the image to be compared is a moving image, in order to view the processed image at the same time, it is necessary to provide a plurality of image processing units and allow them to perform processing simultaneously in parallel.

  However, after a comparative study, after deciding to one of a plurality of image processing methods or parameters, there is a problem that only one image processing unit is required, and other image processing units cannot be effectively used. It was.

  The present invention has been made in view of such a situation, and makes it possible to effectively utilize a plurality of image processing units.

  An image processing apparatus according to an aspect of the present invention performs an image conversion process for converting one or more input images that are images constituting a moving image and sequentially input as a first image into a second image. By selecting one post-processing image from among a plurality of image conversion processing means to be performed and a plurality of post-processing images that are images processed by each of the plurality of image conversion processing means, a plurality of different image conversion processing A plurality of image conversion processing means, comprising: a first mode for selecting one of the image conversion processes; and a control means for switching a second mode for executing the image conversion process selected in the first mode. Each of the first image using, as a class tap, a pixel used to classify a target pixel of interest in the second image into any one of the plurality of classes. Class tap selection means for selecting from, class classification means for classifying the pixel of interest based on the class tap, and a pixel used for calculation of the tap coefficient when obtaining the pixel of interest as a prediction tap Predictive tap selection means for selecting from the first image, tap coefficient output means for outputting the tap coefficient of the class of the target pixel, tap coefficient of the class of the target pixel, and the selected for the target pixel A calculation unit for calculating the target pixel by performing a calculation using a prediction tap, and the class tap, the prediction tap, and the tap coefficient in the first mode and the second mode. Switch at least one of

  The tap range, which is the range of pixels selected as the class tap or the prediction tap, is the same in each of the plurality of image conversion processing means in the first mode, and in the second mode, the plurality of images. Different in each conversion processing unit, the range obtained by combining the tap ranges of each of the plurality of image conversion processing units can be set to be a range that is larger than the tap range in the first mode.

  In each of the plurality of image conversion processing means, by switching at least one of the class tap, the prediction tap, and the tap coefficient, a predetermined region of the first image is set in the first mode. An enlargement process for enlarging is performed, and in the second mode, a predetermined area of the first image is enlarged, and a time resolution and an enlargement process for improving the time resolution can be performed.

  A feature amount of the input image is extracted, and a synchronization feature amount extraction unit that detects a time code indicating which scene in the moving image the input image is, and detected by the synchronization feature amount extraction unit A plurality of area specifying means for outputting area specifying information for specifying the predetermined area that each of the plurality of image conversion processing means expands in correspondence with the time code; In each of the first mode, an enlargement process for enlarging the first image may be performed based on the region specifying information.

  In the first mode, selection information that is information representing an enlarged image selected by the user from among the plurality of enlarged images obtained by enlarging the first image by the plurality of image conversion processing means, A plurality of user selection information storage means stored in association with the code; and a plurality of areas supplied from the plurality of area specifying means based on the selection information stored in the user selection information storage means in the second mode. Selection means for selecting one area specifying information from the area specifying information and supplying the area specifying information to a plurality of the image conversion processing means, and each of the plurality of image conversion processing means includes the second In the mode, the first image is enlarged based on the region specifying information, and the time resolution and enlargement processing for improving the time resolution are performed. Can.

  An image processing method according to an aspect of the present invention performs an image conversion process of converting one or more input images that are sequentially input as images forming a moving image as a first image into a second image. In the image processing method of the image processing apparatus, a pixel used to classify a target pixel of interest in the second image into any one of the plurality of classes is defined as a class tap. Select from one image, classify the target pixel based on the class tap, and select a pixel to be used for calculation with the tap coefficient when obtaining the target pixel from the first image as a prediction tap And outputting the tap coefficient of the class of the target pixel, and performing an operation using the tap coefficient of the class of the target pixel and the prediction tap selected for the target pixel, A plurality of image conversion processes for obtaining the target pixel are simultaneously performed, and one post-processing image is selected from among the post-processing images obtained by the plurality of image conversion processes. And switching between the first mode and the second mode, and switching between the first mode and the second mode for executing the image conversion process selected in the first mode. Switch at least one of the tap, the prediction tap, and the tap coefficient.

  A program according to one aspect of the present invention is a computer that performs image conversion processing for converting one or more input images that are images that constitute a moving image and are sequentially input into a second image, as a first image. In the program to be executed, a pixel used to classify a target pixel of interest in the second image into any one of the plurality of classes is used as a class tap from the first image. Selecting, classifying the target pixel based on the class tap, selecting a pixel used for calculation with the tap coefficient when obtaining the target pixel from the first image as a prediction tap, and selecting the target By outputting the tap coefficient of the pixel class, and performing the calculation using the tap coefficient of the pixel of interest class and the prediction tap selected for the pixel of interest. A plurality of different image conversion processes are performed by simultaneously performing a plurality of image conversion processes for obtaining the pixel of interest and selecting one processed image from the processed images obtained by the plurality of image conversion processes. A step of switching between a first mode for selecting a process and a second mode for executing the image conversion process selected in the first mode, the first mode and the second mode, A program that causes a computer to execute image conversion processing for switching at least one of a class tap, the prediction tap, and the tap coefficient.

  In one aspect of the present invention, there are a first mode for selecting one image conversion process from a plurality of different image conversion processes, and a second mode for executing the image conversion process selected in the first mode. Switching is performed, and at least one of a class tap, a prediction tap, and a tap coefficient is switched between the first mode and the second mode.

  The image processing apparatus may be an independent apparatus or a block that performs image processing of a predetermined apparatus.

  According to one aspect of the present invention, a plurality of image processing units can be effectively utilized.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment which is described in the specification or the drawings but is not described here as an embodiment corresponding to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  An image processing apparatus according to an aspect of the present invention performs an image conversion process for converting one or more input images that are images constituting a moving image and sequentially input as a first image into a second image. A plurality of image conversion processing means (for example, the image processing units 213-1 to 213-3 in FIG. 14) and one of a plurality of processed images that are images after processing by each of the plurality of image conversion processing means. By selecting one processed image, a first mode for selecting one image conversion process from among a plurality of different image conversion processes, and a second mode for executing the image conversion process selected in the first mode Control means (for example, the control unit 217 in FIG. 14) for switching the mode, and each of the plurality of image conversion processing means sets a target pixel of interest in the second image to the plurality of classes. No Based on the class tap, a class tap selection means (for example, the tap selection unit 13 in FIG. 7) that selects pixels used for classifying into any one of the classes from the first image as a class tap, From the first image, a pixel used for calculation of class classification means for classifying the pixel of interest (for example, the class classification unit 14 in FIG. 7) and the tap coefficient when obtaining the pixel of interest is used as a prediction tap. Prediction tap selection means (for example, tap selection section 12 in FIG. 7) to select, tap coefficient output means (for example, coefficient output section 55 in FIG. 7) for outputting tap coefficients of the class of the target pixel, and the target pixel Computing means for obtaining the target pixel by performing a calculation using the tap coefficient of the class and the prediction tap selected for the target pixel (example) If, and a prediction computation unit 16) of Figure 7, between the first mode and the second mode, the class tap, the prediction taps, and switching the at least one of said tap coefficients.

  The feature amount of the input image is extracted, and a synchronization feature amount extraction unit (for example, the synchronization feature amount extraction unit 471 in FIG. 18) that detects a time code indicating which scene in the moving image the input image is. And a plurality of area specifying means for outputting area specifying information for specifying the predetermined area that each of the plurality of image conversion processing means expands corresponding to the time code detected by the synchronization feature amount extracting means (For example, the sequence re-Seibu 472A to 472C in FIG. 18), and each of the plurality of image conversion processing means includes the first image based on the region specifying information in the first mode. Can be enlarged.

  In the first mode, selection information that is information representing an enlarged image selected by the user from among the plurality of enlarged images obtained by enlarging the first image by the plurality of image conversion processing means, Based on the selection information stored in the user selection information storage means in the second mode in the user selection information storage means (for example, the recording / reproducing unit 474 of FIG. 18) that stores the code in association with the code, Selection means (for example, switcher unit 473 in FIG. 18) that selects one area specifying information from among the plurality of area specifying information supplied from the plurality of area specifying means and supplies the area specifying information to the plurality of image conversion processing means. In each of the plurality of image conversion processing means, the first image is enlarged based on the region specifying information in the second mode. And it can be configured to perform time resolution and enlargement processing to improve the time resolution.

  An image processing method or program according to one aspect of the present invention is an image conversion process for converting one or more input images sequentially constituting a moving image as a first image into a second image. In the image processing method of the image processing apparatus or the program for causing the computer to execute the image conversion process, the target pixel of interest in the second image is selected from any of the plurality of classes. The pixel used for classifying is selected from the first image as a class tap from the first image, the pixel of interest is classified based on the class tap, and the calculation with the tap coefficient when obtaining the pixel of interest Are selected from the first image as prediction taps, output the tap coefficients of the class of the target pixel, and the tap coefficients of the class of the target pixel By performing a calculation using the prediction tap selected for the target pixel, a plurality of image conversion processes for obtaining the target pixel are simultaneously performed, and one process is performed from among the processed images by the plurality of image conversion processes. A first mode in which one image conversion process is selected from a plurality of different image conversion processes by selecting a subsequent image, and a second mode in which the image conversion process selected in the first mode is executed. , And switching at least one of the class tap, the prediction tap, and the tap coefficient between the first mode and the second mode.

  Hereinafter, an embodiment of an image processing apparatus (system) to which the present invention is applied will be described. Before that, a class classification adaptive process used for signal processing performed by the image processing apparatus will be described. The class classification adaptive processing is an example of processing used for signal processing performed by the image processing apparatus, and the signal processing performed by the image processing apparatus may not use the class classification adaptive processing.

  Here, the class classification adaptation process will be described by taking an image conversion process for converting the first image data (image signal) to the second image data (image signal) as an example.

  The image conversion processing for converting the first image data into the second image data is various signal processing depending on the definition of the first and second image data.

  That is, for example, if the first image data is image data with low spatial resolution and the second image data is image data with high spatial resolution, the image conversion process creates a spatial resolution that improves the spatial resolution ( (Improved) processing.

  Also, for example, if the first image data is low S / N (Signal / Noise) image data and the second image data is high S / N image data, the image conversion process may reduce noise. It can be referred to as noise removal processing to be removed.

  Further, for example, if the first image data is image data having a predetermined number of pixels (size) and the second image data is image data in which the number of pixels of the first image data is increased or decreased, The image conversion process can be referred to as a resizing process for resizing (enlarging or reducing) an image.

  In addition, for example, if the first image data is image data with low time resolution and the second image data is image data with high time resolution, the image conversion process creates time resolution that improves the time resolution ( (Improved) processing.

  Further, for example, the first image data is set as decoded image data obtained by decoding image data encoded in units of blocks such as MPEG (Moving Picture Experts Group) encoding, and the second image data. Is image data before encoding, it can be said that the image conversion process is a distortion removal process for removing various distortions such as block distortion caused by MPEG encoding and decoding.

  In the spatial resolution creation process, when converting the first image data that is image data with low spatial resolution into the second image data that is image data with high spatial resolution, the second image data is converted into the second image data. The image data can have the same number of pixels as the one image data, or the image data can have more pixels than the first image data. When the second image data is image data having a larger number of pixels than the first image data, the spatial resolution creation process is a process for improving the spatial resolution and resizing to increase the image size (number of pixels). It is also a process.

  As described above, according to the image conversion process, various signal processes can be realized depending on how the first and second image data are defined.

  In the class classification adaptive processing as the image conversion processing as described above, the target pixel of interest (pixel value) in the second image data is classified into one of a plurality of classes. The pixel of interest (pixel value) is obtained by calculation using the tap coefficient of the class obtained by the above and the pixel (pixel value) of the first image data selected for the pixel of interest.

  That is, FIG. 1 shows a configuration example of an image conversion apparatus 1 that performs image conversion processing by class classification adaptation processing.

  In the image conversion apparatus 1, the image data supplied thereto is supplied to the tap selection units 12 and 13 as the first image data.

  The pixel-of-interest selection unit 11 sequentially sets pixels constituting the second image data as the pixel of interest, and supplies information representing the pixel of interest to a necessary block.

  The tap selection unit 12 selects some of the pixels (the pixel values) constituting the first image data used to predict the target pixel (the pixel values thereof) as prediction taps.

  Specifically, the tap selection unit 12 selects, as prediction taps, a plurality of pixels of the first image data that are spatially or temporally close to the temporal and spatial positions of the target pixel.

  The tap selection unit 13 selects some of the pixels constituting the first image data used for classifying the target pixel into any of several classes as class taps. That is, the tap selection unit 13 selects a class tap in the same manner as the tap selection unit 12 selects a prediction tap.

  Note that the prediction tap and the class tap may have the same tap structure or may have different tap structures.

  The prediction tap obtained by the tap selection unit 12 is supplied to the prediction calculation unit 16, and the class tap obtained by the tap selection unit 13 is supplied to the class classification unit 14.

  The class classification unit 14 classifies the target pixel based on the class tap from the tap selection unit 13 and supplies a class code corresponding to the resulting class to the coefficient output unit 15.

  Here, as a method of classifying, for example, ADRC (Adaptive Dynamic Range Coding) or the like can be employed.

  In the method using ADRC, the pixels constituting the class tap are subjected to ADRC processing, and the class of the pixel of interest is determined according to the ADRC code obtained as a result.

In the K-bit ADRC, for example, the maximum value MAX and the minimum value MIN of the pixels constituting the class tap are detected, and DR = MAX-MIN is set as the local dynamic range of the set, and this dynamic range Based on DR, the pixel value of each pixel constituting the class tap is requantized to K bits. That is, the pixel value of each pixel forming the class taps, the minimum value MIN is subtracted, and the subtracted value is divided by DR / 2 ^K (requantization). A bit string obtained by arranging the pixel values of the K-bit pixels constituting the class tap in a predetermined order is output as an ADRC code. Therefore, for example, when the class tap is subjected to 1-bit ADRC processing, the pixel value of each pixel constituting the class tap is divided by the average value of the maximum value MAX and the minimum value MIN (rounded down). Thereby, the pixel value of each pixel is set to 1 bit (binarized). Then, a bit string in which the 1-bit pixel values are arranged in a predetermined order is output as an ADRC code.

Note that, for example, the level distribution pattern of the pixel values of the pixels constituting the class tap may be output to the class classification unit 14 as it is as a class code. However, in this case, if the class tap is composed of pixel values of N pixels and K bits are assigned to the pixel values of each pixel, the number of class codes output by the class classification unit 14 Is (2 ^N ) ^K , which is a huge number that is exponentially proportional to the number K of bits of the pixel value of the pixel.

  Accordingly, the class classification unit 14 preferably performs class classification by compressing the information amount of the class tap by the above-described ADRC processing or vector quantization.

  The coefficient output unit 15 stores tap coefficients for each class obtained by learning described later, and is further stored in an address corresponding to the class code supplied from the class classification unit 14 among the stored tap coefficients. The tap coefficient (the tap coefficient of the class represented by the class code supplied from the class classification unit 14) is output. This tap coefficient is supplied to the prediction calculation unit 16.

  Here, the tap coefficient corresponds to a coefficient that is multiplied with input data in a so-called tap in the digital filter.

  The prediction calculation unit 16 acquires the prediction tap output from the tap selection unit 12 and the tap coefficient output from the coefficient output unit 15, and uses the prediction tap and the tap coefficient to predict the true value of the target pixel. Predetermined calculation for obtaining is performed. Thereby, the prediction calculation unit 16 obtains and outputs the pixel value (predicted value) of the pixel of interest, that is, the pixel value of the pixels constituting the second image data.

  Next, image conversion processing by the image conversion apparatus 1 in FIG. 1 will be described with reference to the flowchart in FIG.

  In step S11, the pixel-of-interest selecting unit 11 selects one of the pixels constituting the second image data for the first image data input to the image conversion device 1 that has not yet been set as the pixel of interest. Select as the pixel of interest. For example, among the pixels constituting the second image data, those not yet set as the target pixel in the raster scan order are selected as the target pixel.

  In step S12, the tap selection unit 12 and the tap selection unit 13 respectively select a prediction tap and a class tap for the target pixel from the first image data supplied thereto. The prediction tap is supplied from the tap selection unit 12 to the prediction calculation unit 16, and the class tap is supplied from the tap selection unit 13 to the class classification unit 14.

  The class classification unit 14 receives the class tap for the target pixel from the tap selection unit 13, and classifies the target pixel based on the class tap in step S13. Further, the class classification unit 14 outputs a class code representing the class of the target pixel obtained as a result of the class classification to the coefficient output unit 15.

  In step S14, the coefficient output unit 15 acquires and outputs the tap coefficient stored at the address corresponding to the class code supplied from the class classification unit 14. Furthermore, in step S14, the prediction calculation unit 16 acquires the tap coefficient output by the coefficient output unit 15.

  In step S <b> 15, the prediction calculation unit 16 performs a predetermined prediction calculation using the prediction tap output from the tap selection unit 12 and the tap coefficient acquired from the coefficient output unit 15. Thereby, the prediction calculation part 16 calculates | requires and outputs the pixel value of an attention pixel.

  In step S <b> 16, the target pixel selection unit 11 determines whether there is second image data that has not yet been set as the target pixel. If it is determined in step S16 that there is still the second image data that is not the pixel of interest, the process returns to step S11, and the same process is repeated thereafter.

  If it is determined in step S16 that there is no second image data that has not yet been set as the target pixel, the process ends.

  Next, the prediction calculation in the prediction calculation unit 16 of FIG. 1 and the learning of the tap coefficient stored in the coefficient output unit 15 will be described.

  Now, for example, the high-quality image data (high-quality image data) is used as the second image data, and the high-quality image data is filtered by an LPF (Low Pass Filter) to reduce the image quality (resolution). Using the low-quality image data (low-quality image data) as the first image data, a prediction tap is selected from the low-quality image data, and using the prediction tap and the tap coefficient, pixels (high-quality image data) Consider that the pixel value of (image quality pixel) is obtained (predicted) by a predetermined prediction calculation.

  For example, when a linear primary prediction calculation is adopted as the predetermined prediction calculation, the pixel value y of the high-quality pixel is obtained by the following linear linear expression.

In Equation (1), x _n represents a pixel value of an nth low-quality image data pixel (hereinafter referred to as a low-quality pixel as appropriate) that constitutes a prediction tap for the high-quality pixel y, and w _n represents the nth tap coefficient multiplied by the nth low image quality pixel (its pixel value). In equation (1), the prediction tap is assumed to be composed of N low image quality pixels x ₁ , x ₂ ,..., X _N.

  Here, the pixel value y of the high-quality pixel can be obtained not by the linear primary expression shown in Expression (1) but by a higher-order expression of the second or higher order.

Now, when the true value of the pixel value of the high-quality pixel of the k-th sample is expressed as y _k and the predicted value of the true value y _k obtained by the equation (1) is expressed as y _k ′, the prediction error e _k Is expressed by the following equation.

Now, since the predicted value y _k ′ of Equation (2) is obtained according to Equation (1), the following equation is obtained by replacing y _k ′ of Equation (2) according to Equation (1).

In Equation (3), x _{n, k} represents the nth low-quality pixel that constitutes the prediction tap for the high-quality pixel of the k-th sample.

Tap coefficient w _n for the prediction error e _k 0 of the formula (3) (or Equation (2)) is, is the optimal for predicting the high-quality pixel, for all the high-quality pixel, such In general, it is difficult to obtain a large tap coefficient w _n .

Therefore, as the standard for indicating that the tap coefficient w _n is optimal, for example, when adopting the method of least squares, optimal tap coefficient w _n, the sum E of square errors expressed by the following formula It can be obtained by minimizing.

However, in Equation (4), K is the high image quality pixel y _k and the low image quality pixels x _{1, k} , x _{2, k} ,..., X _N constituting the prediction tap for the high image quality pixel y _{k. , k} represents the number of samples (the number of learning samples).

The minimum value of the sum E of square errors of Equation (4) (minimum value), as shown in Equation (5), given that by partially differentiating the sum E with the tap coefficient w _n by w _n to 0.

Therefore, when partial differentiation of above equation (3) with the tap coefficient w _n, the following equation is obtained.

  From the equations (5) and (6), the following equation is obtained.

By substituting equation (3) into e _k in equation (7), equation (7) can be expressed by the normal equation shown in equation (8).

Normal equation of Equation (8), for example, by using a like sweeping-out method (Gauss-Jordan elimination method) can be solved for the tap coefficient w _n.

The normal equation of Equation (8), by solving for each class, the optimal tap coefficient (here, the tap coefficient that minimizes the sum E of square errors) to w _n, can be found for each class .

Figure 3 shows an example of the configuration of a learning apparatus 21 performs learning for determining the tap coefficient w _n by solving the normal equations in equation (8).

Learning image storage unit 31 stores the learning image data used for learning of the tap coefficient w _n. Here, as the learning image data, for example, high-resolution image data with high resolution can be used.

  The teacher data generation unit 32 reads learning image data from the learning image storage unit 31. Further, the teacher data generation unit 32 generates, from the learning image data, teacher data for the tap coefficient learning (true value), that is, teacher data that becomes a pixel value of a mapping destination as a prediction calculation according to the equation (1). And supplied to the teacher data storage unit 33. Here, for example, the teacher data generation unit 32 supplies high-quality image data as learning image data to the teacher data storage unit 33 as teacher data as it is.

  The teacher data storage unit 33 stores high-quality image data as teacher data supplied from the teacher data generation unit 32.

  The student data generation unit 34 reads learning image data from the learning image storage unit 31. Further, the student data generation unit 34 generates, from the learning image data, a student who learns the tap coefficient, that is, student data that is a pixel value to be converted by mapping as a prediction calculation according to Expression (1). The data is supplied to the storage unit 174. Here, the student data generation unit 34 generates low-quality image data by filtering the high-quality image data as the learning image data, for example, to reduce the resolution, and this low-quality image data is converted into the low-quality image data. , And supplied to the student data storage unit 35 as student data.

  The student data storage unit 35 stores the student data supplied from the student data generation unit 34.

  The learning unit 36 sequentially sets pixels constituting the high-quality image data as the teacher data stored in the teacher data storage unit 33 as the target pixel, and the student data stored in the student data storage unit 35 for the target pixel. Among the low image quality pixels constituting the low image quality image data, a low image quality pixel having the same tap structure as that selected by the tap selection unit 12 in FIG. 1 is selected as a prediction tap. Further, the learning unit 36 uses each pixel constituting the teacher data and the prediction tap selected when the pixel is the target pixel, and establishes a normal equation of Expression (8) for each class. By solving, the tap coefficient for each class is obtained.

  That is, FIG. 4 shows a configuration example of the learning unit 36 of FIG.

  The target pixel selection unit 41 sequentially selects pixels constituting the teacher data stored in the teacher data storage unit 33 as the target pixel, and supplies information representing the target pixel to a necessary block.

  The tap selection unit 42 is the same pixel as the pixel of interest selected by the tap selection unit 12 in FIG. 1 from the low image quality pixels constituting the low image quality image data as the student data stored in the student data storage unit 35. As a result, a prediction tap having the same tap structure as that obtained by the tap selection unit 12 is obtained and supplied to the adding unit 45.

  The tap selection unit 43 is the same pixel as the pixel of interest selected by the tap selection unit 13 in FIG. 1 from the low image quality pixels constituting the low image quality image data as the student data stored in the student data storage unit 35. Thus, a class tap having the same tap structure as that obtained by the tap selection unit 13 is obtained and supplied to the class classification unit 44.

  The class classification unit 44 performs the same class classification as the class classification unit 14 of FIG. 1 based on the class tap output from the tap selection unit 43, and adds the class code corresponding to the resulting class to the addition unit 45. Output.

  The adding unit 45 reads the teacher data (pixels) that are the target pixel from the teacher data storage unit 33, and configures the target pixel and the prediction tap for the target pixel supplied from the tap selection unit 42. Addition for data (pixels) is performed for each class code supplied from the class classification unit 44.

That is, the addition unit 45 is supplied with the teacher data y _k stored in the teacher data storage unit 33, the prediction tap x _{n, k} output from the tap selection unit 42, and the class code output from the class classification unit 44. .

Then, the adding unit 45 uses the prediction tap (student data) x _{n, k} for each class corresponding to the class code supplied from the class classifying unit 44 _, and uses the prediction data (student data) x _{n, k} for the student data in the matrix on the left side of equation (8). (X _{n, k} x _{n ′, k} ) and computation corresponding to summation (Σ).

Furthermore, the adding unit 45 also uses the prediction tap (student data) x _{n, k} and the teacher data y _k for each class corresponding to the class code supplied from the class classification unit 44, and the equation (8) Multiplication (x _{n, k} y _k ) of student data x _{n, k} and teacher data y _k in the vector on the right side and calculation corresponding to summation (Σ) are performed.

That is, the adding unit 45 performs the left-side matrix component (Σx _{n, k} x _{n ′, k} ) and the right-side vector component ( Σx _{n, k} y _k ) is stored in its built-in memory (not shown), and the matrix component (Σx _{n, k} x _{n ′, k} ) or vector component (Σx _{n, k} y _k) ), The corresponding component x _{n, k + 1} x _n calculated using the teacher data y _{k + 1} and the student data x _{n, k + 1} for the teacher data newly set as the pixel of interest. _{', k + 1} or x _{n, k + 1} y _{k + 1} is added (addition represented by the summation of equation (8) is performed).

  Then, the addition unit 45 performs the above-described addition using all the teacher data stored in the teacher data storage unit 33 (FIG. 3) as the target pixel, so that the normality shown in Expression (8) is obtained for each class. When the equation is established, the normal equation is supplied to the tap coefficient calculation unit 46.

Tap coefficient calculating section 46 solves the normal equations for each class supplied from the adder 45, for each class, and outputs the determined optimal tap coefficient w _n.

The coefficient output unit 15 in the image conversion apparatus 1 in FIG. 1, the tap coefficient w _n for each class determined as described above is stored.

  Here, as described above, various tap coefficients may be used depending on how image data to be used as student data corresponding to the first image data and image data to be used as teacher data corresponding to the second image data are selected. What performs the image conversion process of this can be obtained.

  That is, as described above, the high-quality image data is used as teacher data corresponding to the second image data, and the low-quality image data obtained by degrading the spatial resolution of the high-quality image data is used as the first image data. As a tap coefficient, the first image which is low-quality image data (SD (Standard Definition) image) is shown as the first tap coefficient from the top of FIG. It is possible to obtain what performs image conversion processing as spatial resolution creation processing for converting data into second image data that is high-quality image data (HD (High Definition) image data) with improved spatial resolution. .

  In this case, the first image data (student data) may have the same or less number of pixels as the second image data (teacher data).

  Also, for example, the high-quality image data is used as teacher data, and the tap coefficient is learned by using, as the student data, image data on which noise is superimposed on the high-quality image data as the teacher data. As shown second from the top in FIG. 5, the first image data, which is low S / N image data, is high S / N image data obtained by removing (reducing) noise contained therein. What performs an image conversion process as a noise removal process converted into 2nd image data can be obtained.

  Further, for example, by performing tap coefficient learning by using a certain piece of image data as teacher data and learning the tap coefficient using image data obtained by thinning out the number of pixels of the image data as the teacher data as the tap data, FIG. As shown in the third from the top, enlargement processing (resizing processing) for converting first image data that is a part of image data into second image data that is enlarged image data obtained by enlarging the first image data ) For performing the image conversion process.

  Note that the tap coefficient for performing the enlargement processing is a tap using high-quality image data as teacher data and low-quality image data obtained by degrading the spatial resolution of the high-quality image data by thinning out the number of pixels as student data. It can also be obtained by learning coefficients.

  In addition, for example, by using the high frame rate image data as the teacher data and learning the tap coefficient using the image data obtained by thinning out the frames of the high frame rate image data as the teacher data as the student data, taps are performed. As a coefficient, as shown in the fourth (first bottom) from the top in FIG. 5, as a time resolution creation process for converting the first image data having a predetermined frame rate into the second image data having a high frame rate. What performs an image conversion process can be obtained.

  Next, processing (learning processing) of the learning device 21 in FIG. 3 will be described with reference to the flowchart in FIG.

  First, in step S 21, the teacher data generation unit 32 and the student data generation unit 34 generate teacher data and student data from the learning image data stored in the learning image storage unit 31, and the teacher data storage unit 33. Are supplied to and stored in the student data generation unit.

  Note that what kind of student data and teacher data are generated in the teacher data generation unit 32 and the student data generation unit 34, respectively, is a tap coefficient for which of the types of image conversion processing as described above. It depends on what you learn.

  Thereafter, the process proceeds to step S22, and in the learning unit 36 (FIG. 4), the target pixel selection unit 41 sets the target pixel of the teacher data stored in the teacher data storage unit 33 that has not yet been set as the target pixel. Choose as. In step S23, the tap selection unit 42 selects a pixel as student data to be a prediction tap from the student data stored in the student data storage unit 35 for the target pixel, and supplies the selected pixel to the addition unit 45 and tap selection. Again, the unit 43 selects student data as class taps from the student data stored in the student data storage unit 35 for the pixel of interest, and supplies the selected class data to the class classification unit 44.

  In step S <b> 24, the class classification unit 44 performs class classification of the pixel of interest based on the class tap for the pixel of interest, and outputs a class code corresponding to the class obtained as a result to the adding unit 45.

  In step S <b> 25, the adding unit 45 reads the target pixel from the teacher data storage unit 33, and obtains the target pixel and student data constituting the prediction tap selected for the target pixel supplied from the tap selection unit 42. The addition of the target expression (8) is performed for each class code supplied from the class classification unit 44.

  In step S <b> 26, the pixel-of-interest selection unit 41 determines whether teacher data that is not yet a pixel of interest is stored in the teacher data storage unit 33. If it is determined in step S26 that teacher data that is not a pixel of interest is still stored in the teacher data storage unit 33, the process returns to step S22, and the same process is repeated thereafter.

  If it is determined in step S26 that teacher data that is not a pixel of interest is not stored in the teacher data storage unit 33, the process proceeds to step S27, and the adding unit 45 performs steps S22 to S26 so far. The matrix on the left side and the vector on the right side in the equation (8) for each class obtained by the above process are supplied to the tap coefficient calculation unit 46.

Furthermore, in step S27, the tap coefficient calculation unit 46 solves the normal equation for each class constituted by the matrix on the left side and the vector on the right side in the equation (8) for each class supplied from the addition unit 45, for each class, and determines and outputs the tap coefficient w _n, the process ends.

  It should be noted that due to the number of learning image data being insufficient, etc., there may occur a class in which the number of normal equations necessary for obtaining the tap coefficient cannot be obtained. The tap coefficient calculation unit 46 outputs a default tap coefficient, for example.

  FIG. 7 shows a configuration example of an image conversion device 51 that is another image conversion device that performs image conversion processing by class classification adaptation processing.

  In the figure, portions corresponding to those in FIG. 1 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, the image conversion apparatus 51 is configured in the same manner as the image conversion apparatus 1 in FIG. 1 except that a coefficient output unit 55 is provided instead of the coefficient output unit 15.

  In addition to the class (class code) supplied from the class classification unit 14, the coefficient output unit 55 is supplied with, for example, a parameter z input from the outside in response to a user operation. The coefficient output unit 55 generates a tap coefficient for each class corresponding to the parameter z as will be described later, and predicts the tap coefficient of the class from the class classification unit 14 out of the tap coefficients for each class. To the unit 16.

  FIG. 8 shows a configuration example of the coefficient output unit 55 of FIG.

  The coefficient generator 61 generates a tap coefficient for each class based on the coefficient seed data stored in the coefficient seed memory 62 and the parameter z stored in the parameter memory 63, and supplies the tap coefficient to the coefficient memory 64. Remember to overwrite.

  The coefficient seed memory 62 stores coefficient seed data for each class obtained by learning of coefficient seed data described later. Here, the coefficient seed data is so-called seed data that generates tap coefficients.

  The parameter memory 63 stores the parameter z input from the outside in accordance with a user operation or the like in an overwritten form.

  The coefficient memory 64 stores the tap coefficient for each class supplied from the coefficient generation unit 61 (the tap coefficient for each class corresponding to the parameter z). And the coefficient memory 64 reads the tap coefficient of the class supplied from the class classification | category part 14 (FIG. 7), and outputs it to the prediction calculating part 16 (FIG. 7).

  In the image conversion apparatus 51 of FIG. 7, when the parameter z is input from the outside to the coefficient output unit 55, the parameter z is overwritten in the parameter memory 63 of the coefficient output unit 55 (FIG. 8). Remembered.

  When the parameter z is stored in the parameter memory 63 (when the storage content of the parameter memory 63 is updated), the coefficient generation unit 61 reads out the coefficient seed data for each class from the coefficient seed memory 62 and from the parameter memory 63. The parameter z is read, and the tap coefficient for each class is obtained based on the coefficient seed data and the parameter z. Then, the coefficient generation unit 61 supplies the tap coefficient for each class to the coefficient memory 64 and stores it in the form of overwriting.

  The image conversion device 51 stores tap coefficients, and generates and outputs tap coefficients corresponding to the parameter z in a coefficient output unit 55 provided in place of the coefficient output unit 15 that outputs the tap coefficients. Except for this, processing similar to the processing according to the flowchart of FIG. 2 performed by the image conversion apparatus 1 of FIG. 1 is performed.

  Next, prediction calculation in the prediction calculation unit 16 in FIG. 7, generation of tap coefficients in the coefficient generation unit 61 in FIG. 8, and learning of coefficient seed data stored in the coefficient seed memory 62 will be described.

  As in the embodiment of FIG. 1, low-quality image data in which high-quality image data (high-quality image data) is used as second image data and the spatial resolution of the high-quality image data is reduced. (Low-quality image data) is the first image data, a prediction tap is selected from the low-quality image data, and the pixel value of the high-quality pixel that is a pixel of the high-quality image data is selected using the prediction tap and the tap coefficient. For example, consider obtaining (predicting) by linear primary prediction calculation of equation (1).

  Here, the pixel value y of the high-quality pixel can be obtained not by the linear primary expression shown in Expression (1) but by a higher-order expression of the second or higher order.

In the embodiment of FIG. 8, the coefficient generation unit 61 generates the tap coefficient w _n from the coefficient seed data stored in the coefficient seed memory 62 and the parameter z stored in the parameter memory 63. generating the tap coefficient w _n in the coefficient generating unit 61, for example, to be performed by the following equation using the coefficient seed data and the parameter z.

However, in the equation (9), beta _{m, n} denotes the m-th coefficient seed data used for determining the n-th tap coefficient w _n. In the equation (9), the tap coefficient w _n is obtained using M coefficient seed data β _{1, n} , β _{2, n} ,..., Β _{M, n} .

Here, the formula for obtaining the tap coefficient w _n from the coefficient seed data β _{m, n} and the parameter z is not limited to the formula (9).

Now, a value z ^m−1 determined by the parameter z in equation (9) is defined by the following equation by introducing a new variable t _m .

  By substituting equation (10) into equation (9), the following equation is obtained.

According to equation (11), the tap coefficient w _n will be asked by the linear first-order equation of the coefficient seed data beta _{m, n} and the variable t _m.

Now, when the true value of the pixel value of the high-quality pixel of the k-th sample is expressed as y _k and the predicted value of the true value y _k obtained by the equation (1) is expressed as y _k ′, the prediction error e _k is expressed by the following equation.

Now, since the predicted value y _k ′ of Expression (12) is obtained according to Expression (1), when y _k ′ of Expression (12) is replaced according to Expression (1), the following expression is obtained.

In Equation (13), x _{n, k} represents the n-th low-quality pixel that constitutes the prediction tap for the high-quality pixel of the k-th sample.

To w _n of formula (13), by substituting equation (11), the following equation is obtained.

Prediction error e _k coefficient seed data beta _{m, n} to 0 in Equation (14), is the optimal for predicting the high-quality pixel, for all the high-quality pixel, such coefficient seed data It is generally difficult to obtain β _{m, n} .

Therefore, as a standard indicating that the coefficient seed data β _{m, n} is optimal, for example, when the least square method is adopted, the optimal coefficient seed data β _{m, n} is expressed by the following equation. It can be obtained by minimizing the sum E of square errors.

However, in Expression (15), K is a high-quality pixel y _k and low-quality pixels x _{1, k} , x _{2, k} ,..., X _N that constitute a prediction tap for the high-quality pixel y _{k. , k} represents the number of samples (the number of learning samples).

The minimum value (minimum value) of the sum E of squared errors in Equation (15) is β _{m, where} 0 is a partial differentiation of the sum E with coefficient seed data β _{m, n} as shown in Equation (16) _. given by _n .

  By substituting equation (13) into equation (16), the following equation is obtained.

Now, X _{i, p, j, q} and Y _{i, p} are defined as shown in equations (18) and (19).

In this case, Expression (17) can be expressed by a normal equation shown in Expression (20) using X _{i, p, j, q} and Y _{i, p} .

The normal equation of Expression (20) can be solved for the coefficient seed data β _{m, n} by using, for example, a sweeping method (Gauss-Jordan elimination method) or the like.

In the image conversion device 51 of FIG. 7, a large number of high-quality pixels y ₁ , y ₂ ,..., Y _K are used as teacher data for learning, and prediction taps for the respective high-quality pixels y _k are used. low quality pixels x _{1, k} that _{constitute, x 2, k, ···,} x N, as student data serving as a student learning _k, the vertical and solves learning normal equation of formula (20) for each class The coefficient seed data β _{m, n} for each class obtained by performing is stored in the coefficient seed memory 62 of the coefficient output unit 55 (FIG. 8), and the coefficient generation unit 61 stores the coefficient seed data β _{m, n.} and _n, from the stored parameter z in the parameter memory 63, according to equation (9), the tap coefficient w _n for each class is generated. Then, the prediction calculation unit 16 uses the tap coefficient w _n and the low image quality pixel (the pixel of the first image data) x _n constituting the prediction tap for the pixel of interest as the high image quality pixel, and the equation (1) ) Is calculated, the pixel value of the target pixel as a high-quality pixel (predicted value close to it) is obtained.

FIG. 9 shows a configuration example of a learning device 71 that performs learning for obtaining coefficient seed data β _{m, n} for each class by solving the normal equation of Expression (20) for each class.

  In the figure, portions corresponding to those in the learning device 21 of FIG. 3 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate. That is, the learning device 71 is provided with a student data generation unit 74 and a learning unit 76 instead of the student data generation unit 34 and the learning unit 36, respectively, and a parameter generation unit 81 is newly provided. 3 is configured in the same manner as the learning device 21 in FIG.

  The student data generation unit 74 generates student data from the learning image data, and supplies the student data to the student data storage unit 35 to store the same as in the student data generation unit 34 of FIG.

  However, in addition to the learning image data, the student data generation unit 74 is supplied with some values in the range that can be taken by the parameter z supplied to the parameter memory 63 of FIG. It has become. That is, assuming that the value that can be taken by the parameter z is a real number in the range of 0 to Z, the student data generation unit 74 has, for example, z = 0, 1, 2,. It is supplied from the section 81.

  The student data generation unit 74 filters low-quality image data as student data by filtering high-quality image data as learning image data using, for example, an LPF having a cutoff frequency corresponding to the parameter z supplied thereto. Is generated.

  Accordingly, the student data generation unit 74 generates Z + 1 types of low-quality image data as student data having different spatial resolutions for the high-quality image data as the learning image data.

  Note that, here, for example, as the value of the parameter z increases, the high-quality image data is filtered by using an LPF with a high cutoff frequency, and low-quality image data as student data is generated. Therefore, here, the lower the image quality data corresponding to the larger parameter z, the higher the spatial resolution.

  In the present embodiment, in order to simplify the explanation, the student data generation unit 74 reduces the spatial resolution of the high-quality image data in both the horizontal and vertical directions by an amount corresponding to the parameter z. Assume that low-quality image data is generated.

  The learning unit 76 uses the teacher data stored in the teacher data storage unit 33, the student data stored in the student data storage unit 35, and the parameter z supplied from the parameter generation unit 81, so that the coefficient seed data for each class. Is output.

  The parameter generation unit 81 generates, for example, z = 0, 1, 2,..., Z as described above as several values in the range that the parameter z can take, and learns with the student data generation unit 74. To the unit 76.

  FIG. 10 shows a configuration example of the learning unit 76 of FIG. In the figure, portions corresponding to those in the learning unit 36 in FIG. 4 are denoted by the same reference numerals, and description thereof will be omitted below as appropriate.

  As with the tap selection unit 42 in FIG. 4, the tap selection unit 92 performs the tap in FIG. 7 from the low quality pixels that constitute the low quality image data as the student data stored in the student data storage unit 35 for the target pixel. A prediction tap having the same tap structure as that selected by the selection unit 12 is selected and supplied to the addition unit 95.

  As with the tap selection unit 43 in FIG. 4, the tap selection unit 93 also taps the pixel of interest from the low-quality pixels constituting the low-quality image data as the student data stored in the student data storage unit 35 with respect to the target pixel. A class tap having the same tap structure as the selection unit 13 selects is selected and supplied to the class classification unit 44.

  However, in FIG. 10, the parameter z generated by the parameter generation unit 81 in FIG. 9 is supplied to the tap selection units 42 and 43, and the tap selection units 42 and 43 are supplied from the parameter generation unit 81. Prediction taps and class taps are generated from student data generated corresponding to the parameter z to be generated (here, low-quality image data as student data generated using the LPF of the cutoff frequency corresponding to the parameter z). Select each one.

  The adding unit 95 reads out the target pixel from the teacher data storage unit 33 in FIG. 9, student data constituting the target pixel, the prediction tap configured for the target pixel supplied from the tap selection unit 42, and the student The addition for the parameter z when the data is generated is performed for each class supplied from the class classification unit 44.

In other words, the addition unit 95 includes the teacher data y _k as the target pixel stored in the teacher data storage unit 33 and the prediction tap x _{i, k} (x _{j, k} ) for the target pixel output from the tap selection unit 42. The class of the pixel of interest output from the class classification unit 44 is supplied, and the parameter z when the student data constituting the prediction tap for the pixel of interest is generated is supplied from the parameter generation unit 81.

Then, the adding unit 95 uses the prediction tap (student data) x _{i, k} (x _{j, k} ) and the parameter z for each class supplied from the class classification unit 44 _, and the matrix on the left side of Expression (20). Of the student data and the parameter z for obtaining the component X _{i, p, j, q} defined by the equation (18) (x _{i, k} t _p x _{j, k} t _q ) and summation (Σ ) Is performed. Incidentally, t _p of formula (18), according to equation (10), is calculated from the parameter z. The same applies to _{tq in} equation (18).

Further, the adding unit 95 also uses the prediction tap (student data) x _{i, k} , the teacher data y _k , and the parameter z for each class corresponding to the class code supplied from the class classification unit 44, and uses the formula z Multiplication (x _{i, k} t _p ) of student data x _{i, k} , teacher data y _k , and parameter z for obtaining the component Y _{i, p} defined by equation (19) in the vector on the right side of (20) y _k ) and a calculation corresponding to summation (Σ). Incidentally, t _p of formula (19), according to equation (10), is calculated from the parameter z.

That is, the adding unit 95 performs the left-side matrix component X _{i, p, j, q} and the right-side vector component Y _{i, p} in Expression (20) previously obtained for the teacher data set as the target pixel. Is stored in its built-in memory (not shown), and the teacher data newly set as the pixel of interest for the matrix component X _{i, p, j, q} or the vector component Y _{i, p} , The corresponding component x _{i, k} t _p x _{j, k} t _q or x _i, calculated using the teacher data y _k , student data x _{i, k} (x _{j, k} ), and parameter z _k t _p y _k is added (addition represented by summation in the component X _{i, p, j, q in the} equation (18) or the component Y _{i, p} in the equation (19) is performed).

  Then, the addition unit 95 performs the above-described addition for all the parameter data z of 0, 1,..., Z, using all the teacher data stored in the teacher data storage unit 33 as the target pixel. Thus, when the normal equation shown in the equation (20) is established for each class, the normal equation is supplied to the coefficient seed calculating unit 96.

The coefficient seed calculator 96 obtains and outputs coefficient seed data β _{m, n} for each class by solving the normal equation for each class supplied from the adder 95.

  Next, processing (learning processing) of the learning device 71 in FIG. 9 will be described with reference to the flowchart in FIG.

  First, in step S31, the teacher data generation unit 32 and the student data generation unit 74 generate and output teacher data and student data from the learning image data stored in the learning image storage unit 31, respectively. That is, the teacher data generation unit 32 outputs the learning image data as it is, for example, as teacher data. In addition, the student data generation unit 74 is supplied with a parameter z of Z + 1 values generated by the parameter generation unit 81. For example, the student data generation unit 74 filters the learning image data with an LPF having a cutoff frequency corresponding to the parameter z of Z + 1 values (0, 1,..., Z) from the parameter generation unit 81. Thus, the Z + 1 frame student data is generated and output for the teacher data (learning image data) of each frame.

  The teacher data output from the teacher data generation unit 32 is supplied to and stored in the teacher data storage unit 33, and the student data output from the student data generation unit 74 is supplied to and stored in the student data storage unit 35.

  Thereafter, in step S32, the parameter generation unit 81 sets the parameter z to, for example, 0 as an initial value, and supplies the parameter z to the tap selection units 42 and 43 and the addition unit 95 of the learning unit 76 (FIG. 10). . In step S <b> 33, the target pixel selection unit 41 selects, as the target pixel, the teacher data stored in the teacher data storage unit 33 that has not yet been set as the target pixel.

  In step S34, the tap selection unit 42 stores the student data corresponding to the parameter z output from the parameter generation unit 81 and stored in the student data storage unit 35 for the target pixel (for learning corresponding to the teacher data serving as the target pixel). A prediction tap is selected from the student data generated by filtering the image data with the LPF having the cutoff frequency corresponding to the parameter z, and supplied to the adding unit 95. Furthermore, in step S34, the tap selection unit 43 selects a class tap from the student data corresponding to the parameter z output from the parameter generation unit 81 and stored in the student data storage unit 35 for the target pixel, and the class classification unit. 44.

  In step S 35, the class classification unit 44 classifies the target pixel based on the class tap for the target pixel, and outputs the class of the target pixel obtained as a result to the adding unit 95.

In step S <b> 36, the adding unit 95 reads the target pixel from the teacher data storage unit 33, uses the target pixel, the prediction tap supplied from the tap selection unit 42, and the parameter z output from the parameter generation unit 81. In step 20), the left side matrix component x _{i, K} t _{p xj, K} t _q and the right side vector component x _{i, K} t _p y _K are calculated. Further, the addition unit 95 applies the target pixel, the prediction tap, and the parameter z to the matrix component and the vector component that have already been obtained, corresponding to the class of the target pixel from the class classification unit 44. The matrix components x _{i, K} t _p x _{j, K} t _q obtained from the above are added to the vector components x _{i, K} t _p y _K.

  In step S37, the parameter generation unit 81 determines whether or not the parameter z output by itself is equal to Z which is the maximum value that can be taken. In step S37, when it is determined that the parameter z output from the parameter generation unit 81 is not equal to the maximum value Z (less than the maximum value Z), the process proceeds to step S38, and the parameter generation unit 81 1 is added to z, and the added value is output as a new parameter z to the tap selection units 42 and 43 and the addition unit 95 of the learning unit 76 (FIG. 10). Then, the process returns to step S34, and the same process is repeated thereafter.

  If it is determined in step S37 that the parameter z is equal to the maximum value Z, the process proceeds to step S39, and the pixel-of-interest selection unit 41 stores in the teacher data storage unit 33 teacher data that is not yet a pixel of interest. Determine if it is stored. If it is determined in step S38 that teacher data that is not a pixel of interest is still stored in the teacher data storage unit 33, the process returns to step S32, and the same process is repeated thereafter.

  If it is determined in step S39 that the teacher data that is not the pixel of interest is not stored in the teacher data storage unit 33, the process proceeds to step S40, and the adding unit 95 is obtained by the processing so far. The matrix on the left side and the vector on the right side in the equation (20) for each class are supplied to the coefficient seed calculation unit 96.

Further, in step S40, the coefficient seed calculating unit 96 solves the normal equation for each class constituted by the matrix on the left side and the vector on the right side in the equation (20) for each class supplied from the adding unit 95, The coefficient seed data β _{m, n} is obtained and output for each class, and the process is terminated.

  In addition, due to the number of learning image data being insufficient, etc., there may occur a class in which the number of normal equations necessary for obtaining coefficient seed data cannot be obtained. The coefficient seed calculation unit 96 outputs default coefficient seed data, for example.

By the way, in the learning device 71 of FIG. 9, the high-quality image data as the learning image data is used as the teacher data, and the low-quality image data in which the spatial resolution of the high-quality image data is degraded in accordance with the parameter z. Is the coefficient seed data β _{m, n} that minimizes the sum of the square error of the predicted value y of the teacher data predicted from the tap coefficient w _n and the student data x _{n by} the linear linear expression of Expression (1). However, the learning of the coefficient seed data β _{m, n} can be performed as follows, for example.

That is, the learning device 71 uses the high-quality image data as the learning image data as the teacher data, and filters the high-quality image data with the LPF having the cutoff frequency corresponding to the parameter z, thereby obtaining the horizontal resolution. and a low-quality image data with a reduced vertical resolution as student data, first, the prediction of the teacher data to be predicted by using the tap coefficient w _n and the student data x _n in the linear first-order prediction equation of the formula (1) A tap coefficient w _n that minimizes the sum of square errors of the value y is obtained for each value of the parameter z (here, z = 0, 1,..., Z). Then, the learning apparatus 71, as well as the tap coefficient w _n which is determined for each value of the parameter z and the teacher data, the parameter z as the learner data, the coefficient seed data beta _{m, n} and student data by equation (11) obtaining coefficient seed data beta _{m, n} that minimizes the sum of square errors of the prediction value of the tap coefficient w _n of the parameter z as the teacher data is predicted from the corresponding variable t _m is.

Here, the tap coefficient w _n that minimizes the sum E of the squared errors of the predicted values y of the teacher data predicted by the linear primary prediction expression of Expression (1) is the case in the learning device 21 of FIG. Similarly to the above, by solving and solving the normal equation of the equation (8), each class can be obtained for each value of the parameter z (z = 0, 1,..., Z).

Incidentally, the tap coefficient is obtained from the coefficient seed data β _{m, n} and the variable t _m corresponding to the parameter z, as shown in the equation (11). Now, assuming that the tap coefficient obtained by this equation (11) is represented as w _n ′, the optimum tap coefficient w _n represented by the following equation (21) and equation (11) are obtained. coefficient seed data β _{m, n} of the error e _n of the tap coefficient w _n 'and 0, becomes the optimum coefficient seed data for determining the optimal tap coefficient w _n, for all the tap coefficients w _n, It is generally difficult to obtain such coefficient seed data β _{m, n} .

  Equation (21) can be transformed into the following equation by Equation (11).

Therefore, as a standard indicating that the coefficient seed data β _{m, n} is optimal, for example, if the least square method is adopted, the optimal coefficient seed data β _{m, n} is _expressed by the following equation. This can be obtained by minimizing the sum E of squared errors.

The minimum value (minimum value) of the sum E of square errors in equation (23) is β _{m, where} 0 is a partial differentiation of the sum E with coefficient seed data β _{m, n} , as shown in equation (24) _. given by _n .

  By substituting equation (22) into equation (24), the following equation is obtained.

Now, X _{i, j,} and Y _i are defined as shown in equations (26) and (27).

In this case, Expression (25) can be expressed by a normal equation shown in Expression (28) using X _{i, j} and Y _i .

The normal equation of Expression (28) can also be solved for the coefficient seed data β _{m, n} by using, for example, a sweeping method (Gauss-Jordan elimination method).

FIG. 12 shows a configuration example of the learning apparatus 101 that performs learning for obtaining the coefficient seed data β _{m, n} by solving the normal equation of Expression (28).

  In the figure, portions corresponding to those in the learning device 21 of FIG. 3 or the learning device 71 of FIG. 9 are denoted by the same reference numerals, and description thereof will be omitted as appropriate. That is, the learning device 101 is configured in the same manner as the learning device 71 in FIG. 9 except that a learning unit 106 is provided instead of the learning unit 76.

  FIG. 13 shows a configuration example of the learning unit 106 of FIG. In the figure, portions corresponding to those in the learning unit 36 of FIG. 4 or the learning unit 76 of FIG. 10 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The adding unit 115 is supplied with the class of the pixel of interest output from the class classification unit 44 and the parameter z output from the parameter generation unit 81. Then, the adding unit 115 reads out the target pixel from the teacher data storage unit 33, and adds the target pixel and the student data constituting the prediction tap for the target pixel supplied from the tap selection unit 42 as targets. Is performed for each class supplied from the class classification unit 44 and for each value of the parameter z output from the parameter generation unit 81.

That is, the adding unit 115 includes the teacher data y _k stored in the teacher data storage unit 33 (FIG. 12), the prediction tap x _{n, k} output from the tap selection unit 42, the class output from the class classification unit 44, And the parameter z when the student data which comprises the prediction tap _{xn, k} which the parameter production | generation part 81 (FIG. 12) outputs is produced | generated is supplied.

Then, the adding unit 115 uses the prediction tap (student data) x _{n, k} for each class supplied from the class classification unit 44 and for each value of the parameter z output from the parameter generation unit 81, 8) Multiply (x _{n, k} x _{n ′, k} ) between student data in the left-side matrix and an operation corresponding to summation (Σ).

Furthermore, the addition unit 115 also uses the prediction tap (student data) x _{n, k} and the teacher data for each class supplied from the class classification unit 44 and for each value of the parameter z output from the parameter generation unit 81. with y _k, performed student data x _n in the vector of the right side of equation _{(8), k} and multiplication of teacher data _{y k (x n, k y} k) and, a calculation corresponding to summation (sigma).

That is, the adding unit 115 lastly calculates the left-side matrix component (Σx _{n, k} x _{n ′, k} ) and the right-side vector component ( Σx _{n, k} y _k ) is stored in its built-in memory (not shown), and the matrix component (Σx _{n, k} x _{n ′, k} ) or vector component (Σx _{n, k} y _k) ), The corresponding component x _{n, k + 1} x _n calculated using the teacher data y _{k + 1} and the student data x _{n, k + 1} for the teacher data newly set as the pixel of interest. _{', k + 1} or x _{n, k + 1} y _{k + 1} is added (addition represented by the summation of equation (8) is performed).

  Then, the adding unit 115 performs the above-described addition using all the teacher data stored in the teacher data storage unit 33 as the target pixel, thereby obtaining the equation (8) for each value of the parameter z for each class. When the normal equation shown in (2) is established, the normal equation is supplied to the tap coefficient calculation unit 46.

  Therefore, the adding unit 115 forms the normal equation of the equation (8) for each class, similarly to the adding unit 45 of FIG. However, the addition unit 115 is further different from the addition unit 45 of FIG. 4 in that a normal equation of Expression (8) is established for each value of the parameter z.

The tap coefficient calculation unit 46 solves a normal equation for each value of the parameter z for each class supplied from the adding unit 115, thereby obtaining an optimum tap coefficient w _n for each value of the parameter z for each class. Obtained and supplied to the add-in unit 121.

Adder 121 adds as a parameter generating unit 81 (variable t _m corresponding to) the parameter z supplied from (12), directed to the optimum tap coefficients w _n supplied from the tap coefficient calculating section 46 For each class.

That is, the adding unit 121 uses the variable t _i (t _j ) obtained by the equation (10) from the parameter z supplied from the parameter generating unit 81 (FIG. 12), and in the matrix on the left side of the equation (28), Multiplication (t _i t _j ) between variables t _i (t _j ) corresponding to the parameter z for _{obtaining the} component X _{i, j} defined by the equation (26), and an operation corresponding to summation (Σ) , For each class.

Here, since the component X _{i, j} is determined only by the parameter z and has no relation to the class _, the calculation of the component X _{i, j} does not actually need to be performed for each class, and is performed once. Just do it.

Further, the adding unit 121 uses the variable t _i obtained from the parameter z supplied from the parameter generating unit 81 by the equation (10) and the optimum tap coefficient w _n supplied from the tap coefficient calculating unit 46, Multiplication (t _i w _n ) of the variable t _i corresponding to the parameter z for _{obtaining the} component Y _i defined in the expression (27) and the optimum tap coefficient w _n in the vector on the right side of the expression (28); An operation corresponding to summation (Σ) is performed for each class.

The adding unit 121 obtains the component X _{i, j} represented by the equation (26) and the component Y _i represented by the equation (27) for each class, thereby obtaining the equation (28) for each class. The normal equation is supplied to the coefficient seed calculation unit 122.

The coefficient seed calculating unit 122 calculates and outputs coefficient seed data β _{m, n} for each class by solving the normal equation of the equation (28) for each class supplied from the adding unit 121.

The coefficient seed memory 62 in the coefficient output unit 55 of FIG. 8 may store the coefficient seed data β _{m, n} for each class obtained as described above.

  In the learning of the coefficient seed data, the student data corresponding to the first image data and the teacher data corresponding to the second image data are the same as in the tap coefficient learning described with reference to FIG. Depending on how the data is selected, the coefficient seed data can be obtained by performing various image conversion processes.

  That is, in the above-described case, the learning image data is directly used as teacher data corresponding to the second image data, and the low-quality image data in which the spatial resolution of the learning image data is deteriorated is the first image data. Since the coefficient seed data is learned as the student data corresponding to the image data, the first image data is converted into the second image data with improved spatial resolution as the coefficient seed data. What can perform image conversion processing as spatial resolution creation processing can be obtained.

  In this case, the image conversion apparatus 51 in FIG. 7 can improve the horizontal resolution and vertical resolution of the image data to a resolution corresponding to the parameter z.

  Further, for example, the high-quality image data is used as the teacher data, and the image data obtained by superimposing the noise corresponding to the parameter z on the high-quality image data as the teacher data is used as the student data. By performing learning, the coefficient seed data is one that performs image conversion processing as noise removal processing that converts the first image data into second image data from which the noise contained therein has been removed (reduced). Obtainable. In this case, the image conversion apparatus 51 in FIG. 7 can obtain S / N image data corresponding to the parameter z.

  Further, for example, certain image data is used as teacher data, and image data obtained by thinning out the number of pixels of the image data as the teacher data in accordance with the parameter z is used as student data, or image data of a predetermined size is used. The coefficient seed data is obtained by performing the learning of the coefficient seed data by using the image data obtained by thinning out the pixels of the image data as the student data at the thinning rate corresponding to the parameter z as the teacher data. It is possible to obtain one that performs image conversion processing as resizing processing for converting one image data into second image data whose size is enlarged or reduced. In this case, the image conversion apparatus 51 in FIG. 7 can obtain image data enlarged or reduced to a size corresponding to the parameter z.

Incidentally, in the above case, defines the tap coefficient w _n, in as shown in Equation _{(9), β 1, n} z 0 + β 2, n z 1 + ··· + β M, n z M-1 and, by the equation (9), the spatial resolution in the horizontal and vertical directions, both have been to obtain the tap coefficient w _n for improving corresponding to the parameter z, the tap coefficient w _n is horizontal It is also possible to obtain a resolution and a vertical resolution that are independently improved corresponding to the independent parameters z _x and z _y .

That is, the tap coefficients w _n, instead of the equation (9), for example, cubic polynomial ^{_{β 1, n z x 0 z}} y 0 + β 2, n z x 1 z y 0 + β 3, n z x 2 z y ⁰ + β _{4, n} z _x ³ z _y ⁰ + β _{5, n} z _x ⁰ z _y ¹ + β _{6, n} z _x ⁰ z _y ² + β _{7, n} z _x ⁰ z _y ³ + β _{8, n} z _x ¹ z _y with defined ^{_{1 + β 9, n z x}} 2 z y 1 + β 10, n z x 1 z y 2, the variable t _m defined in formula (10), in place of the equation (10), for example, t ₁ = Z _x ⁰ z _y ⁰ , t ₂ = z _x ¹ z _y ⁰ , t ₃ = z _x ² z _y ⁰ , t ₄ = z _x ³ z _y ⁰ , t ₅ = z _x ⁰ z _y ¹ , t ₆ = Z _x ⁰ z _y ² , t ₇ = z _x ⁰ z _y ³ , t ₈ = z _x ¹ z _y ¹ , t ₉ = z _x ² z _y ¹ , t ₁₀ = z _x ¹ z _y ² . Again, the tap coefficient w _n is finally can be represented by the formula (11), therefore, and the learning device 71 in FIG. 9, in the learning apparatus 101 of FIG. 12, a parameter z _x and z _y Correspondingly, the horizontal resolution and the vertical resolution are obtained by performing learning using the image data obtained by degrading the horizontal resolution and the vertical resolution of the teacher data as student data, and obtaining the coefficient seed data β _{m, n} . in response to independent parameters z _x and z _y, it can be determined tap coefficients w _n to improve independently.

In addition, for example, in addition to the parameters z _x and z _y corresponding to the horizontal resolution and the vertical resolution, respectively, by introducing a parameter z _t corresponding to the resolution in the time direction, the horizontal resolution, the vertical resolution, and the time resolution are independent parameters z _x, z _y, corresponding to the z _t, it is possible to determine the tap coefficient w _n to improve independently.

As for the resizing process, as in the spatial resolution creation processing, the horizontal and vertical directions, other tap coefficients w _n to resize magnification of both corresponding to the parameter z (or reduction ratio), the horizontal and vertical It is possible to obtain a tap coefficient w _n whose size is independently resized with an enlargement factor corresponding to the parameters z _x and z _y , respectively.

Furthermore, and the learning device 71 in FIG. 9, in the learning apparatus 101 of FIG. 12, with deteriorating the horizontal resolution and vertical resolution of the teacher data corresponding to the parameter z _x, the noise to the teacher data corresponding to the parameter z _y the additional image data by performing learning by using as the student data, by obtaining the coefficient seed data beta _{m, n,} improves the horizontal resolution and vertical resolution corresponding to the parameter z _x, corresponding to the parameter z _y it can be obtained tap coefficients w _n to perform noise removal and.

  An embodiment of an image processing apparatus that includes a processing unit that performs the class classification adaptive processing as described above and to which the present invention is applied will be described below. In other words, the image processing apparatus (image processing system) described below is different from the image conversion apparatus described above in the image processing units (the image processing units 213-1 to 213-3 in FIG. 14 and the image processing unit 475 in FIG. 18). 1 to 475-3).

  FIG. 14 shows a configuration example of the first embodiment of an image processing apparatus to which the present invention is applied.

  14 includes an image input unit 211, an image distribution unit 212, image processing units 213-1 to 213-3, an image composition unit 214, an image presentation unit 215, a storage unit 216, and a control unit 217. Has been.

  The image processing apparatus 200 simultaneously displays the input image and the three types of processed images processed by the image processing units 213-1 to 213-3, and performs optimal image processing from the three types of processed images. Two display modes: image comparison mode for selecting (determining) settings and full-screen mode for performing input image processing with the image processing settings determined in the image comparison mode and displaying the processed image in full screen Has a mode.

  A moving image is input to the image processing apparatus 200. A plurality of images constituting the moving image are acquired by the image input unit 211 and sequentially supplied to the image distribution unit 212 as input images.

  The image distribution unit 212 supplies the input image supplied from the image input unit 211 to the image processing units 213-1 to 213-3 and the image composition unit 214.

  Each of the image processing units 213-1 to 213-3 performs predetermined image processing on the input image supplied from the image distribution unit 212 simultaneously (in parallel) by performing the above-described class classification adaptive processing, A processed image, which is a processed image, is supplied to the image composition unit 214. Here, the predetermined image processing performed by the image processing units 213-1 to 213-3 is, for example, an image quality improvement process for improving image quality by improving spatial resolution or removing noise. is there.

  More specifically, when the display mode is the image comparison mode, each of the image processing units 213-1 to 213-3 is adapted to class classification according to the volume value set by the user operating the remote commander or the like. Execute the process. Here, the volume value corresponds to, for example, the parameter z of the class classification process described above, and can be set in the range of 0 to 100. Different volume values are set for the image processing units 213-1 to 213-3, and the image processing units 213-1 to 213-3 perform processing for converting to different image quality. In the image processing units 213-1 to 213-3, the pixel ranges set as class taps or prediction taps (hereinafter referred to as tap ranges) are the same.

  On the other hand, when the display mode is the full screen mode, each of the image processing units 213-1 to 213-3 performs classification according to the volume value selected by the user from the result of comparing a plurality of processed images in the image comparison mode. Perform adaptive processing. At this time, the tap ranges set by the image processing units 213-1 to 213-3 are different for the image processing units 213-1 to 213-3.

  Whether the current display mode is the image comparison mode or the full screen mode is supplied from the control unit 217 to the image processing units 213-1 to 213-3 as mode information. Further, a volume value set by the image processing units 213-1 to 213-3 in the full screen mode is also supplied from the control unit 217.

  The image composition unit 214 is supplied with the input image from the image distribution unit 212 and the processed image from each of the image processing units 213-1 to 213-3.

  The image combining unit 214 combines different images depending on whether the current display mode is the image comparison mode or the full screen mode, and supplies the resultant combined image to the image presenting unit 215. More specifically, when the current display mode is the image comparison mode, the image composition unit 214 displays a composite image in which an input image and three types of processed images are arranged in each area obtained by dividing the screen into four. It is generated and supplied to the image presentation unit 215. On the other hand, when the current display mode is the full screen mode, the image composition unit 214 adds the three processed images supplied from the image processing units 213-1 to 213-3, and an image obtained as a result Are supplied to the image presentation unit 215 as a composite image.

  Whether the current display mode is the image comparison mode or the full screen mode is supplied from the control unit 217 as mode information. The image composition unit 214 is also supplied with screen control information for instructing display of a setting screen for setting a volume value set in each of the image processing units 213-1 to 213-3 in the image comparison mode. The

  The image presentation unit 215 presents the composite image to the user by displaying the composite image supplied from the image composition unit 214 on a predetermined display unit.

  The storage unit 216 stores predetermined information supplied from the control unit 217. For example, the storage unit 216 stores a volume value set in the image processing units 213-1 to 213-3 in the full screen mode, which is determined by a user comparing a plurality of processed images in the image comparison mode.

  The control unit 217 acquires information for setting (changing) the display mode operated by the user using a remote commander (not shown) and supplies the information to the image processing units 213-1 to 213-3 as mode information. In addition, when the user sets the volume value set in each of the image processing units 213-1 to 213-3, the control unit 217 sends screen control information for instructing display of the setting screen to the image composition unit 214. Supply.

  FIG. 14 shows an example in which the image processing device 200 includes three image processing units 213-1 to 213-3. However, the image processing device 200 may include two image processing units 213 or four. It may be more than one.

  FIG. 15 shows screen examples of the image comparison mode and the full screen mode.

  In the image comparison mode, the image composition unit 214 includes the input image 241 supplied from the image distribution unit 212 and the three types of post-processing images 242-1 to 242 processed by the image processing units 213-1 to 213-3. 3 is generated in each area obtained by dividing the screen into four.

  In the example illustrated in FIG. 15, the post-processing image 242-1 processed by the image processing unit 213-1 is an image processed with the volume value set to 20 (vol = 20), and the image processing unit 213 -Processed image 242-2 processed by -2 is an image processed with the volume value set to 40 (vol = 40). Further, the post-processing image 242-3 processed by the image processing unit 213-3 is an image processed with the volume value set to 60 (vol = 60).

  Then, it is assumed that the user has selected 20 as the optimal volume value as a result of comparing the processed images 242-1 to 242-2. In this case, the determined volume value (20) is supplied to and stored in the storage unit 216 as a volume value in the full screen mode.

  In the full screen mode, the image composition unit 214 adds the three processed images supplied from the image processing units 213-1 to 213-3 whose volume values are set to 20, and obtains the resultant composition The image 251 is supplied to the image presentation unit 215.

  FIG. 16 shows an example of the tap range (tap structure) in the image comparison mode and the full screen mode.

  In the image comparison mode, the tap range is the same for each of the image processing units 213-1 to 213-3. For example, in any of the image processing units 213-1 to 213-3, for the pixel corresponding to the target pixel, the pixel, two pixels rightward from the pixel, two pixels leftward, and one upward A total of seven pixels, one pixel in the downward direction and one pixel, is set as the tap range.

  The volume values in the image comparison mode are, for example, 20 (vol = 20) for the image processing unit 231-1, 40 (vol = 40) for the image processing unit 231-2, and 60 (vol) for the image processing unit 213-3. = 60), the image processing units 213-1 to 213-3 are different.

  On the other hand, in the full screen mode, the tap range is different for each of the image processing units 213-1 to 213-3, and the range obtained by combining the tap ranges of the image processing units 213-1 to 213-3 is the tap in the image comparison mode. The range is set to be larger than the range. For example, in the image processing unit 231-1, a total of 7 pixels of 3 pixels on the 2nd row and 4 pixels on the 1st row corresponding to the target pixel are set as the tap range. A total of 7 pixels including 5 pixels in the same row centered on the pixel corresponding to the pixel and 2 pixels arranged above and below the rightmost pixel among the 5 pixels are set as the tap range. In the image processing unit 231-3, a total of 7 pixels, that is, 3 pixels 2 rows below and 4 pixels 1 row below the pixels corresponding to the target pixel are set as the tap range. As described above, when the tap range is set in each of the image processing units 213-1 to 213-3, the synthesized image 251 generated by the image synthesizing unit 214 corresponds to the target pixel as illustrated in FIG. For a pixel, an image equivalent to setting the surrounding 21 pixels including the pixel as a tap range can be obtained. This is because when the class classification adaptive process focuses on individual pixels, it can be expressed by a product-sum operation of a coefficient group and a prediction tap.

  Further, the volume value in the full screen mode is the same as 20 (vol = 20) which is supplied to and stored in the storage unit 216 in any of the image processing units 213-1 to 213-3.

  Next, the first image processing by the image processing apparatus 200 of FIG. 14 will be described with reference to the flowchart of FIG.

  First, in step S <b> 81, the image distribution unit 212 determines whether an input image is supplied from the image input unit 211. If it is determined in step S81 that the input image is not supplied, the process ends.

  On the other hand, if it is determined in step S81 that an input image has been supplied from the image input unit 211, the process proceeds to step S82, and the image distribution unit 212 distributes the input image supplied from the image input unit 211. That is, the image distribution unit 212 supplies the input image to the image processing units 213-1 to 213-3 and the image composition unit 214.

  In step S83, the control unit 217 determines whether the current display mode is the image comparison mode. If it is determined in step S83 that the current display mode is the image comparison mode, the process proceeds to step S84, and the control unit 217 displays mode information indicating that the image comparison mode is in the image processing units 213-1 to 213. -3.

  In step S <b> 85, the image processing units 213-1 to 213-3 perform the image quality improvement process for improving the image quality of the input image by performing the class classification adaptation process. As described with reference to FIG. 16, the volume value in the class classification adaptive processing here is different for each of the image processing units 213-1 to 213-3. The tap range is the same in the image processing units 213-1 to 213-3.

  In step S <b> 86, the image composition unit 214 generates a composite image in which the input image and the three types of processed images are arranged in each region obtained by dividing the screen into four parts, and supplies the composite image to the image presentation unit 215.

  In step S87, the control unit 217 determines whether any of the processed images of the image processing units 213-1 to 213-3 is selected by the user operating the remote commander or the like.

  If it is determined in step S87 that any of the processed images of the image processing units 213-1 to 213-3 has been selected, the process proceeds to step S88, and the control unit 217 corresponds to the selected processed image. The volume value to be supplied is supplied to the storage unit 216 and stored. On the other hand, if it is determined in step S87 that none of the processed images of the image processing units 213-1 to 213-3 has been selected, the process skips step S88 and proceeds to step S92.

  If it is determined in step S83 described above that the current display mode is not the image comparison mode, that is, if the current display mode is the full screen mode, the process proceeds to step S89, and the control unit 217 Mode information indicating the full screen mode is supplied to the image processing units 213-1 to 213-3. In addition, the control unit 217 reads the volume value stored in the storage unit 216 and supplies the volume value to the image processing units 213-1 to 213-3.

  In step S <b> 90, the image processing units 213-1 to 213-3 perform the image quality improvement processing for improving the image quality of the input image by performing the class classification adaptive processing. The volume value in the class classification adaptive processing here is the same in the image processing units 213-1 to 213-3 as described with reference to FIG. On the other hand, the tap range is different for each of the image processing units 213-1 to 213-3.

  In step S91, the image composition unit 214 generates one composite image from the three types of post-processing images by adding the three types of post-processing images supplied from the image processing units 213-1 to 213-3. And supplied to the image presentation unit 215.

  In step S92, the image presentation unit 215 presents the composite image to the user by displaying the composite image supplied from the image composition unit 214 on a predetermined display unit. After the process of step S92, the process returns to step S81, and the subsequent processes are repeatedly executed.

  As described above, according to the first image processing, in the image comparison mode, a plurality of image processing with different parameters (volume values) are performed on the input image that has been input, and the image processing result (after processing) Image) are simultaneously displayed on one screen, the user can select an optimum parameter for the input image from among a plurality of parameters.

  In the full screen mode, the input image can be subjected to image processing with the optimum parameters selected by the user, but the processed image after the image processing is the size of the entire screen (large screen). ), It is possible to perform processing with higher image quality with a wider tap range than in the image comparison mode. The large screen composite image generated in the full screen mode is generated by the image processing units 213-1 to 213-3 sharing the tap range. If one of them is simply performed in a tap range of 21 pixels, the three image processing units 213-1 to 213-3 can be used effectively, and the processing can be performed at high speed.

  FIG. 18 shows a configuration example of the second embodiment of the image processing apparatus to which the present invention is applied.

  Note that portions corresponding to those of the image processing apparatus 200 in FIG. 14 are denoted by the same reference numerals, and description thereof is omitted as appropriate.

  18 includes an image input unit 211, an image distribution unit 212, an image presentation unit 215, a synchronization feature amount extraction unit 471, sequence playback units 472A to 472C, a switcher unit 473, and image processing units 475-1 to 475. -3, an image composition unit 476, a storage unit 477, and a control unit 478.

  The image processing apparatus 200 in FIG. 18 also displays the input image and the three types of processed images processed by the image processing units 475-1 to 475-3 at the same time as in the case of FIG. An image comparison mode for selecting (determining) the optimum image processing setting from the processed images, and a full screen mode for displaying an image subjected to image processing according to the image processing setting determined in the image comparison mode on a full screen. There are two display modes.

  As shown in FIG. 19, the synchronization feature amount extraction unit 471 associates a time code indicating which scene in the moving image each input image is with a synchronization feature amount of the input image. A quantity timetable 461 is stored. Here, the synchronization feature amount is a feature amount of an image used to synchronize the input image, and represents an average value (average luminance value) of the luminance values of the input image represented by 16 bits, and the input image. Is a total of 32 bits of data consisting of the lower 16 bits of the total luminance value (total luminance value) of all pixels. In a reproducible input image such as a movie of about 2 hours, the scene of the input image can be almost certainly determined by using the above-described two feature amounts.

  First, the synchronization feature amount extraction unit 471 calculates the synchronization feature amount of the input image supplied from the image distribution unit 212. FIG. 19 shows an example in which the calculated synchronization feature amount is the average luminance value “24564” and the lower 16 bits “32155” of the total luminance value.

  Then, the synchronization feature amount extraction unit 471 detects a time code having the same synchronization feature amount from the synchronization feature amount time table 461 and supplies the time code to the sequence reproduction units 472A to 472C. In the time table 461 in FIG. 19, the synchronization feature amount corresponding to the time code “2” matches the calculated synchronization feature amount. Therefore, the synchronization feature amount extraction unit 471 supplies the time code “2” to the sequence reproduction units 472A to 472C. The detected time code is also supplied to the recording / reproducing unit 474.

  Of course, other image feature amounts may be employed as the synchronization feature amount.

  Returning to FIG. 18, each of the sequence playback units 472A to 472C stores a parameter table in which a time code and a zoom parameter are associated with each other. The sequence reproduction units 472A to 472C supply the zoom parameter corresponding to the time code supplied from the synchronization feature amount extraction unit 471 to the switcher unit 473. With this zoom parameter, the processing area when the image processing units 475-1 to 475-3 perform the enlargement process is specified. Therefore, it can be said that the sequence reproduction units 472A to 472C output region specifying information for specifying a processing region when the image processing units 475-1 to 475-3 perform enlargement processing.

  The zoom parameters stored as sequence parameter sections 472A to 472C are different from each other. Accordingly, the same time code is supplied from the synchronization feature amount extraction unit 471 to the sequence reproduction units 472A to 472C, but different zoom parameters are supplied to the switcher unit 473 from each of the sequence reproduction units 472A to 472C.

More specifically, the sequence playback unit 472A supplies zoom parameters (x _a , y _a , z _a ) to the switcher unit 473 corresponding to the time code supplied from the synchronization feature amount extraction unit 471. The sequence playback unit 472B supplies the zoom parameters (x _b , y _b , z _b ) to the switcher unit 473 corresponding to the time code supplied from the synchronization feature amount extraction unit 471. The sequence playback unit 472C supplies zoom parameters (x _c , y _c , z _c ) to the switcher unit 473 corresponding to the time code supplied from the synchronization feature amount extraction unit 471.

The zoom parameters (x _a , y _a , z _a ) represent the center position (x _a , y _a ) and the zoom magnification z _a when performing the enlargement process. The same applies to the zoom parameters (x _b , y _b , z _b ) and (x _c , y _c , z _c ). In the following, in order to simplify the explanation, the zoom parameters (x _a , y _a , z), (x _b , y _b , z _b ), and (x _c , y _c , z _c ) are respectively referred to as zoom parameters. Also referred to as A, B, and C.

  In the image comparison mode, the switcher unit 473 supplies the zoom parameters A to C supplied from the sequence reproduction units 472A to 472C to the image processing units 475-1 to 475-3 on a one-to-one basis. More specifically, the switcher unit 473 supplies the zoom parameter A to the image processing unit 475-1, supplies the zoom parameter B to the image processing unit 475-2, and supplies the zoom parameter B to the image processing unit 475-3. .

  In the full screen mode, the switcher unit 473 converts any one of the zoom parameters A to C supplied from the sequence playback units 472A to 472C into the image processing unit according to the selection information supplied from the recording / playback unit 474. Supply to all 475-1 through 475-3.

  The selection information is information representing the zoom parameter selected by the user in the image comparison mode. Therefore, the selection information is specifically one of the zoom parameters A to C.

  In the image comparison mode, the recording / playback unit 474 is supplied with the time code of the input image from the synchronization feature amount extraction unit 471, and from the control unit 478, the enlarged image A selected by the user by operating the remote commander or the like. Information (selection information) about zoom parameters A to C corresponding to C is supplied. The recording / playback unit 474 stores the time code supplied from the synchronization feature amount extraction unit 471 and the selection information in association with each other. Data in which a set of the time code and selection information is stored in time series is a recording sequence. Accordingly, in the full screen mode, the switcher unit 473 selects one of the three types of zoom parameters A to C according to the selection information from the recording / reproducing unit 474 and supplies the selected one to the image processing units 475-1 to 475-3. By repeating this in time series, the image presentation unit 215 can reproduce the enlarged image selected by the user in the image comparison mode.

  On the other hand, in the full screen mode, the time code of the input image is supplied from the synchronization feature amount extraction unit 471, and the recording / playback unit 474 supports the supplied time code based on the recording sequence stored therein. The zoom parameter to be supplied is supplied to the switcher unit 473.

  The image processing units 475-1 to 475-3 perform the input image enlargement process (resizing process) based on the zoom parameters supplied from the switcher unit 473 by performing the above-described class classification adaptation process. In the full screen mode, the image processing units 475-1 to 475-3 perform time resolution creation processing for creating time resolution in addition to the enlargement processing.

  In the image comparison mode, different zoom parameters A to C are supplied to the image processing units 475-1 to 475-3 on a one-to-one basis from the sequence reproduction units 472A to 472C as described above. Accordingly, the image processing units 475-1 to 475-3 perform enlargement processing on different regions on the input image, and supplies the enlarged regions to the image composition unit 476.

  On the other hand, in the full screen mode, any one of the parameters A to C is supplied to all of the image processing units 475-1 to 475-3. Accordingly, the areas to be enlarged by the image processing units 475-1 to 475-3 are the same area. However, the image processing units 475-1 to 475-3 generate enlarged images at different positions (time) in the time axis direction from the same input image. Thereby, it is possible to present an enlarged image obtained by enlarging a predetermined area and creating (improving) time resolution. Detailed processing of the image processing units 475-1 to 475-3 will be described later with reference to FIGS.

  In the image comparison mode, the image composition unit 476 generates a composite image obtained by combining the enlarged images A to C supplied from the image processing units 475-1 to 475-3 and the input image supplied from the image distribution unit 212. And supplied to the image presentation unit 215. On the other hand, in the full screen mode, the image composition unit 476 supplies the enlarged images generated at the different times generated by the image processing units 475-1 to 475-3 to the image presentation unit 215 in order of time. Therefore, in the full screen mode, an enlarged image is displayed at a frame rate three times that of the input image.

  The storage unit 477 stores coefficient seed data for generating tap coefficients used for the class classification adaptation process. Specifically, the storage unit 477 is an enlargement coefficient that is coefficient seed data when performing enlargement processing in the image comparison mode, and time that is coefficient seed data when performing enlargement processing and time resolution creation processing in the full screen mode. Store the coefficients. Various coefficients stored in the storage unit 477 are supplied to the image processing units 475-1 to 475-3 as necessary.

  The control unit 478 acquires operation information when the user operates a remote commander or the like, and based on the user's operation, the recording / playback unit 474, the image processing units 475-1 to 475-3, and the image composition unit 476 are displayed. Control appropriately. For example, the control unit 478 displays selection information indicating which enlarged image among the enlarged images A to C presented by the user in the image presentation unit 215 in the image comparison mode, the recording / reproducing unit 474 and the image composition. Part 476. In addition, when the display mode is switched by the user, the control unit 478 supplies the display mode after switching to the image processing units 475-1 to 475-3 as mode information.

  When the display mode is the image comparison mode, the image processing apparatus 200 configured as described above has an input image and three types of enlarged images A to A processed by the image processing units 475-1 to 475-3. When C is displayed at the same time and the user selects (determines) the optimum enlarged image from the three types of enlarged images A to C, and the display mode is the full screen mode, the image comparison mode is selected. In addition to reproducing the enlarged image, it creates a temporal resolution and displays it at a frame rate three times that of the input image.

  FIG. 20 shows an example of the display screen 400 displayed on the image presentation unit 215 in the image comparison mode.

  In the display screen 400, the area is equally divided into four, the upper left of which is a main screen area 281 and the other three areas are sub-screen areas 282-1 to 282-3.

  The image composition unit 476 arranges the input image supplied from the image distribution unit 212 in the main screen area 281 and supplies the input image from the image processing units 475-1 to 475-3 to the sub screen areas 282-1 to 282-3. A composite image in which the enlarged images A to C are respectively arranged is generated. In addition, the image composition unit 476 causes the highlight display 291 to be displayed in any of the sub-screen areas 282-1 to 282-3 according to the selection information supplied from the control unit 478. In other words, the image composition unit 476 generates an image in which the highlight display 291 is superimposed on a composite image obtained by compositing the input image and the enlarged images A to C, and supplies the generated image to the image presentation unit 215.

  Next, with reference to FIGS. 21 and 22, the processing of the image processing apparatus 200 in FIG. 18 when the display mode is the image comparison mode will be described.

  Then, with reference to FIGS. 23 and 24, the processing of the image processing apparatus 200 in FIG. 18 when the display mode is the full screen mode will be described.

  FIG. 21 is a functional block diagram of the image processing apparatus 200 in FIG. 18 when the display mode is the image comparison mode.

  When the display mode is the image comparison mode, the image processing units 475-1 to 475-3 perform the process of enlarging different areas on the input image as described above. The units 475-1 to 475-3 are enlarged processing units 475A to 475C.

In the image comparison mode, as shown in FIG. 21, the switcher unit 473 supplies the zoom parameters (x _a , y _a , z _a ) supplied from the sequence playback unit 472A to the enlargement processing unit 475A. Further, the switcher unit 473 supplies the zoom parameters (x _b , y _b , z _b ) supplied from the sequence playback unit 472B to the enlargement processing unit 475B, and the zoom parameters (x _c , y _c , z _c ) is supplied to the enlargement processing unit 475C.

First, the enlargement processing unit 475A acquires an enlargement coefficient corresponding to the zoom magnification z _a from the storage unit 477. The relationship between the zoom magnification z _a and the enlargement coefficient is the same as, for example, the relationship between the parameter z and the coefficient seed data in the image conversion apparatus 51 in FIG. 7 described above. That is, in the image conversion apparatus 51 of FIG. 7, the tap coefficient for each class can be generated based on the parameter z and the coefficient seed data, but the zoom magnification z _a corresponds to the parameter z of the image conversion apparatus 51, The enlargement coefficient corresponds to the coefficient seed data of the image conversion device 51. By executing the classification adaptation process using the zoom factor z _a and the tap coefficient generated using the enlargement coefficient acquired from the storage unit 477, the position (x _a , y _a ) of the input image is set as the center position. An enlargement process with a predetermined zoom magnification is performed. The same applies to the relationship between the zoom magnification z _b or z _c and the magnification factor.

Accordingly, the enlargement processing unit 475A performs the enlargement process on the area specified by the zoom magnification z _a with the position (x _a , y _a ) of the input image as the center position, and the enlarged image A obtained as a result is obtained. This is supplied to the image composition unit 476.

Similarly, the enlargement processing unit 475B acquires an enlargement coefficient corresponding to the zoom magnification z _b from the storage unit 477, and is specified by the zoom magnification z _b with the position (x _b , y _b ) of the input image as the center position. Enlargement processing is executed on the area, and an enlarged image B obtained as a result is supplied to the image composition unit 476. The enlargement processing unit 475C obtains an enlargement coefficient corresponding to the zoom magnification z _c from the storage unit 477, and uses the position (x _c , y _c ) of the input image as the center position for the region specified by the zoom magnification z _c. The enlargement process is executed, and the enlarged image C obtained as a result is supplied to the image composition unit 476.

  FIG. 22 illustrates an example of enlargement processing by the enlargement processing units 475A to 475C when an input image in which squares (diamonds) and circles are captured as objects is supplied.

The enlargement processing unit 475A is supplied with zoom parameters (x _a , y _a , z _a ) that substantially cover the range surrounding the square, and the enlargement processing unit 475A generates the enlarged image A correspondingly. .

The enlargement processing unit 475B is supplied with zoom parameters (x _b , y _b , z _b ) that substantially cover the range surrounding both the square and the circle, and in response thereto, the enlargement processing unit 475B B is generated.

The enlargement processing unit 475C is supplied with zoom parameters (x _c , y _c , z _c ) that make the range surrounding the circle almost the entire range. Correspondingly, the enlargement processing unit 475C generates the enlarged image C. .

  The enlarged images A to C are supplied to the image composition unit 476 together with the input image. The image composition unit 476 enlarges the input image to the main screen region 281 and the sub screen region 282-1 as shown in FIG. Image A generates a composite image in which enlarged image B is arranged in sub-screen area 282-2 and enlarged image C is arranged in sub-screen area 282-3.

  FIG. 23 is a functional block diagram of the image processing apparatus 200 in FIG. 18 when the display mode is the full screen mode.

  When the display mode is the full screen mode, the image processing units 475-1 to 475-3 perform the time resolution creation process for creating the time resolution in addition to the enlargement process as described above. The image processing units 475-1 to 475-3 are time processing units 475A ′ to 475C ′.

  In the example of FIG. 23, it is assumed that selection information indicating that the user has selected the enlarged image B is supplied from the recording / playback unit 474 to the switcher unit 473.

  Since the switcher unit 473 is supplied with the selection information representing the zoom parameter B from the recording / reproducing unit 474, the switcher unit 473 performs the zoom parameter among the zoom parameters A to C supplied from the sequence reproducing units 472A to 472C. B is selected and supplied to all of the time processing units 475A ′ to 475C ′.

Time processing unit 475A 'to 475C' obtains a time coefficient corresponding to the zoom magnification z _b from the storage unit 477, performs the class classification adaptive processing using tap coefficients generated from the zoom magnification z _b and time factor Thus, it is an enlarged image obtained by enlarging a predetermined area centered on the position (x _b , y _b ) of the input image, and the time from when one input image is supplied until the next input image is supplied Is one frame time T, the magnified images generated by the time processing units 475A ′ to 475C ′ respectively generate magnified images B _{1 to} 475B ₃ that are shifted by 1 / 3T time.

More specifically, as shown in FIG. 24, at time (time code) t, the same input image in which squares and circles are captured as objects as in FIG. 22 is processed by time processing units 475A ′ to 475C. Suppose that the time processing unit 475A 'is centered on the time t at the same time of the input image and the position (x _b , y _b ), which is the same enlarged image as the enlarged image generated in the image comparison mode. An enlarged image B ₁ enlarged to is generated.

The time processing unit 475B ′ generates an enlarged image B ₂ that is enlarged around the position (x _b , y _b ) at time t + T / 3, which is T / 3 hours after the time t of the input image. Time processing unit 475C 'is the time t + 2T / 3 of 2T / 3 hours after the time t of the input image, the position (x _b, y _b) to generate an enlarged image B ₃ obtained by enlarging mainly.

The image composition unit 476 supplies the enlarged images B _{1 to} 475B ₃ supplied from the time processing units 475A ′ to 475C ′ to the image presentation unit 215 in that order. As a result, the image presentation unit 215 displays an enlarged image at a frame rate three times that of the input image.

  FIG. 25 conceptually shows the difference between the image comparison mode and the full-screen mode in a predetermined continuous time.

  In the image comparison mode, for example, from time t (1) to t (20), the user moves the highlight display 291 to the enlarged image C, selects the enlarged image C, and starts from time t (20). It is assumed that the highlight display 291 is moved to the enlarged image B and the enlarged image B is selected until t (35). Similarly, it is assumed that the user selects the enlarged image C from time t (35) to t (45), and selects the enlarged image A from time t (45) to t (60).

  In this case, the time (time code) and the enlarged image (zoom parameter) selected at that time are recorded as a recording sequence. That is, zoom parameter C from time t (1) to t (20), zoom parameter B from time t (20) to t (35), zoom parameter C from time t (35) to t (45), From time t (45) to t (60), a recording sequence of zoom parameter A is stored in the recording / reproducing unit 474.

  In the full screen mode, the enlarged image is displayed on the full screen of the display screen 400 in accordance with the recording sequence stored in the recording / playback unit 474. However, the frame rate of the moving image presented to the image presentation unit 215 in the full screen mode is three times the frame rate of the input moving image.

  Next, the second image processing by the image processing apparatus 200 in FIG. 18 will be described with reference to the flowcharts in FIGS. 26 and 27. FIG. 26 is a flowchart of processing (image comparison display processing) when the display mode is the image comparison mode in the second image processing, and FIG. 27 is a display mode in the second image processing. 7 is a flowchart of processing (selected image display processing) in a case where the is a full screen mode.

  Before the processing shown in FIGS. 26 and 27, mode information indicating that the display mode is the image comparison mode or the full screen mode is supplied from the control unit 478 to the image processing units 475-1 to 475-3. Has been.

  In the image comparison display process of FIG. 26, first, in step S101, the image distribution unit 212 determines whether an input image is supplied from the image input unit 211. If it is determined in step S101 that no input image is supplied, the process ends.

  On the other hand, if it is determined in step S101 that an input image has been supplied from the image input unit 211, the process proceeds to step S102, and the image distribution unit 212 distributes the input image supplied from the image input unit 211. That is, the image distribution unit 212 supplies the input image to the enlargement processing units 475A to 475C and the image composition unit 214.

  In step S103, the synchronization feature amount extraction unit 471 calculates the synchronization feature amount of the input image supplied from the image distribution unit 212, and in step S104, refers to the time table 461 to obtain the calculated synchronization feature amount. Detect the corresponding time code. The detected time code is supplied to the sequence playback units 472A to 472C and the recording / playback unit 474.

  In step S <b> 105, the recording / playback unit 474 associates data in which the time code supplied from the synchronization feature amount extraction unit 471 and the selection information representing any one of the zoom parameters A to C supplied from the control unit 478 are associated with each other. Store as a recording sequence.

  In step S106, the sequence reproduction units 472A to 472C refer to the parameter table in which the time code and the zoom parameter are associated with each other, and supply the zoom parameter corresponding to the supplied time code to the switcher unit 473. That is, the sequence playback unit 472A supplies the zoom parameter A to the switcher unit 473, and the sequence playback unit 472B supplies the zoom parameter B to the switcher unit 473. The sequence playback unit 472C supplies the zoom parameter C to the switcher unit 473.

  In step S107, the switcher unit 473 supplies the zoom parameters A to C to the enlargement processing units 475A to 475C on a one-to-one basis. That is, the switcher unit 473 supplies the zoom parameter A to the enlargement processing unit 475A, supplies the zoom parameter B to the enlargement processing unit 475B, and supplies the zoom parameter C to the enlargement processing unit 475C.

  In step S <b> 108, the enlargement processing units 475 </ b> A to 475 </ b> C respectively execute enlargement processing and supply the processed enlarged images A to C to the image composition unit 476.

  In step S109, the image composition unit 476 generates a composite image using the input image supplied from the image distribution unit 212 and the enlarged images A to C supplied from the enlargement processing units 475A to 475C, and the image presentation unit 215. To supply.

  In step S110, the image presentation unit 215 presents the composite image to the user by displaying the composite image supplied from the image composition unit 476 on a predetermined display unit. After the process of step S110, the process returns to step S101, and the subsequent processes are repeatedly executed.

  In the selected image display process of FIG. 27, first, in step S121, the image distribution unit 212 determines whether an input image is supplied from the image input unit 211. If it is determined in step S121 that the input image is not supplied, the process ends.

  On the other hand, if it is determined in step S121 that an input image has been supplied from the image input unit 211, the process proceeds to step S122, and the image distribution unit 212 distributes the input image supplied from the image input unit 211. That is, the image distribution unit 212 supplies the input image to the time processing units 475A ′ to 475C ′.

  In step S123, the synchronization feature amount extraction unit 471 calculates the synchronization feature amount of the input image supplied from the image distribution unit 212. In step S124, the synchronization feature amount extraction unit 471 calculates the synchronization feature amount by referring to the time table 461. Detect the corresponding time code. The detected time code is supplied to the sequence playback units 472A to 472C and the recording / playback unit 474.

  In step S <b> 125, the recording / playback unit 474 acquires selection information corresponding to the time code supplied from the synchronization feature amount extraction unit 471 from the recording sequence, and supplies the selection information to the switcher unit 473.

  In step S126, the sequence reproduction units 472A to 472C refer to the parameter table in which the time code and the zoom parameter are associated with each other, and supply the zoom parameter corresponding to the supplied time code to the switcher unit 473. That is, the sequence playback unit 472A supplies the zoom parameter A to the switcher unit 473, and the sequence playback unit 472B supplies the zoom parameter B to the switcher unit 473. The sequence playback unit 472C supplies the zoom parameter C to the switcher unit 473.

  In step S127, based on the selection information supplied from the recording / reproducing unit 474, the switcher unit 473 converts any one of the zoom parameters A to C supplied from the sequence reproducing units 472A to 472C to the time processing unit. Supply to all of 475A 'to 475C'.

In step S128, the time processing units 475A ′ to 475C ′ perform the class classification adaptive process using the tap coefficient generated from the zoom magnification (any one of z _a , z _b , and z _c ) and the time coefficient. The enlargement process and the time resolution creation process in which the time resolution is tripled are executed at the same time, and the enlarged image obtained as a result is supplied to the image composition unit 476. For example, as described with reference to FIG. 24, when the zoom parameter B as the selected information is supplied, expanded image B ₁ of the input image at the same time t, expanded image B ₂ at time t + T / 3, and The enlarged image B _{3 at} time t + 2T / 3 is supplied to the image composition unit 476.

  In step S129, the image composition unit 476 sequentially supplies the three enlarged images supplied from the image distribution unit 212 to the image presentation unit 215 in time series. After the process of step S129, the process returns to step S121, and the subsequent processes are repeatedly executed.

  In the full screen mode, the above processing is performed, so that an enlarged image selected by the user in the image comparison mode is displayed at a frame rate three times that of the input image.

  As described above, according to the image processing apparatus 200 of FIG. 18, in the image comparison mode, the input image supplied from the image distribution unit 212 and the enlarged images A to C that have been enlarged are simultaneously displayed on the image presentation unit 215. Therefore, the user can select a desired enlarged image from the enlarged images A to C while confirming the input image.

  In addition, since selection information indicating which enlarged images A to C are selected by the user is recorded in the recording / playback unit 474 as a recording sequence, in the full-screen mode, as selected by the user based on the recording sequence. An enlarged image of the composition can be reproduced (reproduced).

  In the full screen mode, any one of the image processing units 475-1 to 475-3 is required as long as the composition as selected by the user is reproduced and displayed based on the recording sequence recorded in the recording / reproducing unit 474. However, in the above example, the image processing units 475-1 to 475-3 are in the same area and are slightly shifted in time. Since the enlarged image is generated, it is possible to generate a high-quality enlarged image with improved time resolution, and furthermore, since all of the image processing units 475-1 to 475-3 are used, the image processing unit 475- 1 to 475-3 can be used effectively.

  In the example described above, an example in which user selection information is recorded and reproduced in time series for one moving image (input image) has been described. However, the present invention is applicable to a plurality of moving images (input images). It is also possible to do. For example, in the image comparison mode, two types of enlargement processing are executed on two types of moving images, a composite image obtained by combining the input image and four types of enlarged images is displayed, and the user can select four types of enlarged images. Information indicating which enlarged image is selected from among them can be recorded as selection information.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 28 is a block diagram illustrating an example of a hardware configuration of a computer that executes the above-described series of processes using a program.

  In a computer, a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503 are connected to each other by a bus 504.

  An input / output interface 505 is further connected to the bus 504. The input / output interface 505 includes an input unit 506 made up of a keyboard, mouse, microphone, etc., an output unit 507 made up of a display, a speaker, etc., a storage unit 508 made up of a hard disk or nonvolatile memory, and a communication unit 509 made up of a network interface. A drive 510 for driving a removable medium 511 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 501 loads the program stored in the storage unit 508 to the RAM 503 via the input / output interface 505 and the bus 504 and executes the program, for example. First or second image processing is performed.

  The program executed by the computer (CPU 501) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. The program is recorded on a removable medium 511 that is a package medium including a memory or the like, or provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the storage unit 508 via the input / output interface 505 by attaching the removable medium 511 to the drive 510. The program can be received by the communication unit 509 via a wired or wireless transmission medium and installed in the storage unit 508. In addition, the program can be installed in the ROM 502 or the storage unit 508 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  In addition, in the present specification, the steps described in the flowcharts are executed in parallel or individually even if they are not necessarily processed in time series, as well as processes performed in time series in the described order. It also includes processing.

  In the present specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

It is a block diagram which shows the structural example of the image conversion apparatus which performs the image conversion process by a class classification adaptive process. It is a flowchart explaining the image conversion process by an image converter. It is a block diagram which shows the structural example of the learning apparatus which learns a tap coefficient. It is a block diagram which shows the structural example of the learning part of a learning apparatus. It is a figure for demonstrating various image conversion processes. It is a flowchart explaining the learning process by a learning apparatus. It is a block diagram which shows the structural example of the image conversion apparatus which performs the image conversion process by a class classification adaptive process. It is a block diagram which shows the structural example of the coefficient output part of an image converter. It is a block diagram which shows the structural example of the learning apparatus which learns coefficient seed data. It is a block diagram which shows the structural example of the learning part of a learning apparatus. It is a flowchart explaining the learning process by a learning apparatus. It is a block diagram which shows the structural example of the learning apparatus which learns coefficient seed data. It is a block diagram which shows the structural example of the learning part of a learning apparatus. It is a block diagram which shows the structural example of 1st Embodiment of the image processing apparatus to which this invention is applied. It is a figure which shows the example of a screen of image comparison mode and full screen mode. It is a figure which shows the example of the tap range in image comparison mode and full screen mode. It is a flowchart explaining a 1st image process. It is a block diagram which shows the structural example of 2nd Embodiment of the image processing apparatus to which this invention is applied. It is a figure explaining the detection of a time code. It is a figure which shows the example of a screen displayed in image comparison mode. It is a functional block diagram of an image processing device when a display mode is an image comparison mode. It is a figure which shows the example of a process in case display mode is image comparison mode. It is a functional block diagram of an image processing device when a display mode is a full screen mode. It is a figure which shows the process example in case display mode is full screen mode. It is a figure which shows notionally the difference between image comparison mode and full screen mode. It is a flowchart explaining an image comparison display process. It is a flowchart explaining a selection image display process. It is a block diagram which shows the structural example of one Embodiment of the computer to which this invention is applied.

Explanation of symbols

  200 image processing device, 213-1 to 213-3 image processing unit, 214 image composition unit, 215 image presentation unit, 216 storage unit, 217 control unit, 471 synchronization feature amount extraction unit, 472A to 472C sequence reproduction unit, 473 switcher Unit, 474 recording / reproducing unit, 475-1 to 475-3 image processing unit, 476 image synthesizing unit, 477 storage unit, 478 control unit

Claims (7)

A plurality of image conversion processing means for respectively performing image conversion processing for converting one or more input images sequentially constituting a moving image as a first image into a second image;
One image conversion process is selected from a plurality of different image conversion processes by selecting one post-process image from among a plurality of post-process images that are images processed by each of the plurality of image conversion processing means. And a control means for switching the second mode for executing the image conversion process selected in the first mode,
Each of the plurality of image conversion processing means includes:
Class tap selection for selecting a pixel used for classifying a target pixel of interest in the second image into one of the plurality of classes as a class tap from the first image Means,
Class classification means for classifying the pixel of interest based on the class tap;
Prediction tap selection means for selecting, as a prediction tap, a pixel used for calculation with the tap coefficient when obtaining the target pixel from the first image;
Tap coefficient output means for outputting a tap coefficient of the class of the target pixel;
Computing means for obtaining the pixel of interest by performing computation using the tap coefficient of the class of the pixel of interest and the prediction tap selected for the pixel of interest; and the first mode, An image processing apparatus that switches at least one of the class tap, the prediction tap, and the tap coefficient in the second mode. The tap range, which is the range of pixels selected as the class tap or the prediction tap, is the same in each of the plurality of image conversion processing means in the first mode, and in the second mode, the plurality of images. The image processing apparatus according to claim 1, wherein each of the conversion processing units is different, and a range obtained by combining tap ranges of the plurality of image conversion processing units is a range that is larger than the tap range in the first mode. Each of the plurality of image conversion processing means expands a predetermined region of the first image in the first mode by switching at least one of the class tap, the prediction tap, and the tap coefficient. The image processing apparatus according to claim 1, wherein an enlargement process is performed, and in the second mode, a predetermined area of the first image is enlarged and a time resolution and an enlargement process are performed to improve the time resolution. Synchronous feature amount extraction means for extracting a feature amount of the input image and detecting a time code indicating which scene in the moving image the input image is,
A plurality of area specifying means for outputting area specifying information for specifying the predetermined area that each of the plurality of image conversion processing means expands corresponding to the time code detected by the synchronization feature amount extracting means; Prepared,
The image processing apparatus according to claim 3, wherein each of the plurality of image conversion processing units performs an enlargement process for enlarging the first image based on the region specifying information in the first mode. In the first mode, selection information that is information representing an enlarged image selected by the user from among the plurality of enlarged images obtained by enlarging the first image by the plurality of image conversion processing means, User selection information storage means for storing the code in association with the code;
In the second mode, based on the selection information stored in the user selection information storage means, one area specifying information is selected from the plurality of area specifying information supplied from the plurality of area specifying means. Selecting means for selecting and supplying to a plurality of said image conversion processing means,
Each of the plurality of image conversion processing means performs time resolution and enlargement processing for enlarging the first image and improving the time resolution based on the region specifying information in the second mode. Item 5. The image processing apparatus according to Item 4. In an image processing method of an image processing apparatus for performing an image conversion process for converting one or more input images sequentially constituting a moving image as a first image into a second image,
A pixel used for classifying a target pixel of interest in the second image into any one of the plurality of classes is selected from the first image as a class tap, and the class Based on the tap, classify the pixel of interest, select a pixel to be used for calculation with the tap coefficient when obtaining the pixel of interest from the first image as a prediction tap, and tap the class of the pixel of interest A coefficient, and by performing a calculation using the tap coefficient of the class of the target pixel and the prediction tap selected for the target pixel, a plurality of image conversion processes for obtaining the target pixel are performed simultaneously, By selecting one post-processing image from among multiple post-processing images, a single image conversion process can be selected from a plurality of different image conversion processes. Wherein for a first mode, a second step of switching the mode for executing the first mode in the selected image conversion process,
An image processing method for switching at least one of the class tap, the prediction tap, and the tap coefficient between the first mode and the second mode. In a program for causing a computer to execute an image conversion process for converting one or more input images sequentially constituting a moving image as a first image into a second image.
A pixel used for classifying a target pixel of interest in the second image into any one of the plurality of classes is selected from the first image as a class tap, and the class Based on the tap, classify the pixel of interest, select a pixel to be used for calculation with the tap coefficient when obtaining the pixel of interest from the first image as a prediction tap, and tap the class of the pixel of interest A coefficient, and by performing a calculation using the tap coefficient of the class of the target pixel and the prediction tap selected for the target pixel, a plurality of image conversion processes for obtaining the target pixel are performed simultaneously, By selecting one post-processing image from among multiple post-processing images, a single image conversion process can be selected from a plurality of different image conversion processes. Wherein for a first mode, a second step of switching the mode for executing the first mode in the selected image conversion process,
A program that causes a computer to execute an image conversion process for switching at least one of the class tap, the prediction tap, and the tap coefficient between the first mode and the second mode.

Images