JPH10336598A - Converter for image signal, conversion method, generator for coupling coefficient of neural network used for the conversion and generating method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method by which a pixel signal is predicted optimally at all times. SOLUTION: An NTSC system video signal is converted into a High Vision video signal. A ROM table 107 stores coupling coefficients of a neural network for each class generated by learning, in advance. A class code CL denoting a class (spatial class and motion class), to which HD pixel data (y) to be predicted belongs, is fed to the ROM table 107 as read address information. A neural network prediction circuit 109 uses a neural network which is based on a coupling coefficient Wi read from the ROM table 107 and SD pixel data xi segmented corresponding to the HD pixel data (y) to be predicted to obtain the HD pixel data (y). Since nonlinear (including linear) prediction is conducted, the pixel signal can be predicted optimally at all times, in comparison with a conventional linear prediction.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention
The present invention relates to an image signal conversion device and a conversion method for converting a video signal of a system into a high-definition video signal, and a generation device and a generation method of a coupling coefficient of a neural network used for the same. More specifically, the present invention relates to an image signal conversion device that can perform nonlinear prediction by using a neural network when predicting a pixel signal, and can always perform optimal pixel signal prediction. .

[0002]

2. Description of the Related Art In recent years, with the increase in audio-visual orientation, it has been desired to develop a television receiver capable of obtaining a higher-resolution image. In response to this demand, a so-called Hi-Vision has been developed. The number of scanning lines for Hi-Vision is 1125, which is more than double that of 525 for the NTSC system. In addition, the aspect ratio of Hi-Vision is 9:16, while the aspect ratio of NTSC is 3: 4. For this reason, high-definition images can be displayed with higher resolution and a sense of realism as compared with the NTSC system.

[0003] Hi-Vision has such excellent characteristics, but it is impossible to display an image by the Hi-Vision system even if the video signal of the NTSC system is supplied as it is. This is because the standards differ between the NTSC system and Hi-Vision as described above.

Therefore, in order to display an image corresponding to an NTSC system video signal in a high-vision system, the present applicant has previously proposed a conversion device for converting an NTSC system video signal into a high-vision video signal. (See Japanese Patent Application No. 6-205934). In this converter, a signal of each pixel constituting a high-definition video signal is converted into an NT signal.
Prediction is performed using a linear linear expression from a signal of a pixel in a predetermined area of the SC system and coefficient data (prediction coefficient value).

[0005]

In the above-described conversion apparatus, the signal of each pixel constituting a high-definition video signal is predicted by a linear linear equation. In some cases, an optimum pixel signal cannot be obtained. is there.

Accordingly, an object of the present invention is to provide an image signal conversion device and the like which can always perform optimal prediction of a pixel signal.

[0007]

An image signal conversion device according to the present invention converts a first image signal into a second image signal having the same or more pixels than the first image signal. In the image signal conversion device described above, a first pixel cutout unit that cuts out a signal of a pixel in a first region from the first image signal, and a pixel of the first region cut out by the first pixel cutout unit Class determining means for detecting a level distribution pattern of a signal, determining a class to which a signal of a predetermined pixel constituting a second image signal to be predicted based on the pattern belongs, and outputting class information; Coupling coefficient storage means for storing coupling coefficients of the neural network corresponding to each class shown, and coupling coefficient storage corresponding to class information output from the class determination means A coupling coefficient reading unit for reading a coupling coefficient from a stage, a second pixel extracting unit for extracting a signal of a pixel in a second region that is the same as or different from the first region from the first image signal, and a reading from the coupling coefficient storing unit. Neural network for predicting a signal of a predetermined pixel forming a second image signal using a neural network based on the extracted coupling coefficient and a signal of a pixel in a second region extracted by the second pixel extracting unit. And a net prediction unit.

Here, the first and second pixel cutout means may be common, and in this case, the first and second regions are the same. Also, the neural network prediction means
For example, it is configured using a back propagation network. Further, the neural network prediction means may have, at its output, normalization means for converting a value indicating the degree of coupling obtained from a unit in the output layer of the neural network into a pixel signal value.

Further, in the image signal conversion method according to the present invention, the image signal is configured to convert the first image signal into a second image signal having the same number of pixels or more than the first image signal. In the conversion method, a first step of cutting out a signal of a pixel in a first area from a first image signal, and a level distribution pattern of a signal of a pixel in the first area cut out in the first step are detected. A second step of determining a class to which a signal of a predetermined pixel constituting a second image signal to be predicted based on the pattern belongs and outputting class information; and a class output in the second step. A third step of reading out from the storage means a coupling coefficient of the neural network corresponding to each class indicated by the information, and a second step which is the same as or different from the first area from the first image signal. 4 for cutting out the signals of pixels of the band
, The coupling coefficient read out in the third step, and the pixel signal of the second region cut out in the fourth step, a second image signal is formed using a neural network. A fifth step of predicting a signal of a predetermined pixel.

A signal of a pixel in a first region is cut out from the first image signal corresponding to a predetermined pixel constituting a second image signal to be predicted, and the second signal is extracted based on the level detection pattern. , The class to which the signal of the predetermined pixel constituting the image signal belongs is determined, and the class information is output. The coupling coefficient is read from the coupling coefficient storage means in accordance with the class information. Further, the second image signal is converted from the first image signal to the second image signal corresponding to a predetermined pixel constituting the second image signal.
The signal of the pixel in the region is cut out. And this second
Using the neural network, the signal of the predetermined pixel constituting the above-described second image signal is obtained from the signal of the pixel in the region and the coupling coefficient read from the coupling coefficient storage means. As described above, by predicting a pixel signal using a neural network, nonlinear prediction is performed, and it is possible to always perform optimal prediction of a pixel signal as compared with that predicted by a linear linear expression. It becomes possible.

Further, the coupling coefficient generation device according to the present invention is used when converting a first image signal into a second image signal having the same or more pixels as the first image signal. A device for generating a coupling coefficient of a neural network, a signal processing means for processing a teacher signal corresponding to a second image signal to obtain an input signal corresponding to the first image signal, and a plurality of pixels constituting the teacher signal A first pixel cutout means for sequentially cutting out the signals of the pixels in the first area from the input signal in correspondence with the signals of
A pattern of the level distribution of the signals of the pixels in the first region sequentially cut out by the pixel cutout means, and a class to which the signals of a plurality of pixels constituting the teacher signal belong is determined based on the pattern. A class determining means for outputting information, and a second signal for sequentially cutting out signals of pixels in a second area which is the same as or different from the first area from the input signal in correspondence with signals of a plurality of pixels constituting the teacher signal, respectively. A pixel cutout unit, class information output from the class determination unit and indicating a class to which each of a plurality of pixels constituting the teacher signal belongs, a signal of the plurality of pixels constituting the teacher signal, and a second pixel cutout Means for each class from the signal of the pixels in the second area sequentially cut out by the means by learning using a neural network. In which and a neural network learning means for obtaining a coupling coefficient of click.

Here, the first and second pixel cutout means may be common, and in this case, the first and second regions are the same. Further, for example, the neural network learning means is configured using a back propagation network.

The neural network learning means comprises, for example, each unit of an input layer, a hidden layer, and an output layer, and supplies the signals of the pixels in the second area sequentially cut out by the second pixel cutout means to the units of the input layer. A neural network unit, a normalization unit that converts a value indicating the degree of coupling obtained from a unit of an output layer of the neural network unit into a pixel signal value, a pixel signal value output from the normalization unit, Error detection means for comparing the signal of a predetermined pixel constituting the teacher signal corresponding to the signal of the pixel in the second area supplied to the input layer of the neural network unit to detect an error, and indicated by class information For each class, the signal of the pixel of the second area sequentially cut out by the second pixel cutout means and the signal of the predetermined pixel constituting the teacher signal corresponding thereto A coupling coefficient changing unit that changes a coupling coefficient in a unit of an output layer and a hidden layer of the neural network unit in a direction in which an error detected by the error detection unit falls within a predetermined range with respect to the learning data formed by the combination of A coupling coefficient determining unit that uses a coupling coefficient in a unit of an output layer and a hidden layer of the neural network unit when the value falls within a predetermined range as a learning result.

Further, the coupling coefficient generation method according to the present invention is used when converting a first image signal into a second image signal having the same or more pixels as the first image signal. In a method for generating a coupling coefficient of a neural network, a first step of processing a teacher signal corresponding to a second image signal to obtain an input signal corresponding to the first image signal; A second step of sequentially cutting out the signal of the pixel in the first area from the input signal corresponding to the signal of the pixel, and a level distribution of the signal of the pixel in the first area sequentially cut out in the second step And a third step of determining a class to which each of signals of a plurality of pixels constituting the teacher signal belongs based on the pattern and outputting class information, and forming the teacher signal. A teacher signal output in the fourth step and a teacher signal output in the third step for sequentially cutting out the signal of the pixel in the second area which is the same as or different from the first area from the input signal corresponding to the signals of the plurality of pixels, respectively. Neural information is derived from class information indicating the class to which the signals of the constituent pixels belong, the signals of the constituent pixels of the teacher signal, and the signals of the pixels in the second region sequentially cut out in the fourth step. A fifth step of obtaining a coupling coefficient for each class by learning through a network.

A second image signal, for example, a teacher signal corresponding to a high definition video signal is processed to obtain an input signal corresponding to a first image signal, for example, an NTSC video signal. The signals of the pixels in the first area are sequentially cut out from the input signal corresponding to the signals of the plurality of pixels constituting the teacher signal, and the teacher signal is extracted based on the level distribution pattern of the signals of the pixels in the first area. Are determined, and the class information to which the signals of the plurality of pixels each belong is determined and class information is output.

Further, the signals of the pixels in the second area are sequentially cut out from the input signal in correspondence with the signals of the plurality of pixels constituting the teacher signal. Then, learning by a neural network is performed based on the pixel signals of the second region, class information indicating a class to which the signals of the pixels constituting the teacher signal belong, and the signals of the pixels constituting the teacher signal. Thus, a coupling coefficient is obtained for each class.

[0017]

Embodiments of the present invention will be described below with reference to the drawings. FIG. 1 shows a configuration of an image signal conversion device 100 as an embodiment.
The image signal conversion apparatus 100 includes a pixel data (hereinafter, referred to as “SD pixel data”) constituting an NTSC video signal.
To obtain pixel data (hereinafter, referred to as “HD pixel data”) constituting a high-definition video signal.

The image signal conversion apparatus 100 has an input terminal 101 to which SD pixel data is supplied, and an input terminal 1
01, an area extracting circuit 102 as a pixel extracting means for extracting SD pixel data of an area corresponding to predetermined HD pixel data to be predicted from the SD pixel data to be predicted, and an SD extracted by the area extracting circuit 102. ADRC (Adaptive Dynamic
An ADRC circuit 103 that applies a range coding process to determine a class (space class) mainly representing a waveform in space and outputs class information.

FIGS. 4 and 5 show the positional relationship between SD pixels and HD pixels. In the region extraction circuit 102,
For example, as shown in FIG. 6, when trying to predict the HD pixel data y, the SD pixel data k ₁ to k ₅ located in the vicinity of HD pixel data y is cut out.

The ADRC circuit 103 performs an operation for compressing each SD pixel data from, for example, 8-bit data to 2-bit data for the purpose of patterning the level distribution of the SD pixel data extracted by the area extraction circuit 102. Done. Then, the ADRC circuit 103 outputs compressed data (requantized code) qi corresponding to each SD pixel data as class information of the space class.

Originally, ADRC is a VTR (Video Tape Rec)
order), which is an adaptive requantization method developed for high-performance coding. Since the local pattern of the signal level can be efficiently expressed with a short word length, the region cutout circuit is used in this embodiment. It is used for patterning the level distribution of the SD pixel data cut out at 102.

In the ADRC circuit 103, the maximum value of the SD pixel data in the area is MAX, the minimum value is MIN, and the dynamic range in the area is DR (= MAX−MIN +
1) Assuming that the number of requantization bits is p, each SD in the area
A re-quantization code qi is obtained from the pixel data ki by the operation of equation (1). However, in equation (1),
[] Means truncation processing. When the area cutout circuit 102 cuts out Na SD pixel data, i = 1 to Na. qi = [(ki−MIN + 0.5) · 2 ^p / DR] (1)

The image signal converting apparatus 100 further includes an area extracting circuit 104 for extracting SD pixel data of an area corresponding to predetermined HD pixel data to be predicted from the SD pixel data supplied to the input terminal 101; A class (motion class) mainly representing the degree of motion from the SD pixel data cut out by the region cutout circuit 104
Class determining circuit 1 for determining class and outputting class information
05.

In the area extracting circuit 104, for example, FIG.
As shown in (1), when trying to predict HD pixel data y, ten SD pixel data m _{1 to} m ₅ and n _{1 to} n ₅ located near these HD pixel data y are cut out.

In the motion class determining circuit 105, the SD pixel data mi and ni extracted by the area extracting circuit 104
, A threshold value process is performed on the average value of the absolute value of the difference, and class information MV of a motion class, which is a motion index, is output.

That is, the motion class determination circuit 105 calculates the average value AV of the absolute value of the difference by the equation (2). When the area cutout circuit 104 cuts out, for example, ten pieces of SD pixel data m _{1 to} m ₅ and n _{1 to} n ₅ as described above, Nb in equation (2) is 5.

[0027]

(Equation 1)

Then, in the motion class determination circuit 105,
The average value AV calculated as described above is compared with one or more threshold values to obtain class information MV.
For example, three thresholds th ₁ , th ₂ , th ₃ (th ₁ <
th ₂ <th ₃₎ are prepared, when determining the four motion classes, if the _{AV ≦ th 1 MV = 0,} th 1 <AV ≦
When the th ₂ when the _{MV = 1, th 2 <AV} ≦ th 3 when the MV = 2, th ₃ <AV are MV = 3.

The image signal conversion device 100 is an ADR.
The requantization code qi as the class information of the space class output from the C circuit 103 and the motion class determination circuit 10
5, a class code generating circuit 106 for obtaining a class code CL indicating the class to which the HD pixel data to be predicted belongs, based on the motion class information MV outputted from 5
have. In the class code generation circuit 106,
The calculation of the class code CL is performed by the equation (3). In equation (3), Na is the number of SD pixel data cut out by the area cutout circuit 102, and p is AD
4 shows the number of requantization bits in the RC circuit 103.

[0030]

(Equation 2)

The image signal conversion device 100 also has a ROM table 107 as storage means in which coupling coefficients of a neural network used in a neural network prediction circuit described later are stored for each class.
The class code generation circuit 106 is stored in the ROM table 107.
The output class code CL is supplied as read address information, and a coupling coefficient Wi corresponding to the class code CL is read from the ROM table 107.

The image signal conversion apparatus 100 further includes an area extracting circuit 108 for extracting SD pixel data of an area corresponding to predetermined HD pixel data to be predicted from the SD pixel data supplied to the input terminal 101, SD pixel data extracted by the area extraction circuit 108;
A neural network predicting circuit 109 for obtaining HD pixel data to be predicted from the coupling coefficient Wi read from the ROM table 107 using a neural network;
Output terminal 1 for deriving HD pixel data determined in step 09
10 is provided.

In the area extracting circuit 108, for example, FIG.
As shown in the figure, when the HD pixel data y is to be predicted, 25 S pixels located near the HD pixel data y
D pixel data x _{1 to} x ₂₅ are cut out. Here, the region cutout circuit 108 is the same as the region cutout circuit 10 described above.
2 and may be common.

FIG. 2 shows a neural network prediction circuit 109.
Is an example in which a back propagation neural network is used. The neural network prediction circuit 109 includes a neural network unit 121 and a normalization unit 122 that converts a value indicating the degree of coupling obtained from a unit in an output layer of the neural network unit 121 into a pixel signal value and outputs the pixel signal value. It is configured.

The neural network unit 121 is composed of three layers. The input layer has n units, the hidden layer has four units, and the output layer has one unit.
In this case, each unit of the input layer includes n pieces of SD pixel data x extracted by the above-described area extraction circuit 108.
₁ ~x _n are supplied. Each unit of the input layer receives the supplied SD pixel data x ₁ to x ₁ .
Distribute x _n to each unit in the hidden layer without any change.

Each unit in the hidden layer is connected to each unit in the input layer by a coupling coefficient. Then, a value indicating the degree of coupling output from each unit in the hidden layer is supplied to the unit in the output layer. Further, the units of the output layer are connected to the units of the hidden layer by a coupling coefficient. Then, a value indicating the degree of coupling output from the unit of the output layer is supplied to the normalization unit 122 as an output value of the neural network unit 121.

FIG. 3 shows an example of the configuration of the units of the hidden layer and the output layer. As the number of units of the previous layer is m-number, the output value of each unit of the prior layer and A ₁ to A _m,
Also, the coupling coefficient in coupling with each unit of the preceding layer is W ₁
When と す る W _m , the output value U of the unit is expressed by equation (4). In the equation (4), f () is a transfer function as a non-linear element, and a function such as the equation (5) is often used. In the equation (5), a is an appropriate real number.

[0038]

(Equation 3)

As described above, the ROM table 10
Coupling coefficient Wi read out in response from 7 to the class code CL is to be used as a coupling coefficient W ₁ to _W-m in units of the hidden layer and output layer.

The value indicating the degree of coupling obtained from the unit of the output layer of the neural network unit 121 is a value larger than 0 and smaller than 1. for that reason,
For example, when the pixel signal value is represented by 8 bits, the normalization unit 122 multiplies the value indicating the degree of combination by “256” to obtain the HD pixel data y as the prediction target pixel value.
To get

The neural network prediction circuit 10 shown in FIG.
9 is an example and is not limited to this. For example, the neural network unit 121 can be configured to have four or more layers by increasing the number of hidden layers. Also,
The number of units in the hidden layer and the output layer can be arbitrarily set.

The operation of the image signal converter 100 shown in FIG. 1 will be described. Predetermined HD pixel data y to be predicted
In response to the above, SD pixel data ki of a predetermined area is cut out from the SD pixel data supplied to the input terminal 101 by the area cutout circuit 102, and the ADRC circuit 103 performs ADRC processing on each of the cut out SD pixel data ki. To obtain a requantization code qi as class information of a space class (mainly a class classification for representing a waveform in space).

Also, the SD supplied to the input terminal 101 corresponding to the HD pixel data y to be predicted
The region extraction circuit 104 extracts the S of a predetermined region from the pixel data.
D pixel data mi and ni are cut out, and a motion class determination circuit 10 is obtained from each of the cut out SD pixel data mi and ni.
5 obtains class information MV indicating a motion class (mainly a class classification for representing the degree of motion).

The motion class information MV and the above-mentioned ADR
From the requantized code qi obtained by the C circuit 103, a class code CL as class information indicating a class to which the HD pixel data y to be predicted by the class code generation circuit 106 belongs is obtained. And this class code C
L is supplied as read address information to the ROM table 107, and the coupling coefficient Wi of the neural network corresponding to the class to which the HD pixel data y to be predicted belongs is read from the ROM table 107.

The SD supplied to the input terminal 101 corresponding to the HD pixel data y to be predicted as described above.
The area extraction circuit 108 extracts the S of a predetermined area from the pixel data.
D pixel data xi is cut out. Then, the neural network predicting circuit 109 uses the neural network to predict the HD pixel to be predicted from the extracted SD pixel data xi and the coupling coefficient Wi read from the ROM table 107 as described above. Data y
Is required. Then, the HD pixel data y sequentially output from the neural network prediction circuit 109 is derived to an output terminal 110.

The image signal conversion apparatus 100 shown in FIG. 1 predicts HD pixel data y using a neural network, and performs nonlinear prediction (including linear prediction). It is possible to always optimally predict the HD pixel data y as compared with the prediction by the linear expression.

The RO of the image signal conversion device 100
As described above, the coupling coefficient of the neural network corresponding to each class is stored in the M table 107. The coefficient data is generated in advance by learning. FIG. 9 shows a configuration example of a coupling coefficient generation device 200 that generates coupling coefficients of a neural network for each class by learning.

The coupling coefficient generator 200 performs an input terminal 201 to which HD pixel data constituting a HDTV video signal as a teacher signal is supplied, and performs horizontal and vertical thinning filter processing on the HD pixel data. A signal processing unit 202 for obtaining SD pixel data constituting an NTSC video signal as an input signal. In the signal processing unit 202, the HD pixel data is thinned by a vertical thinning filter so that the number of lines in the vertical direction in the field is halved, and the number of pixels in the horizontal direction is further thinned by the horizontal thinning filter. The thinning processing is performed so as to reduce to ２. Therefore, the positional relationship between the SD pixel and the HD pixel is shown in FIGS.
It becomes as shown in.

The coupling coefficient generation device 200 outputs a signal from the signal processing unit 202 corresponding to each of a plurality of HD pixel data as a prediction target pixel value obtained from the HD pixel data supplied to the input terminal 201. A region extracting circuit 203 for sequentially extracting SD pixel data of a predetermined region from the SD pixel data to be extracted, and ADRC processing is applied to the SD pixel data sequentially extracted by the region extracting circuit 203 to mainly generate a waveform in space. ADRC circuit 20 for determining a class to be represented (space class) and outputting class information
And 4.

The area cutout circuit 203 has the same configuration as the area cutout circuit 102 of the image signal conversion apparatus 100 described above. For example, as shown in FIG. 6, SD pixel data k _{1 to} k ₅ located near the HD pixel data y corresponding to the HD pixel data y as the prediction target pixel value are output from the area cutout circuit 203. It is cut out. Also,
The ADRC circuit 204 is also used for the image signal converter 10 described above.
The configuration is the same as that of the 0 ADRC circuit 103. This AD
From the RC circuit 204, the SD of the predetermined area cut out corresponding to each HD pixel data as the prediction target value is
The requantization code qi is output as class information indicating a space class for each pixel data.

The coupling coefficient generation device 200 sequentially converts the SD pixel data in a predetermined area from the SD pixel data output from the signal processing unit 202 in correspondence with each of the HD pixel data as the above-described prediction target pixel values. An area extracting circuit 205 for extracting, and a motion class determining circuit 206 for determining a class (motion class) mainly representing the degree of motion and outputting class information based on the SD pixel data extracted by the area extracting circuit 205; have.

The area cutout circuit 205 is configured similarly to the area cutout circuit 104 of the image signal conversion device 100 described above. As shown in FIG. 7, for example, as shown in FIG. 7, the area cutout circuit 205 generates ten SD pixel data m ₁ to m 10 located in the vicinity of the HD pixel data y corresponding to the HD pixel data y as the prediction target pixel value. m ₅ and n _{1 to} n ₅ are cut out. The motion class determination circuit 206 of the image signal conversion device 100 described above
5 is configured in the same manner. This motion class determination circuit 206
, The motion class information MV, which is a motion index, is output for each of the SD pixel data of the predetermined region cut out corresponding to each HD pixel data as the prediction target pixel value.

Further, the coupling coefficient generating device 200 is adapted to execute the ADR
The requantization code qi as the class information of the space class output from the C circuit 204 and the motion class determination circuit 20
6 has a class code generation circuit 207 for obtaining a class code CL based on the class information MV of the motion class output from the control unit 6. This class code generation circuit 207
Has the same configuration as the class code generation circuit 106 of the image signal conversion device 100 described above. The class code generation circuit 207 outputs a class code CL indicating a class to which the HD pixel data belongs, corresponding to each HD pixel data as a prediction target pixel value.

The coupling coefficient generation device 200 sequentially converts the SD pixel data in a predetermined area from the SD pixel data output from the signal processing unit 202 in correspondence with each of the HD pixel data as the above-described pixel values to be predicted. An area extracting circuit 208 is provided. Area extraction circuit 2
08 is configured in the same manner as the above-described area cutout circuit 108 of the image signal conversion device 100. As shown in FIG. 8, for example, as shown in FIG. 8, from the area cutout circuit 208, 25 SD pixel data x located near the HD pixel data y corresponding to the HD pixel data y as the prediction target pixel value
₁ ~x ₂₅ is cut out.

Further, the coupling coefficient generation device 200 converts each HD pixel data y as the prediction target pixel value obtained from the HD pixel data supplied to the input terminal 201 and each HD pixel data y as the prediction target pixel value. The SD pixel data xi sequentially cut out by the area cutout circuit 208 and the HD pixel data y as the prediction target pixel value respectively.
, A neural network learning circuit 209 that obtains a coupling coefficient for each class by learning using a neural network from the class code CL output from the class code generation circuit 207, and a neural network learning circuit 209 that obtains a coupling coefficient for each class. Coupling coefficient W for each class
and a memory 210 for storing i.

FIG. 10 shows a neural network learning circuit 20.
9 shows an example of the configuration of the main part of FIG. 9, in which a back propagation neural network is used as in the neural network prediction circuit 109 shown in FIG. The neural network learning circuit 209 includes a neural network unit 221, a normalization unit 222 that converts a value indicating the degree of coupling output from a unit of the output layer of the neural network unit 221 into a pixel signal value and outputs the pixel signal value. The pixel signal value yo output from the normalization unit 222 is compared with the HD pixel data y as the pixel value to be predicted, and an error ε is calculated.
And an error detection unit 223 that detects Neural network unit 221 and normalization unit 2
22 has the same configuration as the neural network unit 121 and the normalization unit 122 of the neural network prediction circuit 109 shown in FIG.

In the neural network learning circuit 209, a coupling coefficient is generated for each class using a plurality of learning data. Here, one piece of learning data is composed of a combination of one piece of HD pixel data y as a prediction target pixel value and n pieces of corresponding SD pixel data x _{1 to} x _n .

Here, the generation of the coupling coefficient in a certain class is performed as follows. First, in the state where the coupling coefficients in the units of the hidden layer and the output layer of the neural network unit 221 are initial values, n pieces of S constituting the first learning data among a plurality of learning data of a certain class are set.
The D pixel data x _{1 to} x _n are sent to the neural network unit 22
1 input layer unit. In this state, the value indicating the degree of coupling obtained from the unit of the output layer is converted into the pixel signal value yo by the normalization unit 222. Then, the error detection unit 223 detects the error ε by comparing the pixel signal value yo with one piece of HD pixel data y constituting the first learning data. When this error ε is within a sufficiently small fixed range, the coupling coefficient in the unit of the hidden layer and the output layer in this state is supplied to the memory 210 as the coupling coefficient Wi of a certain class and stored therein. Is completed.

On the other hand, when the error ε is not within a certain range,
The coupling coefficient in the output layer of the neural network unit 221 and further in the unit of the hidden layer is changed so that the error ε falls within a certain range. That is, learning is performed by propagating an error in the output layer of the neural network unit 221 toward the input layer. Then, after changing the coupling coefficient in the unit of the thus output layer and the hidden layer, the second n pieces of SD pixel data x ₁ constituting the training data ~x _n among the plurality of training data of a class
To the input layer unit of the neural network unit 221.

In this state, the value indicating the degree of coupling obtained from the unit of the output layer is converted into a pixel signal value yo by the normalization section 222. Then, the error detection unit 223 calculates the pixel signal value yo and the 1 that constitutes the second learning data described above.
The error .epsilon. Is detected by comparing with the HD pixel data y.
When this error ε is within a sufficiently small fixed range, the coupling coefficient in the unit of the hidden layer and the output layer in this state is supplied to the memory 210 as the coupling coefficient Wi of a certain class and stored therein. Is completed. On the other hand, when the error ε is not within the certain range, the coupling coefficient in the output layer of the neural network unit 221 and further, the unit of the hidden layer is changed so that the error ε falls within the certain range.

In the same manner, the coupling coefficient in the unit of the output layer and the hidden layer of the neural network unit 221 is sequentially changed using the next learning data until the error ε falls within a certain range. However, if the error ε does not fall within a certain range even when all of a plurality of learning data of a certain class are used, the coupling coefficients finally used in the units of the output layer and the hidden layer are combined into a certain class. The coefficient Wi is supplied to and stored in the memory 210, and the operation of generating the coupling coefficient in a certain class is completed.

If the error ε does not fall within a certain range even when all of a plurality of learning data of a certain class are used, the initial value of the coupling coefficient in the unit of the output layer and the hidden layer and the equation (5) May be changed from the beginning by changing the real number a in the transfer function.

The operation of the coupling coefficient generator 200 shown in FIG. 9 will be described. The input terminal 201 is supplied with HD pixel data constituting a high definition video signal as a teacher signal, and the signal processing unit 2
In step 02, the horizontal and vertical decimation processes are performed to obtain SD pixel data constituting an NTSC video signal as an input signal.

Also, an area extracting circuit from the SD pixel data output from the signal processing unit 202 corresponding to each HD pixel data y as a prediction target pixel value obtained from the HD pixel data supplied to the input terminal 201. 203
, The SD pixel data ki of the predetermined area is sequentially cut out, and ADRC is performed on each of the cut out SD pixel data ki.
The circuit 204 performs an ADRC process to obtain a requantized code qi as class information of a space class (mainly a class classification for representing a waveform in space).

Further, in correspondence with each of the HD pixel data y as the pixel value to be predicted, the SD pixel data mi and ni of the predetermined area are sequentially extracted from the SD pixel data output from the signal processing unit 202 by the area extracting circuit 205. Cut out,
From the extracted SD pixel data mi and ni, the motion class determination circuit 206 obtains class information MV indicating a motion class (mainly a class classification for representing the degree of motion). Then, based on the class information MV and the requantized code qi obtained by the above-described ADRC circuit 204, each of the HD
A class code CL is obtained as class information indicating the class to which the pixel data y belongs.

Further, in accordance with each HD pixel data y as a prediction target pixel value, SD pixel data xi of a predetermined area is sequentially cut out from the SD pixel data output from the signal processing unit 202 by the area cutout circuit 208. . Then, each HD pixel data y as a prediction target pixel value obtained from the HD pixel data supplied to the input terminal 201
And SD pixel data xi sequentially cut out by the area cutout circuit 208 in correspondence with each HD pixel data y as a prediction target pixel value, and each H as a prediction target pixel value.
From the class code CL output from the class code generation circuit 207 corresponding to each of the D pixel data y, a coupling coefficient Wi is generated for each class by learning using a neural network. Then, the coupling coefficient Wi
Are stored in the memory 210 divided into addresses by class.

FIG. 11 shows a neural network learning circuit 20.
9 shows a learning flow of a coupling coefficient in No. 9; First,
In step ST1, one H as a prediction target pixel value obtained from the HD pixel data supplied to the input terminal 201 is output.
D pixel data y and the corresponding region extracting circuit 2
In combination with the n pieces of SD pixel data x ₁ ~x _n cut out 08, it produces one of the learning data. Next, in step ST2, the class classification of the learning data generated in step ST1 is performed based on the class code CL supplied from the class code generation circuit 207.

Next, in step ST3, the input terminal 201
It is determined whether or not the generation of the learning data corresponding to all of the HD pixel data y as the prediction target pixel values obtained from the HD pixel data supplied to is completed, and the generation of the learning data is not completed Returns to step ST1,
Continue generating learning data. On the other hand, when the generation of the learning data is completed, N = 1 is set in step ST4, and in step ST5, the N-class coupling coefficients Wi are generated by using a plurality of learning data classified into N classes. Process.

Next, in step ST6, it is determined whether or not the generation of the coupling coefficients Wi of all the classes has been completed. If not completed, N is incremented by 1 in step ST7, and thereafter, the process returns to step ST5 to perform processing for generating the N-class coupling coefficient Wi as described above. on the other hand,
When completed, the learning flow of the coupling coefficient is terminated.

FIG. 12 shows an N class coupling coefficient generation processing flow. First, in step ST11,
It is assumed that M = 1. Next, in step ST12, n SD pixel data constituting the M-th learning data of the N class is supplied to the unit of the input layer of the neural network unit 221. In this state, the degree of connection is determined by the unit of the output layer. Get the value shown.

Next, in step ST13, a value indicating the degree of coupling obtained from the unit of the output layer of the neural network unit 221 is supplied to the normalization unit 222 and normalized to obtain a pixel signal value yo. Then, in step ST14, one HD pixel data y constituting the M-th learning data of the N class is compared with the pixel signal value yo obtained in step ST13 by the error detection section 223 to detect the error ε. I do.

Next, in step ST15, step ST
It is determined whether the error ε detected in step 14 is within a sufficiently small fixed range. When the error ε is within a certain range, in step ST16, the coupling coefficient in the unit of the hidden layer and the output layer in that state is supplied to the memory 210 as an N-class coupling coefficient Wi and stored therein.
Thereafter, the operation of generating the coupling coefficient in the N class is completed.

On the other hand, when the error ε is not within a certain range,
In step ST17, the coupling coefficient in the unit of the output layer and the hidden layer of the neural network unit 221 is changed so that the error ε falls within a certain range. And
In step ST18, M is incremented by 1, and then in step ST19, it is determined whether M is greater than the number Mn of learning data classified into N classes.

When M> Mn, it means that all the learning data classified into N classes have been used. In step ST16, the coupling coefficient in the unit of the hidden layer and the output layer in that state is determined by the N class. Is supplied to and stored in the memory 210 as the coupling coefficient Wi, and then the generation operation of the coupling coefficient in the N class is terminated. On the other hand, M> M
If it is not n, the process returns to step ST12, and the above operation is repeated.

Although not shown in FIG. 12, if the error ε does not fall within a certain range even when all the learning data are used, the initial value of the coupling coefficient in the unit of the output layer and the hidden layer and (5 )), Changing the real number a in the transfer function
To restart the above-described generation operation from the beginning, when M> Mn in step ST19, the initial value of the coupling coefficient or the real number a of the transfer function may be changed, and the process may return to step ST11.

In the above-described embodiment, the ADRC circuits 103 (FIG. 1) and 204 (FIG. 9) are provided as information compression means for patterning a spatial waveform with a small number of bits. This is merely an example, and any information compression means can be provided as long as it can be represented by a class having a small number of signal waveform patterns.
M (Differential Pulse Code Modulation) or VQ (Ve
Compression means such as ctor Quantization) may be used.

In the above-described embodiment, the HD pixel data is obtained from the SD pixel data. However, for example, when obtaining other SD pixel data having a higher resolution than a certain SD pixel data, or obtaining a certain HD pixel data, The same applies to the case of obtaining other HD pixel data with a higher resolution. In this case, the number of pixels before and after the conversion is the same.

In the above embodiment, the NTS
Although an example in which a C-system video signal is converted to a Hi-Vision video signal has been described, the present invention is not limited to this, and the first image signal may have the same or a larger number of pixels as the first image signal. It is needless to say that the present invention can be similarly applied to the case of converting into the second image signal.

[0079]

According to the present invention, when a first image signal is converted into a second image signal having the same or larger number of pixels as the first image signal, the second image signal is converted to a second image signal using a neural network. And predicts a pixel signal that constitutes the image signal of the above. Since nonlinear prediction (including linear prediction) is performed, an optimal pixel signal is always obtained as compared with the conventional linear linear prediction. You can make predictions.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image signal conversion device according to an embodiment.

FIG. 2 is a diagram illustrating a configuration example of a main part of a neural network prediction circuit of the image signal conversion device.

FIG. 3 is a diagram illustrating a configuration example of units of a hidden layer and an output layer that constitute a neural network.

FIG. 4 is a schematic diagram for explaining a positional relationship between SD pixels and HD pixels.

FIG. 5 is a schematic diagram for explaining a positional relationship between SD pixels and HD pixels.

FIG. 6 is a schematic diagram for explaining SD pixel data used for space class classification.

FIG. 7 is a schematic diagram for explaining SD pixel data used for motion class classification.

FIG. 8 is a schematic diagram for explaining SD pixel data used for prediction of HD pixel data using a neural network.

FIG. 8 is a block diagram illustrating a configuration example of a coupling coefficient generation device that generates coupling coefficients used in a neural network prediction circuit by learning;

FIG. 10 is a diagram illustrating a configuration example of a main part of a neural network learning circuit of the coupling coefficient generation device.

FIG. 11 is a flowchart showing a learning flow of a coupling coefficient.

FIG. 12 is a flowchart for explaining N class coupling coefficient generation processing in a learning flow.

[Explanation of symbols]

100 ... image signal conversion device, 101, 201 ...
Input terminals, 102, 104, 108, 203, 205,
208 ... area cutout circuit, 103, 204 ...
ADRC circuit, 105, 206: motion class determination circuit, 106, 207: class code generation circuit, 10
7 ROM table, 109 neural network prediction circuit, 110 output terminal, 121, 221
..Neural network sections, 122, 222, etc.
Normalization unit, 209 ... Neural network learning circuit, 2
10 memory, 223 error detection unit

────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] July 11, 1997

[Procedure amendment 1]

[Document name to be amended] Statement

[Correction target item name] Brief description of drawings

[Correction method] Change

[Correction contents]

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of an image signal conversion device according to an embodiment.

FIG. 2 is a diagram illustrating a configuration example of a main part of a neural network prediction circuit of the image signal conversion device.

FIG. 3 is a diagram illustrating a configuration example of units of a hidden layer and an output layer that constitute a neural network.

FIG. 4 is a schematic diagram for explaining a positional relationship between SD pixels and HD pixels.

FIG. 5 is a schematic diagram for explaining a positional relationship between SD pixels and HD pixels.

FIG. 6 is a schematic diagram for explaining SD pixel data used for space class classification.

FIG. 7 is a schematic diagram for explaining SD pixel data used for motion class classification.

FIG. 8 is a schematic diagram for explaining SD pixel data used for prediction of HD pixel data using a neural network.

FIG. 9 is a block diagram illustrating a configuration example of a coupling coefficient generation device that generates coupling coefficients used in a neural network prediction circuit by learning;

FIG. 10 is a diagram illustrating a configuration example of a main part of a neural network learning circuit of the coupling coefficient generation device.

FIG. 11 is a flowchart showing a learning flow of a coupling coefficient.

FIG. 12 is a flowchart for explaining N class coupling coefficient generation processing in a learning flow.

[Description of Signs] 100: Image signal conversion device, 101, 201 ...
Input terminals, 102, 104, 108, 203, 205,
208 ... area cutout circuit, 103, 204 ...
ADRC circuit, 105, 206: motion class determination circuit, 106, 207: class code generation circuit, 10
7 ROM table, 109 neural network prediction circuit, 110 output terminal, 121, 221
..Neural network sections, 122, 222, etc.
Normalization unit, 209 ... Neural network learning circuit, 2
10 memory, 223 error detection unit

Claims (8)

[Claims] 1. An image signal conversion device for converting a first image signal into a second image signal having the same or more pixels as the first image signal, wherein the first image signal is A first pixel cutout unit that cuts out a signal of a pixel in a first region from a first pixel, and a level distribution pattern of a signal of a pixel in the first region cut out by the first pixel cutout unit,
Class determining means for determining a class to which a signal of a predetermined pixel constituting the second image signal to be predicted based on the pattern belongs and outputting class information; and a class corresponding to each class indicated by the class information. Coupling coefficient storage means for storing a coupling coefficient of the neural network; coupling coefficient reading means for reading the coupling coefficient from the coupling coefficient storage means in accordance with the class information output from the class determining means; A second pixel extracting unit for extracting a signal of a pixel in a second region that is the same as or different from the first region from one image signal; a coupling coefficient read from the coupling coefficient storage unit; From the signals of the pixels in the second region cut out by the pixel cutout means of (2), using the neural network, And a neural network predicting means for predicting a signal of a predetermined pixel forming the image signal. 2. The image signal conversion device according to claim 1, wherein said neural network prediction means is configured using a back propagation network. 3. The neural network predicting means includes a normalizing means at an output part thereof for converting a value indicating a degree of coupling obtained from a unit of an output layer of the neural network into a pixel signal value. The image signal conversion device according to claim 1. 4. An image signal conversion method for converting a first image signal into a second image signal having the same or more pixels as the first image signal, wherein the first image signal is A first step of cutting out the signal of the pixel in the first area from, detecting the level distribution pattern of the signal of the pixel in the first area cut out in the first step, and making a prediction based on this pattern. A second step of determining a class to which a signal of a predetermined pixel constituting a second image signal belongs and outputting class information; and a class corresponding to each class indicated by the class information output in the second step. A third step of reading out the obtained coupling coefficient of the neural network from the storage means, and disconnecting a signal of a pixel in a second area which is the same as or different from the first area from the first image signal. Using the neural network from a fourth step of outputting, the coupling coefficient read in the third step, and a signal of a pixel in the second region extracted in the fourth step. A fifth step of predicting a signal of a predetermined pixel constituting the second image signal. 5. An apparatus for generating a coupling coefficient of a neural network used in converting a first image signal into a second image signal having the same or more pixels than the first image signal. Signal processing means for processing a teacher signal corresponding to the second image signal to obtain an input signal corresponding to the first image signal; and a signal processing means corresponding to a plurality of pixels constituting the teacher signal. A first pixel extracting means for sequentially extracting a signal of a pixel in a first area from the input signal; and a pattern of a level distribution of a signal of a pixel in the first area sequentially extracted by the first pixel extracting means. Class determining means for determining a class to which each of the plurality of pixels constituting the teacher signal belongs based on the pattern and outputting class information; and A second pixel cutout unit for sequentially cutting out signals of pixels in a second region that is the same as or different from the first region from the input signal, respectively, corresponding to the signals of the plurality of pixels constituting the signal; Class information indicating a class to which the signals of the plurality of pixels constituting the teacher signal output from the determining unit belong; signals of the plurality of pixels constituting the teacher signal; and a second pixel cutout unit A coupling coefficient generation device comprising: a neural network learning unit that obtains coupling coefficients of the neural network for each class by learning using a neural network from sequentially extracted pixel signals of the second region. 6. The coupling coefficient generation apparatus according to claim 5, wherein said neural network learning means is configured using a back propagation network. 7. The neural network learning means comprises units of an input layer, a hidden layer, and an output layer, and the second pixel extracting means sequentially cuts the second pixel.
A neural network unit in which a signal of a pixel in the region of the above is supplied to the input layer; The pixel signal value output from the normalizing means and the second signal supplied to the input layer of the neural network unit.
Error detecting means for detecting an error by comparing a signal of a predetermined pixel constituting the teacher signal corresponding to a signal of a pixel in the area, and extracting the second pixel for each class indicated by the class information The error detected by the error detecting means is sufficient for the learning data formed by the combination of the signal of the pixel of the second area sequentially cut out by the means and the signal of the predetermined pixel constituting the teacher signal corresponding thereto. Coupling coefficient changing means for changing a coupling coefficient in a unit of an output layer and a hidden layer of the neural network unit in a direction falling within a small fixed range; and an output of the neural network unit when the error is within the fixed range. 7. A coupling coefficient determining unit that uses a coupling coefficient in a unit of a layer and a hidden layer as a learning result. If coefficient generating device. 8. A method for generating a coupling coefficient of a neural network used in converting a first image signal into a second image signal having the same or more pixels than the first image signal. A first step of processing a teacher signal corresponding to the second image signal to obtain an input signal corresponding to the first image signal, and respectively corresponding to signals of a plurality of pixels constituting the teacher signal. A second step of sequentially cutting out signals of pixels of a first area from the input signal; and detecting a level distribution pattern of the signals of the pixels of the first area sequentially cut out in the second step. A third step of determining a class to which each of the signals of the plurality of pixels constituting the teacher signal belongs based on this pattern and outputting class information; and A fourth step of sequentially cutting out signals of pixels in a second area, which is the same as or different from the first area, from the input signal, corresponding to signals of a plurality of pixels, respectively, and is output in the third step. Class information indicating a class to which each of the signals of the plurality of pixels constituting the teacher signal belongs; signals of the plurality of pixels constituting the teacher signal; and the second signal sequentially cut out in the fourth step A fifth step of obtaining the above-mentioned coupling coefficient for each class from the signals of the pixels of the area by learning using a neural network.