JPH1040363A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processor having an input image element selecting part in which a rational input structure and input/output relation can be constructed at the time of using an interlace image as an object in an image processor constituted of neural networks. SOLUTION: An input image element selecting part 12 which inputs image data from an original image supplying part 11, and selects an input image element to an image conversion processing part 13 prepares a field image obtained by separating pixel information in a different time from a frame image, selects an input kernel constituted of only pixels in the same time including at least an object pixel for a spatial direction, and selects the input pixel so that the input kernel can be made smaller with the objective pixel as a center according as it goes back to the past from the objective pixel for a time direction. Thus, the input to a neural network as the image conversion processing part 13 can be rationally selected.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for irradiating or penetrating an object with an information carrier such as an ultrasonic wave or light or an electromagnetic wave including X-rays and measuring a change in the information carrier to obtain image information. Image processing apparatus such as a medical image diagnostic apparatus, a broadcast image processing apparatus, an industrial image processing apparatus, or a consumer image capturing and displaying apparatus, and in particular, input information when the image information is an image obtained by interlaced scanning, that is, a so-called interlaced image And an image processing apparatus that can rationally select the image processing device.

[0002]

2. Description of the Related Art Conventionally, in image processing in an image processing apparatus, as shown in FIG. 2, time-series data 2 of a predetermined area including a target pixel 1 in a symmetric image is selected and image processing of the target pixel is performed. I was

The above-described method of selecting input pixels is effective for non-interlaced images. However, the above-described method for selecting input pixels can be applied to an interlaced image represented by an ultrasonic tomographic image which is one of moving images. When the selection method is used, since information before and after the time is mixed for each raster scan, that is, for each line, there is a contradiction that the past and the future of the time of the processing pixel are switched by the processing line, or the relationship between the inputs is changed. It becomes complicated, and it becomes difficult to rationally construct a desired input / output relationship in the learning operation.

[0003] Accordingly, the present invention provides an image processing apparatus having an input pixel selection unit capable of constructing a reasonable input structure and input / output relation when an interlaced image is to be processed in an image processing apparatus constituted by a neural network. It is intended to provide a device.

[0004]

In order to achieve the above object, the present invention provides an original image supply section for outputting image data to be processed including a pixel of interest, and a predetermined pixel in the vicinity of the pixel of interest from the image data. A pixel selection unit that selects an input pixel group having an area as image processing image data and outputs image processing information; an image conversion processing unit that performs image conversion of a pixel of interest based on the image processing information; And a display unit for displaying the image information of the selected target pixel, wherein the pixel selection unit divides the image data from the original image supply unit into a field image group for the same time information, and Image processing by selecting the input pixel group from the set of field images or an input pixel group formed of input pixels at the same time or in the past as the pixel of interest as image processing image data. And outputs the broadcast.

[0005] The input pixel group may be composed of input pixels having a region symmetrical with respect to the pixel of interest or input pixels whose area becomes smaller as the distance from the pixel of interest increases. Further, the image data for image processing may be constituted by a target pixel and an input pixel group that is a region that is past the target pixel and is symmetrical with respect to the target pixel, and that becomes smaller as the distance from the target pixel in time becomes smaller.

[0006]

An embodiment of the present invention will be described below in detail with reference to the accompanying drawings. FIG. 1 is a block diagram showing an embodiment of the image processing apparatus according to the present invention. As shown in FIG. 1, the image processing apparatus includes an original image supply unit 1.
1, an input pixel selection unit 12, an image conversion processing unit 13, and a display unit 14.

The original image supply section 11 outputs image data to be processed.
Measurement or imaging of objects in medical image diagnostic devices such as radiographic devices, X-ray CT devices, nuclear medicine imaging devices or magnetic resonance imaging devices, broadcast image processing devices, industrial image processing devices, consumer image capturing and displaying devices, etc. This is a measurement unit that performs measurement, or a storage unit that stores image data measured for an object and reads it out as necessary.

[0008] The input pixel selection unit 12 converts the image data output from the original image supply unit 11 into an image conversion processing unit 1.
The input pixels are selected so that the input to 3 has a reasonable structure. The image conversion processing unit 13 performs a required image conversion by inputting the image data selected by the input pixel selection unit 12, and includes, for example, a filter unit.

Further, the display unit 14 is provided with the image processing unit 1.
3, for inputting and displaying the image information that has been subjected to the image conversion processing and output, and comprises, for example, a television monitor.

Here, the operation of the input pixel selecting section 12 thus configured in the present invention will be described. First,
The relationship between the pixel and the time in the target interlaced image will be described with reference to FIG. Reference numeral 21 in FIG. 3 represents a pixel in the interlaced image. In the frame image 22, the even field 23 and the odd field 24 are y
It exists alternately in the axial direction. In the interlaced image, the odd field 24 is the even field 23
For example, since the data is captured with a delay of 1/60 second, the pixels adjacent in the y-axis direction have different times.

Next, how the input pixel selection section 12 selects an input pixel will be described. In general, an input pixel required to obtain an output by performing processing on a certain target pixel at a certain time (frame) by an image processing device formed of a neural network includes at least neighboring pixels at the same time as the target pixel itself and It is a neighboring pixel of the past frame. However, when an interlaced image as shown in FIG. 3 is to be processed, using an input pixel selected by the above general method may cause a contradiction that the past and future times of the processed pixels are switched by the processing line. However, the relationship between the input pixels becomes complicated, and it becomes difficult to construct a reasonable input / output relationship in the learning operation of the image conversion processing unit 13.

This contradiction will be described with reference to FIG.
In a general method, a frame image is processed as an image having field images having different times alternately, and is selected as shown in FIG. In FIG. 4, the case where the pixel 32 at the time t is processed as the processing target pixel,
Comparing the case where the pixel 35 of −1 is processed as the processing target pixel, for example, the pixels 36 and 37 adjacent to both the processing target pixels 32 and 35 are pixels at different times. In other words, there is a contradiction that information in which the future and the past are exchanged is handled in the same frame image.

A description will be given of a specific example of how the input pixel selection unit in the present invention selects an input pixel. In FIG. 3, the input pixel in the y-axis direction is selected by dividing even and odd in the field as follows. When processing the target pixel at time t, that is, the even field target pixel 25 at time T, the target pixel itself and its neighboring pixels at the same time t in the spatial direction, and the kernel becomes smaller in the time direction as it goes back in the past, the target pixel becomes smaller. Input pixels that maintain symmetry with respect to are selected. In other words, in this case even
The input pixel 27, that is, the input pixel unit 27 whose target field is even is selected.

Further, when processing the target pixel 26 at the time t-1, ie, the odd field at the time T, the target pixel itself and its neighboring pixels at the same time t-1 in the spatial direction and the past in the time direction. The input pixel is selected such that the kernel becomes smaller as it goes back and the symmetry with the target pixel is maintained. That is, in this case, in the frame image T, input pixels are selected every other raster scan, and odd input pixels 28 are selected.

As described above, in the y-axis direction, input pixels are selected every other pixel in the frame image. In the x-axis direction, since adjacent pixels are at the same time, adjacent pixels are selected. As a method of selecting while maintaining symmetry, for example, the relationship between the number of input pixels Nt at time t and the number of input pixels Nt-1 at time t−1 is Nt = Nt−1 + n. Here, n is an odd number that is positive or negative. In FIG. 3, n is -1.

In other words, according to the present invention, when information at two preceding and succeeding times in one frame of an interlaced image is alternately arranged for each raster scan, an input pixel is selected every other raster scan as neighboring pixels at the same time. In addition, the input pixel is selected so as to maintain the symmetry with respect to the processing target pixel so that the input structure in the time direction becomes rational, and further away from the time at which the processing target pixel exists, that is, in the past, The input pixels are selected such that the input kernel in the spatial direction becomes smaller as going back, that is, the input structure becomes a cone.

Here, the reason why the input pixel is selected so that the input kernel becomes smaller as it goes back in the past and the reason why the input pixel is selected so as to maintain the symmetry are as follows. This is for fading with symmetry around the pixel. For example, as shown in FIG. 4, when the input pixel 31 having the same kernel size is selected for each frame in order to process the target pixel 32, if the partial input pixel 33 at the time t-1 and the time t-2 is noted, , The kernel size increases as it goes back to the past from the target pixel 32. This is an irrational input structure because a pixel having a low relation with the target pixel is selected more. For the above reasons, an input pixel selection method is adopted in which the kernel size is reduced as the pixel goes backward from the target pixel.

FIG. 5 shows an embodiment in which the above input pixel selection method is applied to an ultrasonic tomographic image which is an interlaced image. In the time-series ultrasonic tomographic image of FIG. 5, three time-series frame images 41 in which information at different times are mixed are separated into a time-series field image 42 including only information at the same time. In the case of processing the even target pixel 43, the 9 × 5 pixels each including the target pixel are set so that the input kernel becomes smaller from the fields t, t-1, t-2, t-3, and t-4 in the past. = 45 pixels, 7 × 4 = 28 pixels, 5 ×
A total of 95 input pixels 45, 3 = 15 pixels, 3 × 2 = 6 pixels, and one pixel, are selected. When processing the odd target pixel 44, the fields t-1, t-2, t-3, t-4,
The input pixel 46 is selected in the same manner as in the case of even from t-5. However, FIG. 5 is an example, and the number of input pixels may change in the spatial direction or the number of frames may change in the time direction.

Next, the operation of the image conversion processing unit for performing image conversion based on the input selected by the input pixel selection unit 12 will be described. The image conversion processing unit 13 is configured by a neural network 51 as shown in FIG. The neural network 51 inputs the data selected by the input pixel selection unit 12 and performs an image conversion process, and outputs data corresponding to a target pixel. The artificial neural element 52 includes an input layer and an intermediate layer. In addition, a network is formed by combining the output layers so as to form a layer structure, and a linear function is used as an input / output function of the final output layer. FIG. 7 is an explanatory diagram showing an artificial neural element 52 constituting the neural network 51. As shown in FIG. 7, an artificial neural element (hereinafter referred to as a “neuron model”) 52 is a multi-input one-output element that simulates the function of a biological neural element (neuron).
The input Ii (I1-In) and the coupling coefficient Wji (Wj1-W
jn) determines the output 0j. That is,
It is determined as in Equation 1 using the input / output function f (x).

(Equation 1) Here, θj is an offset of the input / output function corresponding to the threshold, and n is the number of inputs.

FIG. 6 described above shows a neural network 51 connected to form a layered structure and formed into a network.
FIG. As shown in the figure, the neural network 51 uses a large number of the neuron models 52 and connects the input layer, the intermediate layer, and the output layer so as to form a layered structure. It is configured to realize. In FIG. 6, reference numeral 53 denotes a branch connecting the input layer and the intermediate layer, and each neuron model 52 in the input layer is connected to all the neuron models 52 in the intermediate layer. The neural network 52 converts input information 55 supplied to the input layer and outputs the converted information as output information 56 from the output layer.

With this configuration, the neural network 52 is an information processing mechanism having a learning function and a self-organizing function. Then, the neural network 51 self-organizes by inputting a teacher signal 57 and learning so as to output desired output information 56 from the output layer with respect to the input information 55 supplied to the input layer. Here, in the input layer, an identity function as shown in Expression 2 is used as an input / output function.

(Equation 2) Thus, the input is output as it is. The modulation may be performed using another function instead of the identity function.

FIG. 8 is a graph showing a sigmoid function used as an input / output function of the intermediate layer of the neural network 51. In the above-mentioned intermediate layer, a sigmoid function that monotonically increases when the output fs is in the range of “0” to “1” is used as an input / output function as shown in FIG.

This sigmoid function is expressed as in Equation 3.

(Equation 3) Here, U0 is a parameter for controlling the inclination. Then, when this is differentiated, it is expressed by Equation 4 and has a feature that it can be expressed by the original sigmoid function.

(Equation 4) As described above, by using a non-linear function such as a sigmoid function for the intermediate layer, the neural network 51 can be used.
Can handle complicated nonlinear conversion processing.

FIG. 9 is a graph showing a linear function used as an input / output function of the output layer of the neural network 51. In the output layer, a linear function whose output increases and decreases linearly with respect to the input as shown in the figure is used as the input / output function instead of the sigmoid function. Now, assuming that the slope of the straight line is a, it is expressed as in Equation 5, for example, and differentiating it, it is expressed as in Equation 6.

(Equation 5)

(Equation 6) As a result, the output from the output layer becomes an analog value,
It becomes possible to handle analog values such as image information.

In the neural network 51, learning is performed so that desired output information 57 is obtained for the input information 55. That is, the teacher signal 57 serving as a teacher
The coupling coefficient between the branch 53 connecting the input layer and the intermediate layer and the branch 54 connecting the intermediate layer and the output layer is changed so that the error between the output information 56 and the output layer 56 is reduced. Self-organization to obtain FIG. 10 shows a learning algorithm of the neural network 51 for analog output. The meanings of the symbols in FIG. 10 are as follows. I; input information 0k; output of the k-th layer neuron model Wk; coupling coefficient θk between the k-th layer and the (k + 1) -th layer; offset T of the k-th layer neuron model; teacher signal ηw; learning coefficient ηθ for W; learning coefficient for θ δk; error amount fi for correcting the (k-1) th layer fi; identity function fl; linear function

The input information I, the output 0k of the k-th layer neuron model, the offset θk of the k-th layer neuron model, the teacher signal T, and the error amount δk for correcting the (k-1) -th layer are 1 The coupling coefficient Wk between the k-th layer and the (k + 1) -th layer is represented by a two-dimensional vector. Also, since the neural network shown in the example has three layers, the first layer is the input layer, and the second layer is the second layer.
The third layer is an intermediate layer, and the third layer is an output layer.

The learning algorithm will be described below. Mean square error E between teacher signal T and output 03 of output layer
Is represented by Equation 7.

(Equation 7) To reduce this, the coupling coefficient Wk between the k-th layer and the (k + 1) -th layer is corrected by the correction amount ΔWk shown in Expression 8.

(Equation 8)

Here, assuming Equation 9, δ3 that determines the amount of correction between the intermediate layer and the output layer is expressed by Equation 10.

(Equation 9)

(Equation 10) Therefore, the correction amount ΔW2 is expressed as in Expression 11.

[Equation 11] Accordingly, the output from the output layer becomes an analog value as described above. Further, δ2 which determines the correction amount between the input layer and the intermediate layer is expressed by Expression 12, and the correction amount ΔW1 is expressed by Expression 13.

(Equation 12)

(Equation 13) By similarly obtaining the offset, FIG.
The algorithm is shown below.

Then, based on the teacher signal 57 given by the operator, learning of the neural network 51 is performed by such an algorithm. Here, although an example is shown for the three-layer neural network 51,
By increasing the number of hidden layers, a neural network 51 of three or more layers can be configured. In this case, the learning algorithm only has to propagate the error δ back and asymptotically obtain the correction amount, and includes a more complicated transformation. It is possible to deal with image information, and the neural network 51
The image data converted in step (1) is converted into an appropriate density scale (not shown) and input to the display unit 14 shown in FIG.

FIG. 11 is a diagram for explaining the second embodiment of the present invention, and shows a quasi-linear region 60 of a sigmoid function. This embodiment uses a function other than the linear function, for example, a sigmoid function shown in FIG. 11 as an input / output function of the output layer at the final stage of the network in which the neural network 51 is connected in a layered structure in FIG. , The quasi-linear region 60 is limited and used as the range of the output image information. Also in this case, the image information can be output as an analog value in the same manner as described above.

FIG. 12 is a block diagram showing a third embodiment of the present invention. In this embodiment, the image conversion unit 1 shown in FIG.
3 is connected to the neural network 51 as the teacher image supply unit 15 for inputting a teacher signal 57 (see FIG. 6) for causing the neural network 51 to perform a learning operation. In the embodiment shown in FIG. 1, an average, a standard, or a specific teacher signal 57 is given at the time of shipment from a factory or at a use facility to perform necessary learning in advance. In some cases, the situation does not match the usage situation of individual users at the site, and therefore, it is intended to improve the situation. That is, in FIG. 12, the teacher image supply unit 15 is a measuring device for measuring or photographing an object or a storage device for storing measured image data of an object and reading it out as necessary. Is connected to the output end of the neural network 51 from which the output information 56 is output from the output layer (see FIG. 6), and each user or the like is selected in the input pixel selection unit 12 using the teacher image supply unit 15. An ideal teacher image for the image data is supplied as needed as a teacher signal 57, and the neural network 51 performs learning.

Next, the process of inputting the input pixels selected by the input pixel selection unit 12 to the image conversion processing unit 13 and processing the target image will be described with reference to FIG. Will be described. The input pixel selection unit 12 normalizes the density values of the input pixels 61 selected from the ultrasonic tomographic images, which are interlaced images, to values of −1 to 1 in the input density value normalization unit 63. The standardized value is given as an input to the input layer unit of the neural network 64 as the image conversion processing unit 13, and the information is sequentially propagated from the intermediate layer to the output layer, and the output 65 of the output unit is obtained. The output 65 is denormalized to a density value in an output pixel density value normalization unit 66 in a method reverse to the normalization performed in the input pixel density value normalization unit, and an output pixel density value 67 is obtained. This output pixel density value 67 is the image conversion processing result of the target pixel. The output image 68 is obtained by performing the above processing on the entire target image.

Next, a learning operation for enabling the neural network 64 to obtain a desired output for the pixel to be processed will be described. First, a teacher image 69 is created. When the purpose is to improve the image quality of an ultrasonic tomographic image, a high-quality ultrasonic tomographic image is used as a teacher image. A high-quality ultrasonic tomographic image is created by selecting a plurality of images from a time-series ultrasonic tomographic image original image and averaging them. In addition, in the case of an ultrasonic tomographic image having a moving part, an image synchronized with the movement is selected from the time-series image in order to prevent the moving part from being blurred by addition, and the averaging is performed. Create an image. The above-described method of obtaining a high-quality image is an example, and other filter processing or the like is used, or a plurality of filter processes that are different for each region in the image so as to be an image desired by the operator, for example, a high-frequency emphasis filter or the like. May be created.

The learning operation is performed as follows. An error is calculated between the value 71 obtained by normalizing the target pixel 70 in the created teacher image 69 by the output pixel normalizing unit and the output value 65 from the neural network, and the neural network is designed to reduce the error. Is changed. This operation is repeated in the attention area 72, and learning is performed until the number of repetitions reaches the specified number of times or the error becomes equal to or less than an allowable error. By this learning,
When an original image is input, a neural network is automatically constructed so that a processed image desired by the operator is output, and the neural network 64 acquires desired image filter input / output characteristics.

[0035]

According to the present invention, when an interlaced image is to be processed in an image processing apparatus constituted by a neural network, information at different times contained in one frame of the interlaced image is separated, and the information in the spatial direction is separated. By selecting the input information only from the image information at the same time, it is possible to perform consistent input selection such as predicting past information from future information. In addition, by reducing the input kernel centering on the target pixel as it goes back from the target pixel in the past, a lean and rational input structure is constructed in which the input kernel becomes smaller as the frame is less relevant to the target pixel. At the same time, the input-output relationship can be rationally constructed in the learning operation of the image processing apparatus formed by the neural network.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 illustrates a conventional input pixel selection method.

FIG. 3 is a diagram showing a relationship between pixels and time in an interlaced image and an input pixel selection method.

FIG. 4 is an explanatory diagram when input kernels of the same size are selected.

FIG. 5 is a diagram showing an input pixel selection method in an ultrasonic tomographic image.

FIG. 6 is an explanatory diagram showing a hierarchical structure of a neural network.

FIG. 7 is an explanatory view showing an artificial neural element forming the neural network.

FIG. 8 is a diagram showing a sigmoid function of an input / output function of an intermediate layer of the neural network.

FIG. 9 is a diagram showing a linear function of an input / output function of an output layer of the neural network.

FIG. 10 is an explanatory diagram showing a learning algorithm of a neural network for outputting an analog value.

FIG. 11 is a diagram illustrating a quasi-linear region of a sigmoid function according to the second embodiment of the present invention.

FIG. 12 is a block diagram showing a third embodiment of the present invention.

FIG. 13 is an explanatory diagram showing a process of obtaining a processing result in the neural network and a learning method.

[Explanation of symbols]

Reference Signs List 11 original image supply unit 12 input pixel selection unit 13 image conversion processing unit 14 display unit 15 teacher image supply unit 21 pixels in interlaced image 22 frame image 23 even field 24 odd field 25 target pixel in even field 26 inside odd field 27 The input pixel in the y-axis direction selected for the target pixel in the even field 28 The input pixel in the y-axis direction selected for the target pixel in the odd field 41 Time-series frame image 42 Time-series field Image 43 Target pixel in the even field 44 Target pixel in the odd field 45 Input pixel selected for the target pixel in the even field 46 Input pixel selected for the target pixel in the odd field 51 Neural network 52 Artificial God Element 53 Coupling coefficient between input layer and intermediate layer 54 Coupling coefficient between intermediate layer and output layer 55 Input information 56 Output information 57 Teacher signal 60 Quasi-linear region of sigmoid function 61 Input pixel 62 Target pixel 63 Input pixel density value normalization unit 64 Neural Network 65 Output of Neural Network 66 Output Pixel Density Value Normalization Unit 67 Output Density Value 68 Output Image 69 Teacher Image 70 Target Pixel Density Value in Teacher Image 71 Standardized Teacher Signal 72 Region of Interest

Claims (9)

[Claims] An original image supply unit for outputting image data to be processed including a target pixel, and an input pixel group having a target pixel and a predetermined area near the target pixel is selected as image processing image data from the image data. A pixel selection unit that outputs image processing information, an image conversion processing unit that performs image conversion of the pixel of interest based on the image processing information, and a display unit that displays the image information of the pixel of interest that has undergone the image conversion. In the image processing apparatus, the pixel selection unit divides the image data from the original image supply unit into field image groups for the same time information, and converts the input pixel group from the divided field image groups into image processing image data. And outputting image processing information. 2. An original image supply unit for outputting image data to be processed including a target pixel, and an input pixel group having a target pixel and a predetermined area near the target pixel is selected from the image data as image processing image data. A pixel selection unit that outputs image processing information, an image conversion processing unit that performs image conversion of the pixel of interest based on the image processing information, and a display unit that displays the image information of the pixel of interest that has undergone the image conversion. In the image processing device, the pixel selection unit divides the image data from the original image supply unit into field image groups for the same time information, and outputs the input pixels at the same time or in the past as the target pixel from the divided field image groups. An image processing apparatus for selecting an input pixel group composed of the following as image data for image processing and outputting image processing information. 3. The input pixel group according to claim 1, wherein the input pixel group includes input pixels having a region symmetrical with respect to a target pixel.
The image processing apparatus according to any one of the preceding claims. 4. The image processing apparatus according to claim 1, wherein the input pixel group includes input pixels whose area becomes smaller as the distance from the target pixel increases. 5. The image processing image data comprises a pixel of interest and an input pixel group that is a region that is past the pixel of interest, is symmetric about the pixel of interest, and has a smaller area as the distance from the pixel of interest increases. 3. The image processing apparatus according to claim 1, wherein 6. The image conversion processing unit forms a network by connecting artificial neural network elements so as to form a layer structure of an input layer, an intermediate layer, and an output layer, and forms image data from the input pixel selection unit. 3. The image processing apparatus according to claim 1, wherein the image processing apparatus comprises a neural network configured to perform image conversion by inputting the image data and output data corresponding to the pixel of interest. 7. The image processing apparatus according to claim 6, wherein the neural network uses a linear function as an input / output function of an output layer of a final stage, and outputs image information as an analog value. 8. The neural network according to claim 1, wherein the input / output function of the final output layer is axisymmetric about 0 and 0
The vicinity is linear, and a continuous function having a characteristic of exhibiting a saturation characteristic when moving away from 0 is used, the quasi-linear region is limited to a range of output image information, and the image information is output as an analog value. 7. The image processing device according to 6. 9. The image processing apparatus according to claim 6, wherein said neural network is connected to a teacher image supply unit for inputting a teacher signal for causing said neural network to perform a learning operation.