JP2976302B2 - Image processing device - Google Patents

Description

Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus using a computer having a neural network structure that performs arithmetic processing on image information based on a learning rule created in advance.

[Conventional technology]

In a computer having a neural network structure, the amount of internal calculation is very large, and thus image data input to the input layer at one time cannot be increased. For this purpose, a conventional image processing apparatus having a neural network structure inputs a signal to one image frame in units of a part of image data, that is, an image block of 8 × 8 pixels, instead of entire image data, Arithmetic processing is executed based on this signal. This is discussed, for example, in the IEICE, IEICE Preprints IE88-62 (1988), “Study of Image Data Pressure (Neuro-CODEC) Using Neural Networks” in Paragraphs 57 to 64. ing.

[Problems to be solved by the invention]

However, in the above-described conventional technique, when the number of pixels constituting an image frame is large, only a local image can be set as a calculation target, and a calculation cannot be performed on contents expressed in the entire image. There is.

Further, if the position and size of the target object in the image frame are different, it is difficult to effectively use the learning rule created in advance, and there is a problem that the accuracy of the calculation result is reduced.

An object of the present invention is to provide an image processing apparatus capable of improving the accuracy of a calculation result by normalizing a position and a size of a target object while making the entire image signal in an image frame a calculation target. It is to be.

[Means for solving the problem]

In order to achieve the above object, an image processing apparatus according to the present invention comprises: an image frame accumulating unit that accumulates at least one image frame; Sample transfer means for sampling and transferring image data having a size of, learning rule storage means for storing a plurality of types of learning rules for normalizing the position and size of the target image, an input layer,
An arithmetic means for forming a neural network structure comprising an intermediate layer and an output layer, for performing an arithmetic operation on the image data from the sample transfer means in a stepwise manner based on the learning rule; And selecting means for setting sampling conditions of the sample transfer means while taking in the calculation result and sequentially selecting learning rules from the learning rule storage means.

Further, the image processing apparatus according to the present invention comprises: an image frame accumulating means for accumulating at least one image frame; and image data of an arbitrary size from an arbitrary position with respect to an arbitrary image frame in the image frame accumulating means. A sample transfer means for sampling and transferring, and a neural network structure consisting of an input layer, an intermediate layer and an output layer, and a plurality of learning rules for normalizing the position and size of the target image from other computers. An arithmetic means for performing stepwise arithmetic processing of the image data from the sample transfer means based on the captured learning rule; and each time the arithmetic means performs an arithmetic processing step by step, the arithmetic result is captured. Selecting means for sequentially selecting learning rules from the other computer while setting sampling conditions of the sample transfer means;
It is provided with.

Further, the present invention applies the above-described image processing device to a television camera control device or a high efficiency coding device.

[Action]

According to the above configuration, the sample transfer unit samples image data of an arbitrary size from an arbitrary position with respect to an arbitrary image frame in the image frame storage unit, and converts the sampled image data into a neural network structure. Transfer to arithmetic means. The calculating means performs a first-stage calculating process on the transferred image data based on a certain learning rule in the learning rule storing means. At this time, every time the calculation processing is performed, the calculation result is sent to the selection means. The selection means outputs a signal of the sampling condition setting to the sample transfer means based on the received operation result,
A signal for selecting which learning rule to use next is output to the learning rule storage means. Then, the calculating means converts the next transferred image data into the second data based on the next certain learning rule.
Performs stepwise computation. As described above, by sequentially performing the arithmetic processing, the entire image data is obtained.

If the learning rule accumulation means is installed inside another computer (for example, a large computer), the arithmetic means fetches the learning rule from the other computer. In this case, the selection means outputs a learning rule selection signal to another computer.

Further, if the image processing device of the present invention is applied to a camera control device, the object can be automatically followed by the camera so that the target image is at the center of the screen.

Further, when the image processing device of the present invention is applied to a high-efficiency coding device, a specific region of a target image can be extracted by calculating a difference between frame images.

〔Example〕

 An embodiment of the present invention will be described below with reference to the drawings.

FIG. 1 is a basic configuration diagram of an image processing apparatus according to the present invention. In the figure, the image frame memory 110 is capable of storing a plurality of images. For example, when moving image information that changes over time is input, not only the stationary state of the target image but also a motion component is used. Arithmetic processing can be performed. The sample transfer unit 120 has a function of sampling an image signal of an arbitrary size from an arbitrary position with respect to an arbitrary image frame in the image frame memory 110. When the image frame memory 110 is configured by a semiconductor memory, the sample transfer unit 120 is configured by an address generation circuit that accesses the frame memory.

The computer 130 has a neural network structure and includes an input layer, a plurality of intermediate layers, and an output layer. And
The layers between the input layer and the intermediate layer and the layers between the intermediate layer and the output layer are connected by a network having a weight coefficient.
The neural network structure employed in the computer 130 is composed of a combination of basic circuits as shown in FIG. Here, the weight coefficient W of the network connecting the above layers
_{The ij} can be created in advance by a learning procedure known as a back propagation method, for example. The present invention does not depend on this learning method, and can select an arbitrary learning method.

This learning result is stored in the learning rule storage unit 140,
The learning rule storage unit 140 stores a plurality of types of learning rules according to the purpose of the operation, and can arbitrarily select the learning rules. Then, the scheduler 150 sets the sampling conditions of the sample transfer unit 120, and the learning rule accumulation unit 140 sequentially selects the learning rule, and executes a plurality of arithmetic processes.

Further, by providing the scheduler 150 with a logical operation function for performing a condition determination or the like using an operation result, a plurality of operation processes can be sequentially executed with conditional branching. Thereby, for example, the sequential operation flow described in the flowchart can be easily realized.

FIG. 3 shows an image frame memory 110 and a sample transfer unit 120.
Is shown in detail. The sample transfer section 120 includes an address generation section 120A and a transfer section 120B.
In the address generator 120A, the sampling start point (sx, s
y) and a sampling interval (dx, dy) are designated, and these information are transmitted to the image frame memory 110 as signals x, y. In the image frame memory 110, a sampling pixel area is set based on the transmitted signals x and y. Then, the image data of the sampling pixel area is output to the computer 130 via the transfer unit 120B.

In the present embodiment, the image frame memory 110, the sample transfer unit 120, the computer 130, the learning rule storage unit 140, and the scheduler 150 are respectively an image frame storage unit, a sample transfer unit, a calculation unit, a learning rule storage unit, and a selection unit. Is equivalent to

Next, an example of a signal processing procedure for a human face image will be described with reference to FIG. Here, a combination of computer processing of a neural network structure using a plurality of learning rules, logical condition determination processing using the calculation results of the computer, and inference processing based on a knowledge database based on preset common sense is used. 4 shows a procedure for outputting a specific part of a face image as a calculation result. Hereinafter, the processing content will be described according to the step numbers (1) to (6) shown in the figure.

First, in steps (1) and (2), an operation using learning rule 1 is performed in order to normalize the position and size of the target person face image. As shown in FIG. 5, the position and size of the target image are not constant depending on the imaging conditions. For this reason, the sample transfer unit 120 reduces the size, in other words, roughly samples and transfers it to the input layer so that image information can be input from the entire screen.

As a calculation result, a correction signal for the position and size of the target object in the human-powered image is output. Using this result, the scheduler 150 resets the sampling conditions in the sample transfer unit 120 and performs the operation again using the same learning rule. If the correction signal output as the calculation result is equal to or less than a certain value,
It is determined that the position and size normalization processing has been completed, and the process proceeds to the next step.

The reason why the repetitive execution is performed using the convergence condition is that the calculation result based on the learning rule may include some errors, thereby avoiding erroneous determination due to the error.

The learning rule used here is a result created by a learning procedure with a target value satisfying the convergence condition, and the algorithm thereof may be any method such as back propagation.

Next, in step (3), if the position and size of the human face image can be normalized, the specific part of the face (for example, the position of the lips) can be estimated based on a knowledge database created in advance. This guess is, as shown in FIG.
The position and size signals obtained in the previous steps (1) and (2) are combined with the lip position and size signals based on the database to execute the logical operation.

However, in a target image such as a human face image that varies depending on personality, it is easy to make an error to uniquely determine the position of the lips. Therefore, the result of this step is
As a setting condition of the sample transfer unit 120, an operation for improving the accuracy is performed in the following steps.

Further, in step (4) and step (5), the seventh
As shown in the figure, computer processing of a neural network structure is performed in the same manner as in steps (1) and (2). However, the learning rule 2 is used for the computer processing here. Then, the error component of the position and size of the lips left in step (3) is
The calculation is repeatedly performed until the value converges to a certain value or less.

Then, in step (6), the position and size of the lips are input under certain conditions using the results obtained up to the previous step. In this step, the shape of the lips is determined using learning rule 3. Classify into states. That is, as shown in FIG. 8, a shape having the smallest error is calculated from the shapes of the lips which have been compiled as learning rules in advance using a computer having a neural network structure, and output as a state value.

As shown in FIG. 9, for example, the learning rules shown in the above example can be grouped into a plurality of types and can be selected according to the purpose.

By expressing the shape of the thus obtained face image as a state value, the face image can be realized with an extremely small amount of information.

Also, after calculating the position of a specific part such as a lip, by performing an encoding process with less deterioration compared to other parts on a portion centered on the position, attention is paid to human visual characteristics. The part to be collected can be compressed with less error. For example, in an encoding process using orthogonal transform, it is known that a combination of an image and a bit rate can be set relatively easily by providing a high frequency component selection unit. Therefore, as described above, the apparent image quality can be improved by calculating the positions of the eyes, lips, and the like, and increasing the image quality of the region centered on those portions as compared with other portions.

An image frame memory 110 storing moving image information is also provided.
Then, the position and size of the face image are calculated, and the conditions of the sample transfer unit 120 or the imaging device (for example, a TV camera) are set so that the face image is always located at the center of the screen. In this way, image encoding transmission can be performed while automatically following a person's face image in the center of the screen in a videophone, a video conference, or the like.

In the above-described execution example, the procedure of outputting the shape of the lips as the state value is shown. However, similarly, the shape of the eyes, the facial expression, and the like can be output as the state value. The apparatus according to the present embodiment is an apparatus for identifying the shape of an object (for example, FA (Factoy Automated
ation) device, etc.).

In addition, as shown in FIG.
By storing in the 110, it is possible to detect a motion component between frames. For example, a lip movement component can be detected, and the content of pronunciation can be determined.

Further, as shown in FIG. 11, the background image of the target object is stored in the background frame memory 510 in advance, the input image including the target object is stored in the input frame memory 520, and the difference between the two is compared with the comparison device 530. By calculating and using the output, and outputting the difference component as image information of the target object from the input frame memory 520 through the output device 540, it is possible to configure a device that outputs an image obtained by cutting out only the target object. A high compression rate can be achieved by encoding and transmitting the output result using a conventional image encoding device or the like.

In a case where image information input as a color image is targeted, arithmetic processing using the color correlation of the image can be performed. For example, in a human face image, the entire face has a flesh color, and eyes, lips, and the like have a specific color distribution along with the shape. By accumulating these color correlations as learning rules, the arithmetic processing can be performed. I can do it.

FIG. 12 shows an application example of the present invention, and a device for transmitting a face image of a person input by an image input device 200 such as a TV camera from a transmission side to a reception side via a transmission path 210, or an optical file. 1 shows a configuration of an apparatus for storing data in a database or the like.

Both the transmitting side and the receiving side commonly have information on face images, such as standard eye shapes, face outlines, mouth shapes, etc., as a face image database. Then, the image analysis device 220 uses the contents of the face image database 230 to output the eye position, size, shape difference, and the like of the input face image as a feature amount. On the other hand, the image synthesizing device 240
In step (1), the image is synthesized by means of the computer graphics 250 or the like using the input feature amount and the contents of the face image database 250 and output to the display device 260. As a result, a reproduced image can be obtained by transmitting or storing an extremely small amount of information as compared with the input image.

In such a device for compressing image information by analyzing and synthesizing an image, it is important to accurately compare the image information with the contents of the face image database, and for this purpose, normalization of the input image is essential.

In the above-mentioned application example, the present invention has a great effect in that the position and size of the input image are normalized with high accuracy.

The method of realizing a computer with a neural network structure is as follows.
Although there are various methods using software, hardware, and a combination of both, the present invention does not depend on the configuration.

〔The invention's effect〕

As described above, according to the present invention, a computer having a neural network structure is used to sequentially execute an arithmetic process from the entire image to a local portion, thereby achieving a highly accurate arithmetic result based on a learning rule. Can be obtained.

In addition, a plurality of learning rules, a logical determination means of the operation result,
Arithmetic processing combining knowledge databases and the like can be executed in order, and for example, there is an effect that image information can be efficiently compressed.

Further, by preparing a plurality of image frames, there is also an effect that the detection of the motion component of the moving image information can be executed with high accuracy.

[Brief description of the drawings]

FIG. 1 is a block diagram showing the overall configuration of the image processing apparatus of the present invention, FIG. 2 is a circuit diagram of a sample of a neural network structure, FIG. 3 is a detailed diagram of an image frame memory and a sample transfer section,
FIG. 4 is a flowchart showing an example of a calculation procedure using the present invention, FIG. 5 is a diagram showing an example of an operation based on a learning rule, FIG. 6 is a diagram showing an example of an operation based on a knowledge database, FIG.
FIGS. 8 and 9 show examples of operations based on other learning rules, FIG. 9 shows a method of storing learning rules, FIG. 10 shows a method of processing a moving image, and FIG. 11 shows a target object. FIG. 12 is a block diagram showing an application example of the present invention. 110: image frame memory, 120: sample transfer unit, 130: computer with neuronet structure, 140: learning rule storage unit, 150: scheduler.

──────────────────────────────────────────────────続 き Continuation of the front page (56) References JP-A-60-238984 (JP, A) JP-A-60-159972 (JP, A) JP-A-61-143889 (JP, A) JP-A-1- 116898 (JP, A) (58) Fields investigated (Int. Cl. ⁶ , DB name) G06T 7/00 G06F 15/18 JICST

Claims (5)

(57) [Claims] 1. An image frame storage means for storing at least one image frame; image data of an arbitrary size is sampled and transferred from an arbitrary position to an arbitrary image frame in the image frame storage means. Learning rule storing means for storing a plurality of types of learning rules for normalizing the position and size of a target image; and a neural network structure comprising an input layer, an intermediate layer, and an output layer; Calculating means for performing stepwise calculation processing on the image data from the sample transfer means based on a learning rule; each time the calculation means performs stepwise calculation processing, the calculation result is taken in; Selecting means for setting sampling conditions and sequentially selecting learning rules from the learning rule accumulating means. 2. An image frame accumulating means for accumulating at least one image frame; and sampling and transmitting image data of an arbitrary size from an arbitrary position to an arbitrary image frame in the image frame accumulating means. A sample transfer means for performing; a neural network structure consisting of an input layer, an intermediate layer, and an output layer; a plurality of learning rules for normalizing the position and size of the target image are taken in from another computer; Calculating means for performing stepwise calculation processing on the image data from the sample transfer means based on a learning rule; each time the calculation means performs stepwise calculation processing, the calculation result is taken in and sampled by the sample transfer means. Selecting means for setting conditions and sequentially selecting learning rules from the other computer. 3. An image processing apparatus according to claim 1, wherein said sample transfer means generates an address for designating a starting point of a sampling start and a sampling interval for an arbitrary image frame in said image frame storage means. An image processing apparatus comprising: a transfer unit configured to transfer image data of the designated sampling pixel area to the arithmetic unit. 4. A television camera control device for controlling the orientation of a television camera using the image processing device according to claim 1, 2 or 3 so that the target image is located at the center of the screen. 5. A high-efficiency coding apparatus for extracting a specific area of a target image using the image processing apparatus according to claim 1, 2 or 3.