JPH04264985A - Image recognition system - Google Patents

Abstract

PURPOSE:To recognize a subject consisting of the whole and its constitutional part such as a face at high speed and with high accuracy by recognizing the subject by inputting the feature part of the subject to an individual neural network. CONSTITUTION:The face of a person 1 to be inspected is image-picked up by directing a telecamera 2, and the image of the person 1 to be inspected is displayed on a display monitoring device 3. The person 1 to be inspected freezes the image and sends an instruction for input definition by a key operation from a keyboard input device 4, and also, announcing display whether or not the posture of the face of the person 1 to be inspected is correct is performed by, for example, a tone. Image data is extracted by a mask which extracts the whole and part of the subject, and it is supplied to the neural network at every extracted image data, thereby, learning and recognition are performed. Also, parallel processing is performed by constructing each neural network with different processors.

Description

[Detailed description of the invention]

0001

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image recognition system that can be applied to a security management system such as automatic recognition when withdrawing money from a bank and automatic entry/exit management.

0002

[Background Art] Conventionally, when performing image recognition on a human face, (1) a method of edge detection, (2) a method of pattern matching, (3) a method of integral transformation, (
4) A method using neural networks is known. Among these, (4) the method using neural networks has attracted attention in recent years as a method to improve recognition accuracy. September 26, 1990).

0003

[Problems to be Solved by the Invention] However, when using a neural network, the amount of data input to the input layer increases, and as the number of patterns to be recognized increases, the scale of the neural network increases. learning time becomes longer due to application learning,
There are problems that make it impractical. Furthermore, in the past, individual identification was performed using the captured entire image of the face, which caused the problem that accurate recognition could not be performed if the beard, eyes, or hairstyle changed. Furthermore, even if image data is obtained by capturing images of the same subject (person's face), if the pose of the image is different, correct recognition may not be possible, and in some cases it may be determined that the subject is a different subject.

[0004] Therefore, the present invention provides an image recognition system that can shorten the learning time, accurately recognize even if the beard, eyes, or hairstyle changes, and that can capture images with the correct posture. The purpose is to provide

[0005]

[Means for Solving the Problems] An image recognition system according to the present invention includes an image data input means for inputting image data of a subject, and extracts image data of the entire subject and a predetermined part from the input image data. and a learning/recognition means comprising a neural network that performs learning/recognition using the extracted image data.

Preferably, the image recognition system according to the present invention includes a posture information detection means for obtaining posture information of the subject based on input image data, and an announcement regarding the posture of the subject based on the detection result of the posture information detection means. Announcement occurrence display means for displaying the occurrence of an announcement.

The image recognition system according to the present invention includes a plurality of individual judgment neural networks and one comprehensive judgment neural network, and provides extracted image data of the whole and a predetermined part to the individual judgment neural network. The output is given to the comprehensive judgment neural network for learning and recognition.

[0008]

[Operation] Since the present invention is constructed as described above, not only recognition that captures the entire object but also recognition that captures a portion of the object is performed, thereby improving recognition accuracy. Further, distributed processing and parallel processing can be performed, and processing speed can be increased. Furthermore, an announcement regarding the posture is generated and displayed, and based on this, the posture of the subject is corrected so that the image can be captured in a fixed posture.

[0009]

DESCRIPTION OF THE PREFERRED EMBODIMENTS An embodiment of the present invention will be described below with reference to the accompanying drawings.

FIG. 1 shows an image recognition system according to an embodiment of the present invention. A television camera 2 is directed toward the face of the subject 1 and an image is taken. An image of the face of the subject 1 is displayed on the display monitor device 3. The subject 1 sends instructions to freeze the image and confirm input by operating keys from the keyboard input device 4. Further, the speaker 5 makes an announcement display, for example, using a tone, as to whether or not the posture of the face of the subject 1 is correct.

FIGS. 2 and 3 show a subject 1 and a television camera 2.
, a more specific positional relationship between the display monitor device 3 and the keyboard input device 4 is shown. In this embodiment, a mirror 6 is placed on the display monitor device 3, the face of the subject 1 is imaged by the television camera 2 through the mirror 6, and the direction of movement of the image on the screen of the display monitor device 3 ( left and right) and subject 1
The directions of movement are made to match. For example, the subject 1 sits on a chair, faces the display monitor device 3, and buries his or her head in a head fixture 7 extending from the backrest of the chair.

As shown in FIG. 1, an image signal obtained by imaging with a television camera 2 is processed by an image processing recognition device 8.
The image data is sent to the input/output control section 9 of the computer, digitized, and stored in the image memory 10 as image data. The image data in the image memory 10 is read out by the display control section 11 and given to the input/output control section 9 where it is converted into an analog image signal and sent to the display monitor device 3. The display control unit 11
An image is stored in the image memory 10 so as to generate a vertical input auxiliary line V at the vertical center of the screen of the display monitor device 3 and a horizontal input auxiliary line H at about 1/3 of the height from the top of the screen. Write the data. In addition, a slightly shorter left eye position input auxiliary line L is generated in the vertical direction at a predetermined position to the left of the vertical input auxiliary line V (a position that is half the distance between the eyes of an average person). Corresponding image data is written into the image memory 10. FIG. 4 shows an image in which the image data obtained by the television camera 2 together with the image data corresponding to each of the above-mentioned auxiliary lines are read out from the image memory 10, converted into an image signal, and displayed on the surface of the display monitor device 3. 16 is shown.

The subject 1 buries his or her head in the head fixture 7, looks in the direction of the display monitor 3, and operates the freeze button 17 (FIG. 5) of the keyboard input device 4 to turn on the television camera 2.
start imaging. As shown in FIG. 4, an image of the face of the subject 1 is displayed on the screen of the display monitor device 3 along with each input auxiliary line. Therefore, the subject 1 performs input by moving his head vertically and horizontally so that the center of gravity of both eyes is on the lateral direction input auxiliary line H and the left eye position input auxiliary line L passes through the center of gravity of the left eye. At this time, if the image on the display monitor device 3 is small and it is unclear whether the facial posture is at a predetermined position with respect to the above-mentioned auxiliary line, by operating the enlarge key 18 of the keyboard input device 4, As shown in FIG. 5, the image of both eyes can be enlarged and displayed.

Specifically, the input/output control section 9 and the display control section 11 are composed of a processor and the like, and display images according to a program shown in a flowchart shown in FIG. That is, the input/output control unit 9 detects whether there is an input to start imaging by operating the freeze button 17 from the keyboard input device 14 (S21), and if the input is received, it is captured and stored in the image memory as image data. 10 (S22). The display control section 11 reads out image data including each auxiliary line from the image memory 10, supplies it to the input/output control section 9, and sends it out as an image signal to display the image (S23). Next, it is detected whether an enlargement request is made by operating the enlargement key 18 (S24). When an enlargement request is made, the image data of the areas preset for both eyes in the image memory 10 (areas 19L and 19R in FIG. 5) is read out, and the image data is enlarged 4 times vertically and horizontally (area ratio).
The image data is enlarged, superimposed on the lower part of the original image data, and sent to the input/output control unit 9 for enlarged display (S25). The enlarged left and right eye areas 20L and 20R are shown in Figure 5.
is shown. The input control unit 9 detects a request to end the enlarged display by operating the enlarge key 18 of the keyboard input device 4 again (S26), or detects a confirmation (end) input by operating the freeze button 17 again (S27). When a request to end the enlarged display is made, normal display is performed (S29). On the other hand, when a confirmation input is made, the image data is confirmed and stored in a predetermined area in the image data memory 10 (or an external memory (not shown)) (S28). In the above, the enlarged image is superimposed and displayed, but only the enlarged image may be displayed without superimposing.

In the image recognition system according to a specific example of the present embodiment, an announcement sending unit 1 is provided for generating and displaying an announcement regarding the posture of the subject in the image processing recognition device 8.
3 are provided. By operating the freeze button 17 of the keyboard input device 4, the input/output control unit 9
An instruction to extract a predetermined area of both eyes in the image memory 10 is given to the image memory 10. Based on this, the extraction unit 12 extracts the image memory 1
The image data of each line from line Y+ to line Y- shown in FIG. The announcement sending section 13 includes a buffer memory 3.
0, a processor, a tone generator, etc. When the image data is stored in the buffer memory 30, the announcement sending unit 13 creates an X profile and a Y profile as shown in FIG. That is, regarding the X profile, an X profile is created for each line from line Y+ to line Y- of the buffer memory 30, using the brightness as shown below the buffer memory 30 as a parameter. Regarding the X profile, two minimum values (Min) appear in each line, corresponding to the pupils of the left and right eyes. If the addresses of these minimum values do not match for each line, the facial posture is incorrect. On the other hand, regarding the Y profile, the X of the left eye area L of the buffer memory 30 is
A Y profile with luminance as a parameter is created from address L- to X address L+, and from X address R- to X address R of right eye area R of the buffer memory 30.
For up to +, create a Y profile with brightness as a parameter. For these Y profiles, see Figure 7.
It is shown on the right side of the buffer memory 30. Regarding the Y profile, minimum values (Min) corresponding to the pupils appear in the left eye region L and right eye region R, respectively. If the Y address of this minimum value does not match the Y profile of each X address, the facial posture is incorrect.

When it is detected that the posture is incorrect as a result of profile creation as described above, the announcement sending unit 13 generates and sends out short intermittent tone data, and the input/output control unit 9 converts the tone data into a tone signal. For example, pi,
A tone like beep, beep... is emitted from speaker 5. Hearing this tone, subject 1 can move his head to create the correct facial posture. If the addresses of the minimum values match in each profile, the face posture is correct, so the announcement sending unit 13 generates and sends out long continuous tone data, which is converted into a tone signal by the input/output control unit 9, and is output as a tone such as a beep. is generated from the speaker 5. Upon hearing this, the victim examiner 1 operates the freeze button 17 of the keyboard input/output device 4 again to confirm the input as described in S27 of FIG.

When a key operation confirming the input is performed, the input/output control section 9 gives an extraction instruction to the extraction section 12 for the entire face and each part. As shown in FIG. 8, the extraction unit 12 extracts a whole mask 31 and an eyebrow mask 32 having a predetermined size and position from the face image in the image memory 10.
, comprising an eye mask 33, a nose mask 34, and a mouth mask 35,
Image data is read (extracted) from the area in the image memory 10 corresponding to this mask. The extraction unit 12 provides the extracted image data to the learning/recognition unit 14. Extraction part 12
is equipped with a function to create a profile as explained using FIG. 7. Prior to the above extraction, a profile is created and a profile is created for the area of both eyes of the face image data that has been input as a final input. Create. If the facial posture is correct as a result of creation, the image data is extracted as is, but if the creation is incorrect, the tilt of the image data is corrected as follows.

That is, the extraction section 12 operates according to a program shown in a flowchart shown in FIG. When the Y address is Y0 (FIG. 7), it is determined whether the minimum value X address corresponding to the left eye matches the X address of the left eye position input auxiliary line L (S36). If there is a discrepancy, subject 1
The face image data of is moved by the necessary amount in the X direction and the Y direction (including whether it is positive or negative) (S37). Next, each X of the minimum value of the left eye profile from Y+ to Y-
It is determined whether the address matches the X address of the left eye position input auxiliary line L (S38). If they do not match, the image data is rotated around the intersection of the lateral direction input auxiliary line H and the left eye position input auxiliary line L (S39). Next, it is determined whether the Y address of the minimum value of the Y profile matches the Y address of the horizontal input auxiliary line H (S40). If they do not match, the image data is rotated around the intersection of the lateral input auxiliary line H and the left eye input auxiliary line L (S41). When moving image data in this manner, a technique such as affine transformation may be used, for example.

The image data obtained by the above tilt correction is extracted using masks for the whole and each part as shown in FIG. 8, and the whole pattern (image data), eyebrow pattern, eye pattern, nose pattern, mouth A pattern is obtained and sent to the learning/recognition section 14. Note that the human face has various characteristics, and although it is not necessarily possible to extract each part of the face appropriately by extraction using a mask, this itself represents the characteristics of each person.

In the above explanation, the image processing recognition device 8 has been explained separately for each part, but except for the learning recognition part 14, it is composed of one processor and necessary circuits, and is designed to perform learning on facial image data. The program shown in the flowchart shown in FIG. 10 is executed, and the recognition of facial image data is performed by the program shown in the flowchart shown in FIG.

[0021] As described above, the subjects registered in the system are imaged one after another, and input processing is performed (S4
2). The input image data is confirmed by operating the freeze button 17 again, and the image data is registered in a predetermined memory area (S43). Then, as described above, the extraction unit 12
The entire image data and a predetermined portion are extracted by (S44), and the extracted image data is sent to the learning/recognition unit 14.
(S4
5). The learning/recognition unit 14 performs learning using a backpropagation neural network (S46).
.

FIG. 11 shows an explanatory diagram of the first stage of learning by the neural network. Neural networks are prepared in numbers corresponding to the image data of the whole image, eyebrows, eyes, nose, and mouth. One network consists of three layers: input layer, hidden layer, and output layer, and the output layer neurons are:
It has the number of images to be registered (for the number of people). The number of input layer neurons is equal to the number of input images, and the number of intermediate layer neurons required for learning is prepared. However, when similar images are input, the number of hidden layer neurons is increased because the convergence time will be long if there are few hidden layer neurons.

FIG. 12 shows a flowchart of the learning process by the neural network described above. Here, a registered image is input (S49), the size of the neural network is determined and initialized (S50), extraction processing for each part is performed by the extraction unit 12 (S51), and the extracted image data Learning is performed by each neural network (neuro) by
S52-S55). Then, it is determined whether a predetermined convergence condition has been met, and learning is performed in each neural network (FIG. 11).

[0024] In this way, when the first stage of learning is completed, the process moves to the second stage of learning. Figure 13 shows the second part of learning.
Steps of processing are shown. Here, the whole face, eyebrows, nose,
The recognition results obtained by the mouth individual judgment neural network are given to the input layer neurons of the comprehensive judgment neural network 60, and learning is performed until convergence is achieved according to predetermined convergence conditions (FIG. 12, S56, S57). The resulting weights (W, θV, σ) learned in this way are output,
The information is held in the learning/recognition unit 14 (S58). Similarly, the flowchart in FIG. 10 showing the operation of the image processing recognition device 8 also shows that a determination is made (S47) and the weight is held (S48).

[0025] In the image recognition system in which learning is performed as described above, for example, the entrance/exit management system 15 (FIG. 1)
In order to give recognition results to the system, it enters the actual operating state. That is, as shown in FIG.
The weights that were output and held in step S58 of Step 2 are set (S61), and the image data of the face of subject 1 that actually arrived is input in the same manner as in the case of registration during learning ( S62). The extraction unit 12 extracts the entire body and each part (S63), and the extracted image data is input to a neural network for individual determination (S63).
64). FIG. 15 shows the operation of each part in the image processing recognition device 8. In other words, input processing by each section (S67) until the freeze button 17 is operated again, followed by extraction processing of the whole and each part by the extraction section 12 (S68), and in steps S69 to S73, Recognition is performed by an individual judgment neural network. Then, the output of the neural network for individual determination is input to the neural network for comprehensive determination 60 (FIG. 14, S65), and the person is identified (S6
6). That is, in the learning/recognition unit 14, the comprehensive judgment neural network makes a judgment using the recognition results of the individual judgment neural network (FIG. 15, S74),
The recognition result is output to the room entry/exit management system 15 (S7
5). In this manner, the entry/exit management system 15 controls the opening and closing of a gate, for example, based on the given recognition result.

The learning/recognition unit 14 using the neural network of this embodiment performs parallel processing to speed up processing. In FIG. 16, a parallel processing system is described. With this system, individual judgment neural network 8
The whole face image data, eyebrow image data, eye image data, nose image data, and mouth image data extracted by the extraction unit 12 are given to each of 1 to 85, and these neural networks perform learning and recognition in parallel. The learning and recognition results are sent to the comprehensive judgment neural network 60, where learning and recognition are performed and the final recognition results are output.

The above parallel processing system can be realized by connecting processors in various ways, but two examples will be explained here. FIG. 17 shows that an image input processor 86 processes and digitizes the input from the television camera 2, performs extraction, and then processes the input through a processor 87 that forms a neural network for eyebrow recognition and a processor 89 that forms a neural network for nose recognition. and to a processor 88 that constitutes a neural network for eye recognition, a processor 90 that constitutes a neural network for mouth recognition, and a processor 91 that constitutes a neural network for whole face recognition, which are connected as shown in the figure. This shows a system that transfers corresponding image data and performs learning and recognition. The learning/recognition results are given to the processor 92 that constitutes the comprehensive judgment neural network via the processors 88 and 91, and the final results are sent to the room entry/exit control system 15, for example.

FIG. 18 shows a parallel processing system in which processors 86 to 92 are connected by a common bus 93. In this embodiment, the input image input processor 8
The image data digitized and extracted in step 7 is transferred to processors 87 to 91 that recognize the corresponding region.
The results of learning and recognition by these processors are transferred to a processor 92 that constitutes a comprehensive judgment neural network to obtain a final result.

[0029] If the neural network using the above processor is able to perform recognition as a result of the determination, it can be updated by re-learning by applying the back-propagation learning rule using the latest input image data. ,
Therefore, even if a person's face changes due to beard, glasses, hairstyle, etc., it can be handled.

[0030]

Effects of the Invention As described in detail above, according to the present invention, the characteristic parts of the subject are input into individual neural networks for recognition, so even if the whole part is not similar, recognition can be achieved by matching the individual parts. Accuracy can be increased. Furthermore, a neural network that recognizes the whole and parts can be constructed using separate processors, and parallel processing can be performed to increase speed.

[Brief explanation of the drawing]

FIG. 1 is a block diagram of an image recognition system according to an embodiment of the present invention.

FIG. 2 is a plan view of an input/output system of an image recognition system according to an embodiment of the present invention.

FIG. 3 is a side view of an input/output system of an image recognition system according to an embodiment of the present invention.

FIG. 4 is a diagram showing a display example of a monitor screen.

FIG. 5 is a diagram illustrating a display example of a monitor screen with an enlarged eye portion.

FIG. 6 is a flowchart for explaining the operation of the embodiment.

FIG. 7 is a diagram for explaining profile creation.

FIG. 8 is a diagram for explaining feature part extraction.

FIG. 9 is a flowchart for explaining the operation of the embodiment.

FIG. 10 is a flowchart for explaining the operation of the embodiment.

FIG. 11 is a diagram showing the first stage of learning.

FIG. 12 is a flowchart for explaining operations during learning.

FIG. 13 is a diagram showing the second stage of learning.

FIG. 14 is a flowchart for explaining operations during recognition.

FIG. 15 is a flowchart for explaining operations during recognition.

FIG. 16 is a diagram for explaining parallel processing by a neural network.

FIG. 17 is a block diagram of a configuration example of a parallel processing system.

FIG. 18 is a block diagram of a configuration example of a parallel processing system.

[Explanation of symbols]

1... Subject 2...TV camera 3...Display monitor device 4...Keyboard input device 5...Speaker 6...Mirror 7...Head fixture 8...Image processing recognition device 9...Input/output control section 10...Image memory 11...Display control section 12...Extraction part 13...Announcement sending unit 14...Learning/recognition section 60...Neural network 81-85...Neural network

Claims (6)

[Claims] 1. Image data input means for inputting image data of a subject; extraction means for extracting image data of the entire subject and a predetermined portion of the subject from the image data input by the image data input means; An image recognition system characterized by comprising a learning/recognition means comprising a neural network that performs learning/recognition using image data extracted by the extraction means. 2. The learning/recognition means includes a plurality of individual judgment neural networks and one comprehensive judgment neural network, and provides extracted image data of the whole and a predetermined portion to each of the individual judgment neural networks. 2. The image recognition system according to claim 1, wherein the output is given to the comprehensive judgment neural network to perform learning and recognition. 3. The plurality of individual judgment neural network systems and the comprehensive judgment neural network system are each configured by a different processor,
Claim 2, wherein each processor performs parallel processing.
Image recognition system described. 4. Imaging means for taking an image of a subject; image data creation means for creating image data from an image signal obtained by the imaging means; 3. The camera according to claim 1, further comprising: posture information detection means for obtaining posture information of the subject; and announcement generation and display means for generating and displaying an announcement regarding the posture of the subject based on the detection result of the posture information detection means. The image processing recognition system according to any one of the items. 5. Posture information detection means for obtaining posture information of a subject based on input image data, and extraction when extracting the entire subject and a predetermined portion of the subject from the image data based on the detection result by the posture information detection means. 5. The image recognition system according to claim 1, further comprising extraction means for extracting image data by changing a region. 6. An imaging means for taking an image of a subject; a monitor display means for displaying on a monitor the subject imaged by the imaging means; controlling the monitor display based on an image signal obtained by the imaging means; 6. A display control means for enlarging and displaying a part of the image, and adding and displaying a baseline serving as a reference for the posture of the subject within the enlarged image. The image recognition system according to item 1.