JP3617937B2 - Image monitoring method and image monitoring apparatus - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image monitoring apparatus that can be used for, for example, an interphone with a TV, video mail, or a videophone, and more particularly to an image monitoring apparatus for monitoring a subject such as a person from a moving image including a person.
[0002]
[Prior art]
A TV interphone installed at the entrance of a condominium or at the entrance of a home can check the state of visitors on a monitor, but it is desirable to record video and audio while you are away like an answering machine. . In order to realize such a function, if a VTR device or the like is used to save an image as a moving image, the device becomes large and difficult to handle, and if a storage medium such as a hard disk is used, the storage capacity Because of this problem, it was difficult to reduce the size and price.
[0003]
On the other hand, in order to solve these problems, in the case of saving as a still image, an image that can identify a person is not always stored, and may not be identified when viewed later. Sex exists.
[0004]
Moreover, even if there is an image that can be identified, it is not always easy to identify a collection other than an acquaintance, a delivery person or a suspicious person. In addition, even for acquaintances, it seems likely that people will forget their names with the aging of the population.
[0005]
Recently, with the reduction in the price of personal computers with video cameras, video mail and videophones using personal computers can be easily realized at home. Thus, when moving images are used on a personal computer on a daily basis, there is a limit to storage devices such as hard disks, and a new problem arises that it takes time to read and search data once stored.
[0006]
[Problems to be solved by the invention]
Therefore, the present invention has been made in view of the above problems, and an image monitoring method and method that can easily and reliably recognize a subject such as a person extracted from a moving image or a still image according to the convenience and needs of each user. An object is to provide an image monitoring apparatus using the same.
[0007]
[Means for Solving the Problems]
The image monitoring method of the present invention generates a dictionary in which feature amounts for each category are registered for recognizing a subject extracted from an input image, and includes a feature amount of the subject extracted from the input image By comparing the feature amount of each category registered in the dictionary to recognize the category of the subject, presenting a recognition result, and updating the dictionary based on the feature amount of the subject, In addition, recognition of a subject such as a person extracted from a still image can be easily and reliably performed according to the convenience and needs of each user.
[0008]
The image monitoring apparatus according to the present invention generates a dictionary in which feature amounts for each category are registered for recognizing a subject extracted from an input image, and the feature amount of the subject extracted from the input image Compare the feature quantity of each category registered in the dictionary to recognize the subject category, present the recognition result, update the dictionary based on the feature quantity of the subject, and input voice message Is stored in the storage means in association with the recognition result or the dictionary, so that a subject such as a person extracted from a moving image or a still image can be easily and reliably recognized according to the convenience and needs of each user. This makes it possible to respond more intelligently according to user needs.
[0009]
Preferably, among the plurality of input time-series images, only an image in which a subject whose feature quantity is most similar to any of the categories registered in the dictionary is stored in the storage unit. As a result, only those suitable for recognizing a subject such as a person can be selected and stored from moving images and still images, so that the storage capacity can be reduced.
[0010]
The image monitoring apparatus of the present invention includes a dictionary generating means for generating a dictionary in which feature quantities for each category are registered for recognizing a subject extracted from an input image;
Recognizing means for recognizing the category of the subject by comparing the feature amount of the subject extracted from the input image with the feature amount of each category registered in the dictionary;
Presenting means for presenting the recognition result by the recognizing means;
Updating means for updating the dictionary based on the feature amount of the subject;
Therefore, it is possible to easily and reliably recognize a subject such as a person extracted from a moving image or a still image according to the convenience and needs of each user.
[0011]
Preferably, the image processing apparatus further includes storage means for storing only an image in which a subject whose feature quantity is most similar to any one of the categories registered in the dictionary is extracted from a plurality of input time-series images. As a result, only those suitable for recognizing a subject such as a person can be selected and stored from moving images and still images, so that the storage capacity can be reduced, and the size and cost of the device can be reduced accordingly.
[0012]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
[0013]
(First embodiment)
In the first embodiment, a message or image of a visitor is recorded while the user is away with an interphone with a TV camera installed at an apartment or home entrance, or belongs to a pre-registered category even at home. A visitor monitoring device that enables smooth response by displaying whether or not will be described.
[0014]
FIG. 1 shows a configuration example of a visitor monitoring apparatus according to the present embodiment. An image input unit 1, a voice input unit 2, a person detection unit 3, an information extraction unit 4, a storage unit 5, and a monitor unit 6 are shown. And a user information input unit 7.
[0015]
The image input unit 1 is for inputting an image acquired by a surveillance camera such as a video camera. The voice input unit 2 is for inputting voice acquired from, for example, an interphone microphone.
[0016]
The person detection unit 3 is, for example, a person such as a bell or a buzzer that voluntarily informs the visit, or a person that detects a person by detecting a change from an image acquired from a camera or the like, a knock or a call There is no particular limitation such as one that detects a physical operation.
[0017]
The information extraction unit 4 extracts information necessary for recording or recognition from the input image or sound.
[0018]
The storage unit 5 includes a recording medium such as a magnetic tape, a magnetic disk, a magneto-optical disk, an optical disk, and a semiconductor memory that stores dictionary information, extracted image information, recorded audio information, and the like.
[0019]
The monitor unit 6 is composed of a display device, a speaker, and the like, and is for displaying various extracted information and signals directly input from a camera and a microphone.
[0020]
The user information input unit 7 inputs or corrects the name and category of a visitor, and searches and organizes recorded information.
[0021]
FIG. 2 shows a configuration example of the information extraction unit 4, which includes a face detection unit 8, a dictionary generation unit 9, a face recognition unit 10, and a recording image determination unit 11.
[0022]
The face detection unit 8 detects a human face area from the input image. For example, using a face area template created in advance, an area having the highest correlation value with the template is cut out from the image.
[0023]
The dictionary generation unit 9 is configured to create a dictionary for face recognition from a plurality of cut face area images. For example, when face recognition is performed using a conventional subspace method, feature points such as pupils and nostril feature points are detected from the face area image, and the face is detected using the detected feature points. Perform region normalization. The normalized image information is converted into feature vectors in the feature space, a partial space is generated from these feature vectors, and this is used as a dictionary for face recognition. Then, the partial space is classified into categories and registered in the storage unit 5 as a dictionary for face recognition.
[0024]
However, this operation (new registration of the dictionary) is a case where the visitor does not belong to any of the categories registered in advance, and in this case, it is desirable to update the dictionary of the category. When this dictionary update operation is performed, it is desirable to save not only a partial space but also statistical information such as eigenvalues and correlation matrices of the partial space at the same time. When updating the dictionary, the statistical information of the partial space may be updated based on the feature amount extracted from the visitor's face area image.
[0025]
The face recognition unit 10 calculates the similarity between the input image series and the dictionary of each category as to whether it belongs to a pre-registered category (for example, a conventional method such as obtaining an inner product of feature vectors in each partial space). Good). If it does not belong to any registered category, the dictionary described above is created.
[0026]
Each time a moving image is input from the image input unit 1, the recording image determination unit 11 selects an image that most represents the characteristics of the visitor from the input image series and stores it in the storage unit 5. Specifically, the selection method selects an image having the highest similarity to the visitor's dictionary from a plurality of image frames of the input image series. Storing the input video as it is in the storage unit 5 requires a large storage capacity, and it is troublesome for the user to select which image to leave when, for example, only one of them is left. It is. Also, if you choose mechanically, there is a high possibility that your face will not be captured correctly. However, even if an image in the input image series does not belong to any of the categories in the dictionary (similarity is too low to determine that it belongs to any category), the most similar of them is If only one expensive sheet is selected and stored in the storage unit 5, the above problem will be solved.
[0027]
Next, an example of the operation of the visitor response apparatus in FIG. 1 will be described assuming that a visitor arrives while he / she is away.
[0028]
The user sets in advance that the user is away. At this time, the user may be at home. This operation may be similar to a normal answering machine. When a visitor comes in front of the camera set at the front door of the user's house and rings a doorbell or enters a room number in an apartment house, the action is triggered by the image and microphone input from the camera. The input voice is started to be captured by the image input unit 1 and the voice input unit 2, respectively, and displayed on the monitor unit 6 installed indoors.
[0029]
On the other hand, in the information extraction unit 4, first, a human face area is detected from the image input by the face detection unit 8, the feature vector extracted through the dictionary generation unit 9 and the face recognition stored in the storage unit 5. The face recognition unit 10 recognizes the visitor's face from the image input from the image input unit 1 and determines, for example, the visitor's category. Then, the category name as the recognition result is displayed on the monitor unit 6 together with the image captured from the camera.
[0030]
Category names are registered in advance and are not limited to individual names. For example, newspaper collection, home delivery, postal delivery, etc. are likely to have the same person each time, and their personal names are meaningless, so category names that are not personal names, for example, “newspaper collection” “home delivery” “postal” in this case It is also possible to classify these recognition results by category names such as “delivery”. Similarly, for a category that does not need to be classified, the category name “suspicious person” may be used.
[0031]
If the category does not belong to a pre-registered category, a dictionary is generated from the input image series as described above. If it belongs, the existing dictionary is updated as described above. However, if the classified category is incorrect, a dictionary only for the visitor may be created and saved without deleting the original dictionary for the user so that the user can correct it later. desirable.
[0032]
The recording image determination unit 11 selects one image having the highest similarity to the dictionary created in the input image series from the image input unit 1. Only one image of the category recognized image and the category unrecognized image is stored. The dictionary is generated from the moving image, but only one image is left to save storage capacity. In order to further reduce the storage capacity, only an image obtained by cutting out only the face area may be left.
[0033]
The voice message of the visitor input from the voice input unit 1 is recorded in the storage unit 5 together with the face image and the recognition result.
[0034]
By recognizing a keyword by voice recognition using a keyword spot and leaving the result, a category name for a new dictionary registration can be facilitated later. In order to simplify it, it is also effective to give a prompt such as “Please give your name” as voice or text information.
[0035]
The user checks the message when he / she likes to go home. At that time, the recognized category is corrected, a new category name is input, and unnecessary messages are deleted.
[0036]
Even if the user is at home but wants to respond after confirming the visitor, the image is displayed on the monitor and the recognition result is also displayed, so it is possible to respond only to the visitor who wants to respond Become. At this time, when there is a visitor who does not want to meet the same, it is convenient because it can be handled by only the category name without having to remember the face.
[0037]
(Second Embodiment)
In the second embodiment, an image sent from the other party through a video mail (for example, it is generally possible to transmit a moving image using a predetermined communication means) or a video conference by a personal computer with a camera or the like. It is used to reduce the storage capacity of video mail, to confirm the other party by face recognition, and to create a face recognition dictionary.
[0038]
The visitor monitoring apparatus described in the first embodiment is mainly used for security. In the second embodiment, for example, the visitor monitoring apparatus is also used for easily checking the image contents with a personal computer or the like. An image monitoring apparatus that can be used will be described.
[0039]
FIG. 3 shows a configuration example of the image monitoring apparatus according to the second embodiment. The image input unit 12, the face detection unit 13, the dictionary generation unit 14, the face recognition unit 15, the display unit 16, and the storage unit 17 are illustrated. The recording image determination unit 18 is configured. The visitor monitoring apparatus shown in FIG. 3 may be configured on a personal computer (personal computer). That is, for example, a magnetic disk (floppy disk, hard disk, etc.), an optical disk (CD-ROM, DVD, etc.), a semiconductor memory, etc., can be used as a program that allows a computer to execute the functions of the above-described units using the hardware of a personal computer. It can also be stored in a recording medium and distributed.
[0040]
The image input unit 12 reads a moving image recorded in advance on a hard disk of a personal computer, or reads a moving image sent via a communication line such as a LAN or a telephone.
[0041]
Regarding the face detection unit 13, the dictionary generation unit 14, the face recognition unit 15, the storage unit 17, and the recording image determination unit 18, the face detection unit 8, the dictionary generation unit 9, and the face recognition unit 10 described in the first embodiment. The storage unit 5 and the recording image determination unit 11 are the same.
[0042]
Next, an actual operation example will be described by taking the case of reducing the capacity of video mail as an example.
[0043]
Video mail sent from the other party is usually stored on the hard disk of a personal computer. The image input unit 12 reads the data, extracts only the moving image portion, and the face detection unit 13 detects the face area. The face recognition unit 15 determines whether or not the detected face area belongs to a person category registered in advance. The determination result is displayed on the display device 16 such as a display together with the image.
[0044]
At the same time or according to a user instruction, the dictionary generation unit 14 creates a dictionary from the moving image. At that time, in the case of a new category, the user can newly input a category name. However, in the normal case, it is convenient to use the category name as an e-mail address. If the category already exists, the existing dictionary is updated instead of creating a new dictionary.
[0045]
The recording image determination unit 18 determines a recording image from the moving image. Specifically, as described in the first embodiment, an image having the highest similarity with the dictionary in the input image series is selected.
[0046]
Since video mail is moving image information, it needs a lot of storage capacity if it is stored as it is, but it is troublesome for the user to select which image to leave when only one piece is left, for example, If you choose mechanically, there is a high possibility that the face will not appear correctly, but this will solve the problem.
[0047]
In the first and second embodiments, the case where the monitoring target is a person has been described as an example. However, the present invention is not limited to this case, and any monitoring target may be used. .
[0048]
Furthermore, the present invention is not limited to these examples, and can be applied with various modifications.
[0049]
【The invention's effect】
As described above, according to the present invention, it is possible to easily and reliably recognize a subject such as a person extracted from a moving image or a still image according to the convenience and needs of each user.
[Brief description of the drawings]
FIG. 1 is a diagram showing a configuration example of a visitor monitoring apparatus according to a first embodiment of the present invention.
FIG. 2 is a diagram illustrating a configuration example of an information extraction unit.
FIG. 3 is a diagram showing a configuration example of an image monitoring apparatus according to a second embodiment of the present invention.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 ... Image input part 2 ... Audio | voice input part 3 ... Person detection part 4 ... Information extraction part 5 ... Memory | storage part 6 ... Monitor part 7 ... User information input part 8 ... Face detection part 9 ... Dictionary generation part 10 ... Face recognition part 11 ... Recording image determination unit 12 ... Image input unit 13 ... Face detection unit 14 ... Dictionary generation unit 15 ... Face recognition unit 16 ... Display unit 17 ... Storage unit 18 ... Recording image determination unit

Claims (4)

Image input means for inputting a plurality of time-series images;
First storage means for storing a dictionary in which feature quantities for each category are registered for recognizing a subject extracted from an input image;
Means for obtaining a feature amount of a subject extracted from a plurality of input time-series images;
Recognizing means for comparing the feature amount of the subject obtained from the plurality of images with the feature amount of each category registered in the dictionary, and recognizing the category to which the subject belongs;
When the category to which the subject belongs is recognized by the recognition unit, an image from which a subject most similar to the feature amount of the category is extracted is selected from the plurality of images, and the subject belongs to the recognition unit. Means for selecting an image in which a subject most similar to the feature amount of the subject obtained from the plurality of images is extracted from the plurality of images when the category cannot be recognized;
Second storage means for storing the selected image;
Presenting means for presenting the selected image together with its category name when the category to which the subject belongs is recognized by the recognizing means;
Means for registering a feature amount obtained from the input time-series images together with a new category name in the dictionary when the recognition means cannot recognize the category to which the subject belongs;
Means for updating the feature quantity of the dictionary corresponding to the category name with the feature quantity obtained from the input time-series images when the recognition means recognizes the category to which the subject belongs;
An image monitoring apparatus comprising: Voice input means for inputting voice messages of subjects of the plurality of input time-series images;
The image monitoring apparatus according to claim 1, wherein the voice message is stored in the second storage unit together with the selected image. First storage means for storing a dictionary in which feature amounts for each category are registered for recognizing a subject extracted from a plurality of input time-series images, and one of the plurality of images In the image monitoring method in a computer provided with the 2nd memory | storage means for memorize | storing,
A first step of obtaining a feature amount of a subject extracted from a plurality of input time-series images;
A second step of comparing the feature quantity of the subject obtained from the plurality of images with the feature quantity of each category registered in the dictionary to recognize the category to which the subject belongs;
When the category to which the subject belongs is recognized in the second step, an image from which the subject most similar to the feature amount of the category is extracted is selected from the plurality of images, and in the second step If the category to which the subject belongs cannot be recognized, a third step of selecting an image in which the subject most similar to the feature amount of the subject obtained from the plurality of images is extracted from the plurality of images;
A fourth step of storing the selected image in the second storage means;
A fifth step of presenting the selected image together with the category name when the category to which the subject belongs is recognized in the second step;
A sixth step of registering a feature amount obtained from the input time-series images together with a new category name in the dictionary when the category to which the subject belongs cannot be recognized in the second step;
When the category to which the subject belongs is recognized in the second step, the feature value of the dictionary corresponding to the category name is updated with the feature value obtained from the input time-series images. Steps,
An image monitoring method including: First storage means for storing a dictionary in which feature amounts for each category are registered for recognizing a subject extracted from a plurality of input time-series images, and one of the plurality of images A computer having second storage means for storing
A first step of obtaining a feature amount of a subject extracted from a plurality of input time-series images;
A second step of comparing the feature quantity of the subject obtained from the plurality of images with the feature quantity of each category registered in the dictionary to recognize the category to which the subject belongs;
When the category to which the subject belongs is recognized in the second step, an image from which the subject most similar to the feature amount of the category is extracted is selected from the plurality of images, and in the second step If the category to which the subject belongs cannot be recognized, a third step of selecting an image in which the subject most similar to the feature amount of the subject obtained from the plurality of images is extracted from the plurality of images;
A fourth step of storing the selected image in the second storage means;
A fifth step of presenting the selected image together with the category name when the category to which the subject belongs is recognized in the second step;
A sixth step of registering a feature amount obtained from the input time-series images together with a new category name in the dictionary when the category to which the subject belongs cannot be recognized in the second step;
When the category to which the subject belongs is recognized in the second step, the feature value of the dictionary corresponding to the category name is updated with the feature value obtained from the input time-series images. Steps,
A machine-readable recording medium having recorded thereon a program for executing processing including:

Images