JPH1196373A - Image pickup device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To surely pick up the image of a desired image pickup object by identifying the object while comparing feature information registered in an object identification dictionary with the image area of the extracted object, and displaying the image area of the object and the identified result. SOLUTION: An object area extracting part 4 extracts the image area of the object by comparing the feature information of a retrieval area on each hierarchy registered in an object area extraction dictionary 3 registering the feature information of the respective plural retrieval areas in a hierarchical implication relation while implication the image area of the object for retrieving the image area in the object inside a fetched image with the fetched image. An object identifying part 9 identifies the object by comparing an object identification dictionary 8 registering the feature information of the object with the image area of the object extracted by the object area extracting part 4. Then, a display part 10 displays the image area of the object extracted by the object area extracting part 4 and the identified result at the object identifying part 9. Thus, the image of the desired image pickup object can be surely picked up.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an imaging device for imaging a desired object.

[0002]

2. Description of the Related Art An optical or digital still camera and a television camera are used to detect and identify a desired object in a general image, and the image is analog or digital image signal. Data for presenting to the user the plurality of videos converted and stored to determine whether the user includes the desired object, and associating the storage device with the attribute (name, etc.) of the video and the object. There is an image editing device that saves a structure.

In such a video editing apparatus, an input video is digitized and presented so that a user can easily identify a visual target. However, a video including a desired target is searched from a plurality of image sequences. Since it is the user who performs the identification, the burden is large.

[0004] Further, regarding a sequence (frame) of digital images captured and stored, a break in the scene is found based on the amount of video change between frames and the magnitude and direction of the movement vector of the pixels in the frame. There is an image editing apparatus that presents only a representative frame or a small number of frames around the representative frame to the user, thereby increasing the possibility that the user can search for a desired target without viewing the entire image.

[0005] In such a video editing apparatus, since the fluctuation of the video sequence is captured and the representative image (frame) is determined and presented, there is no burden on the user to view the entire image sequence. It is necessary for a human to determine whether or not.

Further, by applying a function of having a desired image characteristic of a representative frame as a dictionary and determining whether or not the representative frame is similar to only one or several of the dictionary patterns, There is a device for identifying an object.

[0007] Such an apparatus includes means for determining whether or not there is a target in the extracted representative frame, so that the burden on the user is reduced, but the representative image sequence does not include the desired target in the first place. May also be present.

When an image in a general scene is photographed, it is often known in advance what kind of desired object is. For example, the following cases (1) to (3) correspond to such cases.

(1) Using a handy VCR or camera,
When you want to focus on a person who appears in the event of an event or an object (for example, a product) that you know in advance (such as a product), or when you want to focus on a signboard and shoot, and then read the content.

(2) A case in which a store arrival code on a moving bag is read and the delivery destination of the bag is sorted, or the delivery status of the bag is tracked by reading the serial number of the bag.

(3) Attention is paid to the numbers and marks written on vehicles such as vehicles, ships, trains, airplanes, etc., and images are taken, the contents are read, and illegal traffic detection, travel time measurement, and other functions are carried. When realizing with the device of.

In the above-mentioned cases (1) to (3), conventionally, the user takes an image of a desired object with good aim, and there is no desired image. After capturing the image, the user can extract only the desired portion from the image and determine the content, or select some characteristic such as the pixel change amount between consecutive images or the movement vector from the large amount of data brought back. In this method, a scene break is automatically detected, a desired image is selected based on the automatically detected break, and whether or not the selected image is a desired image is automatically identified.

[0013]

As described above, conventionally,
Whether or not the desired object is actually included in the captured image is ultimately determined by the user looking at the captured image, and it is possible to reliably capture the desired object. could not.

In the conventional method, a video is captured and stored, and a desired object is extracted and identified from the stored video by a portable device such as a video camera. Since the processes of performing, object extraction, and identification are performed by another device, it is necessary to collect in advance a large amount of data which is unknown whether or not a desired object is included. Furthermore, the user has to adjust the parameters of the imaging system for extracting the desired target in advance according to the situation when the target is imaged so that the target can be easily identified.

Accordingly, the present invention provides an imaging apparatus capable of reliably imaging a desired imaging target while determining whether or not the imaging target is a desired one according to the imaging state of the imaging target. The purpose is to provide.

[0016]

According to the present invention, there is provided an imaging apparatus comprising: an image capturing means for capturing an image; and an image area of the object for retrieving an image area of the object in the image. , A target area extraction dictionary in which characteristic information of each of a plurality of search areas in a hierarchical inclusion relationship is registered, and the characteristic information of the search area of each hierarchy registered in the target area extraction dictionary and the image capture A target area extracting unit that compares the image captured by the reading unit and extracts an image area of the target object; a target identification dictionary in which feature information of the target object is registered; and a target identification dictionary registered in the target identification dictionary. A target identifying unit that compares the feature information with the image area of the target object extracted by the target region extracting unit to identify the target object; and an image region of the target object extracted by the target region extracting unit. And the identification result of the object identification means is displayed. And shows means by provided with the,
It is possible to reliably image a desired imaging target.

Further, the imaging apparatus of the present invention (claim 2)
An image capturing means for capturing an image, and a hierarchical inclusion relationship including an image area of the object for searching for an image area of the object in the image according to environmental conditions at the time of capturing the image. A plurality of target area extraction dictionaries in which respective feature information of a plurality of search areas are registered, and a selection unit for selecting one of the plurality of target area extraction dictionaries based on the image captured by the capture unit And comparing the feature information of the search area of each hierarchy registered in the target area extraction dictionary selected by the selection means with the image captured by the image capturing means, to determine the image area of the object. Target region extracting means to be extracted;
The target identification dictionary in which the characteristic information of the target is registered, and the characteristic information registered in the target identification dictionary and the image area of the target extracted by the target region extracting means are compared to identify the target. Object identification means, and display means for displaying the image area of the object extracted by the object area extraction means and the identification result of the object identification means, It is possible to reliably image a desired imaging target regardless of the change.

Further, the image pickup apparatus of the present invention (claim 3)
An image capturing means for capturing an image, and a hierarchical inclusion relationship including an image area of the object for searching for an image area of the object in the image according to environmental conditions at the time of capturing the image. A plurality of target region extraction dictionaries in which feature information of each of a plurality of search regions is registered; an environment condition dictionary in which the image feature information corresponding to the environmental condition and an identifier of the target region extraction dictionary are registered; Selecting means for comparing one of the plurality of target area extraction dictionaries by comparing feature information registered in the environmental condition dictionary with the image captured by the capturing means; Comparing the feature information of the search area of each hierarchy registered in the target area extraction dictionary with the image captured by the image capturing means, and extracting an image area of the target object; Register the feature information of the object A target identification dictionary for identifying the target object by comparing feature information registered in the target identification dictionary with an image area of the target object extracted by the target region extraction unit; Display means for displaying the image area of the target object extracted by the region extracting means and the identification result by the target identifying means, and thereby ensuring the desired imaging target object regardless of a change in the imaging environment. Images can be taken.

[0019]

Embodiments of the present invention will be described below with reference to the drawings. (1) Outline of Imaging Apparatus FIG. 1 schematically illustrates an example of the configuration of an imaging apparatus according to the present embodiment. An image input unit 1 including an optical system capable of digitally controlling the aperture and focusing, and a shutter speed Unit 2 comprising a CMOS image sensor array having an electronic shutter capable of digitally controlling the image and an A / D converter, and a digital output of the imaging unit 2 which is randomly accessed and a desired target region is extracted as a target region extraction dictionary 3 Prior to this operation, a target area extracting unit 4 for detecting the image by using a plurality of images that are sequentially accessed and temporarily stored in a predetermined memory and an imaging state dictionary 5 are used. The imaging condition selection unit 6 for estimating the imaging condition of the target and selecting the target region extraction dictionary 3, and the positional relationship between the optical system and the target after the target region is extracted by the target region extraction unit 4. Geometric conversion unit 7 inversely converted into 3-dimensional geometry of the position registered in the object identification dictionary 8 a variation of the extraction region generated
And a target identification unit 9 for identifying the inside of the target region using the target identification dictionary 8 after the extraction and geometric transformation of the desired target region, and a display unit 10 for presenting the target region extraction result and the identification result to the user. Have.

A user operation unit 11 that allows a user to adjust parameters used in the image input unit 1 and the target area extraction unit 4 based on the display result of the display unit 10, and an imaging adjustment parameter dictionary that records the operation parameters 16
And an automatic imaging adjustment unit 1 for automatically performing the adjustment at another imaging opportunity based on the imaging adjustment parameter dictionary 16.
7 is provided.

Further, storage means 12 for temporarily storing a plurality of data for creating respective dictionaries of the target area extraction dictionary 3, the target identification dictionary 8, the imaging situation dictionary 5, and the imaging adjustment parameter dictionary 16. A data input / output unit 13 for outputting the data to an external device or inputting dictionary data from the external device; a target area extraction dictionary 3, a target identification dictionary 8, and an imaging condition using a plurality of input image data. A dictionary creating unit 14 for creating each dictionary of the dictionary 5;

The above components constitute a portable imaging device all housed in one housing. However, since the dictionary is created by a statistical method, it is more accurate to create the dictionary using a larger amount of image data. Therefore, the dictionary creation unit 14 may be built on a high-speed, large-capacity computer outside the imaging apparatus, send data necessary for dictionary creation through the data input / output unit 13, and create the dictionary.

Further, in the imaging apparatus, a CMOS image sensor device having substantially the same structure as the semiconductor dynamic memory is used as an imaging device in the imaging section 2, thereby enabling random access to image data.
By incorporating the target area extraction unit 4 into the same LSI as the imaging unit 2, the target area extraction processing can be parallelized and speeded up.

Hereinafter, each unit constituting the image pickup apparatus of FIG. 1 will be described in detail.

(2) Image input unit 1 When a digital signal for aperture and focusing is given, a correspondence table of lens aperture value and focusing value corresponding to the digital signal is built in, and the aperture value and focusing value as specified are automatically set. An existing image input means having a function set to the above may be used.

(3) Imager 2 An electronic shutter for controlling the time interval (shutter speed) of light incident on the image sensor and its surface by an external digital control signal, and an A / A for converting the output of the image sensor to a digital signal. And a D converter.

As the image pickup device, a CMOS image Mensa array whose output can be directly accessed at random may be used, and as the electronic shutter, a liquid crystal type shutter or the like may be used.

The A / D converter is used to convert the output of the sensor array into digital data having a fixed bit length.

(4) The target area extraction unit 4 and the target area extraction dictionary 3 When identifying what is inside the digital output image of the imaging unit 2, the target area of a desired category is included in the image at the preceding stage. For example, if it is desired to identify a specific person from the video, the area of the desired category in this case is a face area), and the area to be identified is narrowed down. This processing is performed by the target area extraction unit 4. That is,
A two-dimensional template (target area extraction dictionary 3) that roughly represents an image of a desired target candidate that has been captured in advance and input to the imaging apparatus and a digital output image of the imaging unit 2 are obtained by random access. A desired target region is extracted as a region satisfying a certain statistical criterion as a result of performing the template matching with the partial image.

Here, as a criterion for selecting a desired area by template matching, for example, while shifting the template on the output image of the imaging unit 2, the image in the area overlapping the template is regarded as a vector, and the Euclidean A method of selecting a position at which the distance, that is, the square of the sum of the squares of the pixel value difference between corresponding pixels in the template area is the minimum may be used. Further, for example, a similarity calculated by a subspace method (Reference 1: Iijima, Pattern Recognition Theory, PP119) may be used instead of the Euclidean distance. This uses a finite number of eigenvectors obtained by performing principal component analysis of the variance-covariance matrix of a plurality of sample image vectors indicating a desired target as template vectors from the dictionary, and arranging them in finite order from the largest corresponding eigenvalue. When the image in the overlapped area is viewed as a vector, the area where the sum of the inner products of the output image vector and each of the eigenvectors in the template is the maximum is defined as the desired target area.

A similarity similar to that of the subspace method relating only to the desired object as described above, but a mixed similarity (Reference Document 2: Iijima, Pattern Recognition Theory, PP122) using object dictionary data other than the desired area is calculated. Then, a region where the maximum is obtained may be extracted as a desired target region.

Since the output image of the image pickup unit 2 has undergone various geometric transformations depending on the positional relationship between the optical system and the object, the object area dictionary 3 stores a plurality of objects corresponding to one arrangement. The dictionary data (template) for the object created from the sample image will have multiple templates (multi-entries) obtained as a result of applying various geometric transformations, and as a matching result, among these multiple templates, The one that satisfies the statistical standard is selected.

Further, the target area extraction dictionary 3 does not represent only the target itself identified by the target identification unit 9 but includes a dictionary pattern that roughly represents an area geometrically including the target. One or more of them are included, and the smallest one is selected sequentially from the largest one, and the search area for the desired object in the input image is gradually narrowed.

Therefore, in order to perform the above-described hierarchical matching, the target area extraction dictionary 3 has a hierarchical structure as shown in FIG. 2 for each desired object category. There is a multi-entry dictionary of the number of geometric transformation types for each hierarchy, and a dictionary of a more detailed area of the lower hierarchy is described following the dictionary entry of the upper hierarchy, and the entry of the upper hierarchy is included in the dictionary. A coordinate relationship with the lower hierarchical area (the origin of the pattern centroid of the hierarchical level and the number of pixels as a unit) is described. Then, according to this configuration, the desired target area is limited in order from the upper level to the lower level. At this time, when the area including the desired target area in the upper level is determined, the coordinate relationship with the lower level area is referred to. Then, a region to be matched is narrowed down in the next hierarchy, and a desired target region is finally extracted.

The target area extraction processing of the target area extraction unit 4 will be described with reference to the flowchart shown in FIG.

The geometrical transformation for each layer includes, for example, rotation and size in a target screen plane, so that the following processing procedure is actually used.

The search area of the target area by matching the output image of the image pickup unit 2 is set on the entire screen, and the dictionary data of the highest level (for example, “level 1” in FIG. 2) of the target area extraction dictionary 3 is picked up. Matching with the output image of the unit 2 is performed (steps S1 and S2). At this time, matching is performed between a plurality of images created by reducing the output image of the imaging unit 2 at a plurality of resolutions and dictionary data (geometric conversion patterns) of all geometric conversion types in the highest hierarchy (step). S3 to step S4). Then, matching between dictionary data (vector) of one geometric conversion type (rotation) and an image obtained by reducing the output image of the imaging unit 2 at a plurality of resolutions was performed on dictionary data of a plurality of geometric conversion types. Find the ones whose results best meet the aforementioned statistical criteria.

An image having the best matching result (the desired area is detected) is selected, and the search area of the image (in this case, the entire screen) and the search area in the next hierarchy in the dictionary 3 are selected. The search area of the next hierarchy is acquired from the relative position (step S5). That is, the image output from the imaging unit 2 is sequentially matched with the dictionary data of each hierarchy from the upper hierarchy to the lower hierarchy, and based on the result, a smaller area where a desired target object may exist may be obtained. Narrow the search area to.

Then, the matching process between the new search area and the dictionary data of the next hierarchy is performed for a plurality of geometric transformation type dictionaries in that hierarchy, as in the above-described steps S3 to S5, and the statistical standard is set to the maximum. Find what meets you. The processes of steps S3 to S5 are repeated until the lowest hierarchical level is reached.

(5) Processing for Creating Target Area Extraction Dictionary 3 The operation for creating the target area extraction dictionary 3 in the dictionary creation section 14 will be described with reference to the flowchart in FIG.

One of a plurality of images in a situation scene
An area circumscribing the target object area to be actually identified in the image (frame) is specified via the user operation unit 11 (step S11).

Using the plurality of images in the above-mentioned situation scene, the correspondence between the pixels is obtained, and the correspondence (movement vector) between the frames of the designated area designated through the user operation unit 11 is obtained. And its representative vector U
Is obtained (step S12).

Correspondence (movement vector) Uo between frames of an area that includes the specified area in the screen
(X, y) is obtained. Here, x and y are x and y coordinates in the screen (step S13).

With respect to all the movement vectors Uo (x, y) in the area determined in step S13, Ut (x, y) having a distance from U equal to or less than a threshold value Tu is determined, and each vector is determined. A circumscribed area of coordinates (x, y) for Ut (x, y) is obtained (step S14).

The image pattern in the circumscribed area obtained in step S14 is subjected to principal component analysis (step S15).

The calculation of the movement vector in step S12 may use a method based on the correlation of small areas between frames (Reference 3: Takagi, Shimoda, "Image Analysis Handbook", PP.553.).

The threshold value Tu in step S13 determines the Euclidean distance of the difference vector between Uo (x, y) and U in the screen, and performs discriminant analysis (Ref. 4: Otsu, Kurita, Sekida, "pattern Recognition—Theory and Application— ”, Asakura Shoten, pp. 65-71, 1996.).

Further, in order to obtain the circumscribed area in step S14, the circumscribed rectangle of each coordinate of Ut (x, y) is connected area labeling (Ref. 5: Takagi, Shimoda, "Image Analysis Handbook", PP.578. ), And an initial control point is arranged in the circumscribed rectangle.
ine snake (Reference 6: R. Cipolla,
A. Blake, "the dynamic anal
ysis of apparel coats ”
in proc. 3rd int. conf. on c
output vision, pp 616-623
(1990)).

Further, the correlation of the luminance of the corresponding pixel among a plurality of frames may be used for calculating the movement vector,
Sobel-operat beforehand for each frame image
or (Reference Document 7: Takagi and Shimoda, “Image Analysis Handbook”, PP.553.), and the like, and the luminance correlation of an image (edge enhancement) obtained by applying a differential filter may be used. Further, if the desired object is composed of a specific color,
After converting each frame image into a color system that emphasizes those specific colors, the above-described edge-emphasized image may be created and its luminance correlation may be used. As an example of such a case, when a human face is extracted, a YIQ color system is used as a color system that emphasizes the skin color of the face (Ref. 8: Takagi and Shimoda, “Image Analysis Handbook”, PP.103. )) May be used.

Further, in the above description, the correspondence for each pixel is obtained. However, the above-mentioned edge-enhanced image is obtained in advance for the region including the target region, and the edge region obtained by binarizing the image with a certain threshold is used for frame-to-frame association. May be obtained to obtain the movement vector.

(6) Imaging situation selection unit 6 and imaging situation dictionary 5 The imaging situation selection unit 6 uses the imaging situation dictionary 5 to select an imaging situation of a desired target object from a plurality of target area extraction dictionaries 3 for each imaging situation. Is selected (selection by the user, automatic selection).

The imaging situation dictionary 5 describes a representative pattern for each situation and an identification number of the target area extraction dictionary 3 corresponding to the representative pattern.

First, the operation of the dictionary creating section 14 for creating the imaging situation dictionary 5 will be described with reference to the flowchart shown in FIG.

Image pickup under a certain image pickup condition is started, and several digital output images of the image pickup section 2 are stored in the storage section 12 (steps S21 to S22). A representative pattern of the situation statistically obtained for this is created (step S23). The creation of the representative pattern here is performed by using the above-described subspace method (Reference Document 1) for a plurality of frame images of a certain scene. Therefore, not only one representative pattern but the number of the above-described eigenvalues can be generated, and a certain n of them are used.

At this time, the above-described input image is used to perform the above-described target area extraction dictionary creation processing (see FIG. 4) (step S24).

The identification number of the target area extraction dictionary created in step S24 and the representative pattern under the imaging conditions obtained in the processing of steps S22 to S23 are registered in the imaging situation dictionary in association with each other, and the A situation dictionary is created (step S25).

As described above, the processing from step S21 to step S25 is performed for the number of situations that are assumed to be used, and the imaging situation dictionary 5 corresponding to the situation is created (step S26).

Next, a case will be described in which the imaging situation selection unit 6 selects the target area extraction dictionary that is optimal for the imaging situation of the desired object.

First, in the case of user selection, several digital output images of the image pickup unit 2 are stored in the storage unit 12, and
The display unit 10 displays a list of reduced images on the screen. At this time, the imaging situation dictionary 5
The inside dictionary pattern is displayed and selected using the user operation unit 11. Note that the dictionary patterns in the imaging situation dictionary 5 are representative patterns of a plurality of situation scenes as described later. Then, the imaging situation selection unit 6 reads the identification number of the target area extraction dictionary 3 corresponding to the dictionary pattern in the selected imaging situation dictionary 5.

Next, the case where the target area extraction dictionary is automatically selected by the imaging condition selection section 6 will be described with reference to the flowchart of FIG.

Immediately before the actual imaging, a plurality of input images are used and the same statistical method as in step S23 of FIG. 5 is used.
A representative pattern under the imaging condition is obtained (step S
31 to step S33).

The similarity or distance between each representative pattern in the imaging situation dictionary 5 and the representative pattern in the current imaging situation obtained in step S33 is obtained, and the similarity or maximum distance in the imaging situation dictionary 5 is determined. Is selected, and the identification number of the target area extraction dictionary 3 corresponding to the selected dictionary pattern is read (step S35). Thereafter, when the actual imaging operation is started, the target area extraction unit 4 extracts the target area from the image output from the imaging unit 2 by using the selected target area extraction dictionary 3.

The similarity between each representative pattern in the imaging situation dictionary 5 obtained in step S34 and the representative pattern in the current imaging situation may be, for example, the similarity in the subspace method (see Reference Document 1). In addition, as the distance, a certain one of the above-described imaging situation dictionary 5 and the current representative pattern is used.
The Euclidean distance between one of the representative patterns having the largest eigenvalue may be used.

(7) Correction of Object Extraction Result When the display of the desired region extraction result by the target region extraction unit 4 on the display unit 10 is not desired by the user, the user operation unit The imaging parameters of the imaging unit 2 are adjusted via the interface 11 and a desired area is selected. Next, the adjustment amount of the user at this time, the matching result at that time in the target area extraction unit 4, and the number Di of the representative pattern of the target area extraction dictionary 3 used at the time of matching.
Are temporarily stored in the storage unit 12.

The dictionary creating unit 14 creates the imaging adjustment parameter dictionary 16 as described below based on the above-mentioned contents stored in the storage unit 12. Hereinafter, description will be made with reference to the flowchart shown in FIG.

The matching result vectors Vmi corresponding to a plurality of correction operations on the representative pattern Di of a certain target region extraction dictionary 3 are classified in advance into k classes that are statistically likely, and each class has its representative vector Vmi. A vector pattern Vmk is obtained (steps S41 to S42). Here, Vmi is a vector element using a set of the similarity in the above-described target area extraction or the area extracted as the area having the maximum distance from the representative pattern and the similarity or distance at that time. For class classification, a K-means algorithm generally known as a clustering method (Ref. 9: Takagi and Shimoda, “Image Analysis Handbook”, PP.65)
0. ) Can be used.

The correction operation parameter Vpki corresponding to the matching result vector Vmki in the class k closest to each of the representative vector patterns Vmk is obtained (step S43), and the representative pattern number Di of the target area extraction dictionary 3 and the matching result representative The vector Vmk and the correction operation parameter vector Vpki are registered in the imaging adjustment parameter dictionary 16 in association with each other (Step S).
44).

The distance between Vmk and Vmki is Euclidean distance or Mahalanobis distance (see Reference 10:
Takagi, Shimoda, "Image Analysis Handbook" PP. 65
4. ) May be used.

The automatic imaging adjustment section 17 obtains the imaging adjustment parameters as described below using the imaging adjustment parameter dictionary 16 and automatically adjusts the imaging parameters. Less than,
The imaging adjustment processing operation of the automatic imaging adjustment unit 17 will be described with reference to the flowchart shown in FIG.

The target area extraction dictionary 3 is selected by imaging,
Among them, the number Di of the representative pattern currently used is
Is acquired (step S51).

A matching result vector Vmt is obtained from the current target region extraction result (step S52), and a matching result representative vector pattern Vmj of k classes corresponding to the representative pattern number Di of the region extraction dictionary in the imaging adjustment parameter dictionary 16 is obtained. The distance between (j = 1,... K) and Vmt is determined, and the class j with the minimum distance is determined (step S5).
3 to Step S54).

The imaging adjustment parameter vector Vpj corresponding to the class j is obtained from the dictionary corresponding to the representative pattern number Di of the area extraction dictionary in the imaging adjustment parameter dictionary 16 (step S55).

An adjustment operation of the imaging system is automatically performed according to the imaging adjustment parameter vector Vpj obtained in step S55 (step S55), and the resulting image is displayed on the display unit 1.
0 is presented to the user (step S56), and when the user's consent is obtained from the user operation unit 11 with respect to the presentation result, the adjustment operation result is set as the target area extraction result (step S58).

As described above, the extraction result of the target area extraction unit 4 is displayed on the display unit 10, and if the extraction result is not the one desired by the user, the user manually adjusts the imaging parameters of the imaging unit 2, A user operation unit 11 that enables selection of a desired region is provided, and the user's adjustment operation amount, the matching result at that time in the target region extraction unit 4, and the target region extraction dictionary used at the time of matching are stored in association with each other. In addition, the automatic imaging adjustment unit 17 automatically adjusts the stored adjustment operation when a similar result is obtained based on a certain evaluation with respect to the matching result using the stored dictionary during another imaging. The result is presented to the user, and when the user's consent is obtained with respect to the presentation result, the result of the adjustment operation is set as the target area extraction result.

In a case where the user performs an adjustment operation in a plurality of cases, the user's adjustment operation amount is associated with the matching result of each adjustment operation in the target area extraction unit 4 and the dictionary used for the matching. The matching results corresponding to a plurality of correction operations on a certain matching dictionary are classified in advance into statistically plausible numbers of classes, and the target area extracting unit 4 is used for new imaging. Finding the closest representative value of the stored result class based on the evaluation criteria in the matching result at that point in time, presenting to the user the result of performing the adjustment operation stored in association with it,
When the user's consent is obtained for the result, the result of the adjustment operation is set as the target area extraction result.

(8) Geometric Transformation Unit 7 The geometric transformation unit 7 converts a region determined to be a desired target region candidate by the target region extraction unit 4 into a standard desired target region previously set in the target identification dictionary 8. On the other hand, the degree of the geometric deformation is approximated by a geometric transformation in a two-dimensional plane.

The general form of the geometric transformation is a projective transformation, and the mapping of the input (XY w) to the output (X ^* Y ^* w ^* ) is represented by the following linear equation.

[0078]

(Equation 1) Here, the input is the circumscribed polygon coordinates of the standard desired target area,
The output is assumed to be the circumscribed polygon coordinates of the target area candidate extracted by the target area extraction unit 4, and the equations (1), (2), and (3) are converted into the following Affine assuming that the distortion in the two-dimensional plane is small.
Approximate by transformation.

[0079]

(Equation 2) Here, (x, y) are the coordinates of each vertex of the circumscribed rectangle of the standard desired target area, and (x ^* , y ^* ) are the coordinates of each vertex of the circumscribed rectangle of the area candidate extracted by the target area extraction unit 4. , It takes at least three vertices to solve this equation.

Next, when the coordinates are transformed according to the coefficients of the mapping of the equation (4) to obtain the respective pixels in the target extraction area corresponding to all the pixels in the standard desired target area, it is determined that the target extraction area The circumscribed shape is substantially equal to the standard desired target area.

The identification processing performed by the object identification section 9 described below is performed on the geometric inverse transformed image obtained as described above.

(9) Object Identification Unit 9 The object identification unit 9 identifies whether or not the input image is a desired object by performing a collation operation with a standard desired object pattern registered in the object identification dictionary 8 in advance.
In this case, identification is performed by performing a collation operation between the geometric inverse transform image obtained by the geometric transform unit 7 and the standard desired area pattern. The matching operation in this case is based on the subspace method (Ref.
Reference) etc. may be used.

The structure of the object identification dictionary 8 is such that if the subspace method (see Reference 1) is used for the collation operation, a plurality of sample patterns variously changed (optically, geometrically, etc.) with respect to a desired object are vectorized. And the eigenvectors of the results of principal component analysis of those variance-covariance matrices (or correlation matrices) are expressed in a matrix format in which the eigenvalues are arranged in descending order of eigenvalues. It has the coordinates of the surrounding circumscribed rectangle (or circumscribed polygon) as point sequence data.

Here, the point sequence data in the dictionary is used for calculating the geometric transformation parameters in the geometric transformation section 7, and the way of producing the data is a circumscribed rectangle (or circumscribed polygon) surrounding each sample pattern image. When the coordinate sequence of the vertices in ()) is expressed as vectors as follows, statistical representative vectors for all sample patterns of those vectors are obtained. However, there are various methods for calculating the representative vector. Here, an average, a principal component analysis, or the like may be used as described below. Assuming that the number of point sequences around each sample pattern is constant and n, and the number of samples is M, the point sequence vector Pi in the sample pattern i is Pi = (x1i, x2i,... Xni, y1i, y2i,
... Yni) (where i = 1,... M).

When the average Pa of the number of samples in each of these vectors is used as point sequence data to be registered in a dictionary

[0086]

(Equation 3) Indicated by

When the principal component analysis is used, all Pi
, An nxn dimensional variance-covariance matrix Mp is obtained, and an eigenvalue λj and an eigenvector Vi (where i =
1,... N), and the eigenvector corresponding to the largest eigenvalue is Vimax, and this Vimax is point sequence data to be registered in the dictionary. (10) Effects As described above, according to the above-described embodiment, a large amount of image data is stored because rough knowledge about the imaging situation and the imaging target is held in advance as a dictionary and images are determined while judging whether or not it is desired. The need to collect is reduced. Further, by adopting a configuration in which imaging, selection of a desired image area, and identification of an object can be performed only by an integrated device, the device can be used particularly for portable use. Further, since the imaging means and the target area extracting means are formed on the same LSI, only this part can be easily used for another purpose, such as a camera or a handy VCR which outputs only an image similar to a desired target. Can be diverted to Furthermore, if the desired target is not successfully extracted, it is possible to adjust the parameters so that the user can extract, and in that case, the adjustment parameters at the time of region extraction are learned, and when imaging at another opportunity, By using this, it is possible to automatically extract and identify a desired target automatically, so that it is possible to gradually endure fluctuations in imaging conditions and to reduce the frequency of maintenance of device performance.

[0088]

As described above, according to the present invention,
A desired imaging target can be reliably imaged while determining whether or not the imaging target is a desired one according to the imaging state of the imaging target.

[Brief description of the drawings]

FIG. 1 is a diagram showing a configuration example of an imaging apparatus according to an embodiment of the present apparatus.

FIG. 2 is a diagram showing a configuration example of a target area extraction dictionary.

FIG. 3 is a flowchart illustrating a processing operation of a target area extraction unit.

FIG. 4 is a flowchart for explaining an operation of creating a target area extraction dictionary in the dictionary creating unit.

FIG. 5 is a flowchart illustrating an operation of creating a shooting situation dictionary in the dictionary creating unit.

FIG. 6 is a flowchart illustrating a processing operation of an imaging situation selection unit.

FIG. 7 is a flowchart for explaining the operation of a dictionary creation unit for creating an imaging adjustment parameter dictionary.

FIG. 8 is a flowchart illustrating a processing operation of an automatic imaging adjustment unit.

[Explanation of symbols]

 DESCRIPTION OF SYMBOLS 1 ... Image input part 2 ... Imaging part 3 ... Target area extraction dictionary 4 ... Target area extraction part 5 ... Imaging situation dictionary 6 ... Imaging situation selection part 7 ... Geometric conversion part 8 ... Object identification dictionary 9 ... Object identification part 10 ... Display Unit 11: User operation unit 12: Storage unit 13: Data input / output unit 14: Dictionary creation unit 16: Imaging adjustment parameter dictionary 17: Automatic imaging adjustment unit

Claims (12)

[Claims] 1. An image capturing means for capturing an image, comprising: a plurality of search areas which include an image area of the object for searching for an image area of the object in the image and have a hierarchical inclusion relationship. The target area extraction dictionary in which each feature information is registered is compared with the feature information of the search area of each layer registered in the target area extraction dictionary and the image captured by the image capturing means. Target area extracting means for extracting an image area of the target object; a target identification dictionary in which feature information of the target object is registered; feature information registered in the target identification dictionary; and the target object extracted by the target area extracting means. Object identification means for comparing the image area with the object to identify the object, display means for displaying the image area of the object extracted by the object area extraction means and the identification result in the object identification means, Imaging device characterized by comprising: Place. 2. An image capturing means for capturing an image, and an image area of the object for searching for an image area of the object in the image according to an environmental condition at the time of capturing the image. A plurality of target area extraction dictionaries in which respective feature information of a plurality of search areas having a hierarchical inclusion relationship are registered; and one of the plurality of target area extraction dictionaries based on the image captured by the capturing means. And comparing the feature information of the search area of each hierarchy registered in the target area extraction dictionary selected by the selection means with the image captured by the image capturing means, and selecting the target Target area extracting means for extracting an image area of an object; a target identification dictionary in which feature information of the target is registered; feature information registered in the target identification dictionary; and the target extracted by the target area extracting means The image area of An image pickup apparatus comprising: a target identifying unit that identifies an object; and a display unit that displays an image area of the target extracted by the target region extracting unit and a result of identification performed by the target identifying unit. . 3. An image capturing means for capturing an image, and an image area of the object for retrieving an image area of the object in the image according to environmental conditions at the time of capturing the image. A plurality of target area extraction dictionaries in which respective feature information of a plurality of search areas having a hierarchical inclusion relationship are registered; and an environment in which feature information of the image corresponding to the environmental condition and an identifier of the target area extraction dictionary are registered. A condition dictionary, and selecting means for comparing characteristic information registered in the environmental condition dictionary with an image captured by the capturing means and selecting one of the plurality of target area extraction dictionaries; A target for extracting the image area of the target object by comparing the characteristic information of the search area of each hierarchy registered in the target area extraction dictionary selected by the selection means with the image captured by the image capturing means. Region extraction means, the object And a target identification dictionary for identifying the target by comparing the characteristic information registered in the target identification dictionary with the image area of the target extracted by the target region extracting means. Means for displaying an image area of the object extracted by the target area extracting means and a result of identification by the object identifying means. 4. The apparatus according to claim 1, wherein the target area extracting unit extracts the image area of the target object by narrowing the search area from a higher hierarchy to a lower hierarchy. An imaging device according to any one of the preceding claims. 5. A pixel-by-pixel correspondence between an inside of a circumscribed contour of an image area of the object designated as one of a plurality of images captured by the image capturing means and the plurality of images. A plurality of search areas in a hierarchical inclusion relationship including the image area of the target object, and registering the characteristic information of each of the plurality of search areas in accordance with the hierarchical inclusion relationship, 2. The imaging apparatus according to claim 1, further comprising a dictionary creating unit that creates an area extraction dictionary. 6. The luminance of each pixel between an inside of a circumscribed contour of an image area of the object designated as one of a plurality of images captured by the image capturing means and the plurality of images. Based on the corresponding relationship, a plurality of search areas that include the image area of the object and have a hierarchical inclusion relationship are obtained, and the characteristic information of each of the plurality of search areas is registered according to the hierarchical inclusion relationship. 2. The imaging apparatus according to claim 1, further comprising a dictionary creating unit that creates the target area extraction dictionary. 7. An image of the object designated as one of the plurality of captured images under each environmental condition by changing environmental conditions when the image is captured by the image capturing means. A plurality of search regions in a hierarchical inclusion relationship including the image region of the object are determined based on the pixel-by-pixel correspondence between the inside of the circumscribed contour of the region and the plurality of images. 4. The apparatus according to claim 2, further comprising: first dictionary creating means for registering feature information of each area according to the hierarchical inclusion relation and creating a target area extraction dictionary corresponding to the environmental condition. An imaging device according to any one of the preceding claims. 8. An image of the object designated as one of the plurality of captured images under each environmental condition by changing an environmental condition when the image is captured by the image capturing means. Based on the correspondence based on the brightness of each pixel between the inside of the circumscribed contour of the area and the plurality of images, a plurality of search areas in a hierarchical inclusion relationship including the image area of the object are obtained, A first dictionary creating means for registering feature information of each of a plurality of search areas according to the hierarchical inclusion relation and creating a target area extraction dictionary corresponding to the environmental condition. 4. The imaging device according to 2 or 3. 9. Based on a plurality of images captured under different environmental conditions, feature information of an image corresponding to each environmental condition is extracted. 4. The imaging apparatus according to claim 3, further comprising a second dictionary creating unit that creates the environmental condition dictionary by registering a corresponding identifier of the target area extraction dictionary. 10. An image of the object designated as one of the plurality of captured images under each environmental condition by changing an environmental condition when the image is captured by the image capturing means. A plurality of search regions in a hierarchical inclusion relationship including the image region of the object are determined based on the pixel-by-pixel correspondence between the inside of the circumscribed contour of the region and the plurality of images. First dictionary creating means for registering feature information of each area according to the hierarchical inclusion relationship and creating a target area extraction dictionary corresponding to the environmental condition; Image characteristic information of an image corresponding to each environmental condition is extracted from a plurality of images captured under different environmental conditions by image capturing means, and the extracted characteristic information of the image corresponding to each environmental condition and the corresponding characteristic information of the image are extracted. Knowledge of the target area extraction dictionary The imaging apparatus according 3, characterized by comprising a second dictionary creation means for creating said environmental condition dictionary and registers the child, the. 11. An image capturing parameter of the image capturing unit is adjusted based on an image area specified based on an image displayed on the display unit and an adjustment amount of the image capturing parameter of the image capturing unit. The imaging device according to any one of claims 1 to 3, further comprising an adjustment unit configured to perform the adjustment. 12. The image processing apparatus according to claim 1, wherein the target area extracting unit obtains an image of the search area by randomly accessing an image captured by the image capturing unit on a pixel-by-pixel basis. The imaging device according to any one of the above.