JP2001116985A - Camera with subject recognizing function and subject recognizing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To recognize a subject fast with high precision by processing image information of some necessary area although the whole picked-up image should be processed to recognize a subject in the picked-up image and a large quantity of calculations were needed. SOLUTION: This camera is equipped with a subject recognizing means which recognizes a subject, a gaze recognizing means, and an image cutting means which sets the recognition position and range of the subject recognizing means according to the detection result of the gaze detecting means, and the position of an object subject is specified. A range-finding means is provided which measure the detection position of the line of sight detecting means or a nearby subject distance and the image cutting means can vary its image cutting range according to the measurement result.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a camera for recognizing a subject from a photographed image and performing required processing.

[0002]

2. Description of the Related Art In general, a subject recognition technology for recognizing a specific object or person in a scene is known. As a method for extracting a human face from a captured image, a method is known in which skin color data is extracted from an original image, and a cluster of photometric points determined as a skin color range is used as a face. JP-A-52-156624, JP-A-53-14521, JP-A-53-14
No. 5622 and the like. Further, JP-A-4-3463
No. 33 discloses a method of converting a photometric data into hue and saturation, and creating and analyzing a two-dimensional histogram to determine a face area.

[0003] In Japanese Patent Application Laid-Open No. 8-063597, a face candidate area corresponding to the shape of a human face is extracted, and a face area is determined from the feature amount in the area. Another method extracts a human face outline from an image and determines a face area. In another method, a template having a plurality of face shapes is prepared, a correlation between the template and the image is calculated, and a human face is extracted by using the correlation value as a face candidate area. In still another method, learning can be repeated using a neural network to recognize an object or face of a subject. This is known as a model using neocognitron. (Fukushima: Neural network model of pattern recognition function not affected by displacement-Neocognitron, IEICE Transactions A,
J62-A (10), PP658-665, Oct. 1
979). For a method of learning a neural network, see “Study of Face Image Identification Using Neural Network (Technical Report of the Institute of Television Engineers of Japan, Vol. 14, No. 5)
0, 1990.9, 7-12) ".

As well-known techniques for face recognition, face detection, and eye detection, Japanese Patent Application Laid-Open Nos. Hei 9-251534 and Hei 10-232934 disclose detailed techniques and reference materials.

[0005]

In the above-mentioned prior art, in order to recognize a subject from a captured image, image processing must be performed on the entire captured image. Even with a computing device, a large amount of time was required.

The present invention has been made in such a situation, and an object of the present invention is to provide a camera having an object recognizing function and an object recognizing method capable of realizing object recognition with high speed and high accuracy. Things.

[0007]

In order to achieve the above object, according to the present invention, a camera having a subject recognizing function is configured as in the following (1) to (4), and a subject recognizing method is performed in the following (5). ), (6).

(1) A subject recognition means for recognizing a subject from a photographed image, a gaze detection means for detecting a gaze of a photographer, and a recognition position and range of the subject recognition means based on a detection result of the gaze detection means. A camera having a subject recognizing function provided with an image cutout means for setting.

(2) In the camera according to the above (1),
A camera comprising a distance measuring means for measuring a subject distance at or near the detection position of the eye-gaze detecting means, and wherein the image clipping means has a subject recognition function of changing a clipping range based on a distance measurement result of the distance measuring means.

(3) A subject recognizing means for recognizing a subject from a photographed image and a visual line detecting means for detecting a visual line of a photographer. When the subject recognizing means requires learning, the visual recognition is input to the learning input. A camera having a subject recognition function using a detection unit.

(4) In the camera according to any one of (1) to (3), the object recognition function is
Camera used for at least one of F, AE, zoom, and tracking functions.

(5) A method for recognizing a subject in a camera, comprising: a step A for detecting a line of sight of the photographer; a step B for cutting out a recognition range of the subject based on the detection result in the step A; A subject recognition method comprising: performing a subject recognition process in the cut-out range.

(6) A method for recognizing a subject in a camera, comprising: a step A for detecting a line of sight of a photographer; and a step B for performing learning in a subject recognition process using the detection result in step A. Recognition method.

[0014]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described below in detail with reference to embodiments of a single-lens reflex camera. The present invention is not limited to the film camera as in the embodiment, but can be similarly implemented in the form of a still video camera (digital camera) or a video camera. Further, the present invention can be implemented in the form of a subject recognition method supported by the description of the embodiment.

[0015]

FIG. 1 is a block diagram showing the configuration of a "single-lens reflex camera" according to a first embodiment.

In FIG. 1, reference numeral 101 denotes a photographing lens;
2 is a quick return mirror, 103 is a focus plate, 10
4 is a pentaprism, 105 is a light splitting prism, 106
Is an eyepiece, 107 is a distance measurement area sensor, 108 is a line-of-sight area sensor, 109 is a film surface, 110 is a shutter curtain, 111 is an image sensor of an area sensor, 112
Denotes an image cutout processing unit, 113 denotes a subject recognition unit, 114
Is an output unit, 115 is a distance detection unit, and 116 is a line-of-sight detection unit.

The subject image passes through the photographing lens 101, is bent by 90 degrees by the quick return mirror 102, and is primarily formed on the focus plate 103 by the focus plate 103. The focus plate 103 is composed of a mat and a field lens which are located on the primary image forming plane of the photographing lens 101 described above. The image on the focus plate 103 is changed in optical path for a finder by a pentaprism 104, separated into a distance measuring area sensor 107 and an eyepiece 106 via a light splitting prism 105, and observed by a photographer 117. An eye image of the observer 117 is formed on the line-of-sight area sensor 108 via the eyepiece 106 and the light splitting prism 105.

Two photographed images having different pupil positions, which will be described later, formed on the distance measuring area sensor 107 are captured by the image pickup processing unit 11.
1 and is stored as necessary in the image cutout processing unit 11
2. Supply necessary image data to the distance detection unit 115, respectively. The image cutout processing unit 112 selectively cuts out an image of a predetermined position and size from the captured image based on the distance information of the distance detection unit 115 and the line-of-sight position information of the line-of-sight detection unit 116, and outputs image data to the object recognition unit 113. The cutout position at this time is determined from the line-of-sight position information of the photographer in the field of view by the line-of-sight detection unit 116. In addition, the size of the image is determined based on the distance information of the distance detection unit 115 based on preset size information corresponding to the target subject to be predicted and calculated by recognizing the actual size of the subject. At this time, the detection position of the subject by the distance detection unit 115 is set to the position of the captured image or a position near the position based on the information from the line-of-sight detection unit 116. The subject recognition unit 113 recognizes a subject from the image from the image cutout processing unit 112 and outputs
14 is output. The subject recognizing unit 113 only needs to recognize and process only the designated image area based on the image information from the image clipping processing unit 112, and can perform high-speed processing by minimizing the calculation. On the other hand, the distance detection unit 115 measures the distance of the subject or a nearby subject based on the line-of-sight position information from the line-of-sight detection unit 116, and outputs the distance information to the image cutout processing unit 112. Further, the line-of-sight detection unit 116 detects the line-of-sight position from the eye image from the line-of-sight area sensor 108 and outputs the line-of-sight position information to the distance detection unit 115 and the image cutout processing unit 112.

When photographing on a film, the quick return mirror 102 and the shutter 110 are retracted from the photographing light beam, and the film 109 is exposed.

With the above-described series of configurations and operations, the subject recognition processing is an arithmetic processing of only necessary image information based on the area information selected by the line of sight, thereby enabling high-speed processing. In addition, it becomes possible to perform a complicated object recognition process that is more sophisticated than the speedup. Furthermore, since the selection is made based on the line of sight, the recognition can be performed with higher accuracy.

The surroundings of the distance measuring area sensor 107 and the line-of-sight area sensor 108 will be described in more detail with reference to FIG.

Reference numeral 105 denotes a light splitting prism having a semi-transmissive mirror portion 105a. The prism 105 partially transmits the finder light flux so that the photographer 117 can see it,
An action of guiding the remaining light flux to the distance measurement area sensor 107;
It also has the function of guiding the eyeball image of the photographer to the line-of-sight area sensor 108. 202 is a mirror for optical path conversion, 2
Numeral 03 denotes a two-dimensional image forming lens for performing a distance measurement using a phase difference method, and forms an image of an imaged object field on a distance measurement area sensor 107. Reference numerals 204a and 204b denote illuminations of the photographer's eyeball 117, which is arranged near the eyepiece lens 106. Reference numeral 201 denotes a line-of-sight area sensor 108
Is an image forming lens for forming an image on the surface. The eye image of the photographer 117 is included in the eyepiece 10 together with the illuminations 204a and 204b.
6, the semi-transparent mirror 10 in the light splitting prism 105
The light is reflected by 5a, is imaged by the imaging lens 201, and is imaged on the line-of-sight area sensor 108.

The distance measuring operation will be further described with reference to FIG. FIG. 3 corresponds to a summary diagram of the optical system related to the distance detection in FIGS. The image primarily formed on the matte surface of the focus plate 103 is re-imaged on the distance measuring area sensor 107 107a and 107b by the field lens and the secondary image forming lens 203 of the focus plate 103, respectively. At this time, the luminous flux is guided to the apertures 107a and 107b from different pupil positions by the aperture plate 301. With this configuration, imaging screens 107a and 107b having predetermined parallax can be obtained. Each of the imaging screens having this parallax is m × n
When the block is divided into a number of blocks (m = n may be used) and a known correlation operation is performed on the signals in the blocks facing each other, the distance to the object in the previous block and the defocus amount are measured by the principle of triangulation. I can do things. The relationship between the distance and the defocus is a non-linear relationship due to the focal length of the lens, the focal position, and characteristics specific to the lens. Means for correcting and converting these relationships as necessary are known.

The camera having this configuration is disclosed in detail in Japanese Patent Application No. 5-278433.

Next, the photographer looking into the viewfinder of the camera
A line-of-sight detection unit that allows the camera to recognize the viewfinder screen of the camera, that is, the position on the focus plate that the user is looking at will be described.

Conventionally, various devices (for example, eye cameras) for detecting what position on the observation surface the observer is observing, that is, for detecting a so-called line of sight (a visual axis) have been provided. As a method of detecting the line of sight, for example,
In Japanese Patent Publication No. 36, a collimated light beam from a light source is projected to the anterior segment of an eyeball of an observer, and a visual axis is obtained by using a corneal reflection image formed by light reflected from the cornea and an imaging position of a pupil.

Next, further explanation will be given using a flowchart.

FIG. 4 is a flowchart showing the operation. First, in step 401 (abbreviated as S401 in FIG. 4, the same applies hereinafter), the line of sight of the photographer is detected, and the line of sight of the object scene is determined, and the process proceeds to step 402. In step 402, an object scene image in a peripheral portion thereof is captured based on the line-of-sight position information obtained in step 401, and the process proceeds to step 403. In step 403, subject recognition processing is performed from the scene image captured in step 402 to detect a subject, and the process proceeds to step 404. In step 404, the subject detection result obtained in step 403 is output.

With the above series of flows, it is possible to efficiently recognize the subject at the line of sight.

FIG. 5 illustrates the processing content of step 402 in FIG. 4 in more detail. In step 501, distance measurement is performed at the line-of-sight position based on the line-of-sight position information of the photographer obtained in step 401 in FIG. This distance measuring method may be the conventional method described above. Step 502
In, the photographing magnification information at that distance is obtained from the distance information of the line-of-sight position based on various conditions such as the photographing lens and the area sensor by a known method, and the process proceeds to step 503. In step 503, an image range corresponding to the actual size of the object to be recognized is obtained based on the photographing magnification information in step 502, and the process proceeds to step 504. At this time, it is even more preferable that the actual dimension information is prepared in advance, and the relationship between the dimension information and the image recognition processing range is also set in advance. Thus, by setting the processing range to be slightly larger, it is possible to provide a function of preventing a cutting error due to a detection error of the line-of-sight position. Step 50
In step 4, an object scene image to be recognized is captured based on the processing range information of the image set in step 503, and an object recognition station is performed in step 403 in FIG. By the above flow, the object recognition can be satisfactorily realized from the scene image at the line of sight.

An image of the operation will be further described with reference to FIG. In the three figures on the left side of FIG. 6, the position where the line of sight is detected in the captured image is indicated by a frame, and the distance measurement data at that position is indicated by a numeral. In the three figures on the right side, the subject recognition processing ranges obtained based on the respective line-of-sight positions and distance information are shown by frames. In Casel, the distance measurement result of 5 m is selected when the deepest person in the center is selected, Case 2 is the measurement result of 4 m when the left person is selected, and Case 3 is the measurement result of 3 m when the right person is selected. The distance results are shown. Since the actual distance of the subject is different in each case, the image range for the recognition process is also different. Naturally, the closer the selection, the larger the selection range, and the farther the range, the smaller the selection range.

The output result of the object recognition can be applied to various functions such as AF (automatic focus adjustment), AE (automatic exposure adjustment), zooming, tracking, and the like, so that higher-quality shooting can be performed more easily.

(Embodiment 2) The operation of the "camera" of Embodiment 2 will be described with reference to a flowchart.

This embodiment describes an example in which a line of sight is used as an input for learning, for example, in the case of a neuro or the like that requires learning of an object recognition algorithm.

FIG. 7 is a flowchart showing the operation. In step 701, the object scene to be recognized is fetched and the process proceeds to step 702. In step 702, step 70
Step 7: subject the image captured in step 1 to subject recognition processing
Go to 03. In step 703, the result of the recognition processing in step 702 is determined. If OK, the process ends, and if NG, the process proceeds to step 704. At this time, OK and NG may be automatically determined by the recognition processing itself, or may be operated by a photographer or the like. In step 704, the subject recognition processing is set to the learning mode, and the flow advances to step 705. In step 705, the correct answer of the subject recognition processing result is input using the line of sight, and the flow advances to step 706. At this time, the input method using the line of sight may be to input the correct answer while looking at the captured image, or to input the symbol, the correct answer position, or the like. In step 706, learning processing of the object recognition processing is performed based on the correct answer input in step 705, and the recognition accuracy is improved and the processing is terminated.

According to the above flow, it is possible to easily input the learning of the object recognition processing, and it is possible to perform the recognition with higher precision by repeating the input.

[0037]

As described above, according to the present invention,
By using the line-of-sight input, subject recognition can be performed at high speed and with high accuracy, and advanced processing using the result of subject recognition processing is facilitated. As a result, a high-quality image can be automatically and easily captured.

[Brief description of the drawings]

FIG. 1 is a block diagram illustrating a configuration of a first embodiment.

FIG. 2 is a detailed view around a distance measuring area sensor and a line-of-sight area sensor.

FIG. 3 is an explanatory view of a principle of distance measurement.

FIG. 4 is a flowchart showing the operation of the first embodiment.

FIG. 5 is a flowchart showing details of S402.

FIG. 6 is a diagram showing an operation image.

FIG. 7 is a flowchart showing the operation of the second embodiment.

[Explanation of symbols]

 111 imaging processing unit 112 image cutout processing unit 113 subject recognition unit 115 distance detection unit 116 gaze detection unit

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 5/232 H04N 7/18 K 5C023 5/235 G02B 7/11 N 5C054 5/262 G03B 3/00 A 7/18 G06F 15/62 380 F term (reference) 2H002 DB29 DB30 FB31 GA63 HA04 2H011 AA01 AA03 CA01 DA00 DA01 2H051 BA02 DA03 DA04 DA15 DA24 DA25 EB04 EB13 EB20 5B057 AA20 BA02 BA19 CA08 CA12 CB08 DB09 CC08 AA13 AB23 AB28 AB34 AB36 AB62 AB63 AC03 AC12 AC42 AC69 CA00 5C023 AA06 AA18 AA37 BA02 CA06 DA08 EA03 5C054 AA05 CA04 CC05 CH04 EA01 FC15 FF02 HA17 HA31

Claims (6)

[Claims] 1. A subject recognizing means for recognizing a subject from a photographed image, a gaze detecting means for detecting a gaze of a photographer, and a recognition position and a range of the subject recognizing means are set based on a detection result of the gaze detecting means. A camera having a subject recognition function, comprising: 2. A camera according to claim 1, further comprising a distance measuring means for measuring a distance of a subject at or near a position detected by said line-of-sight detecting means, wherein said image clipping means outputs the distance measurement result of said distance measuring means. A camera having a subject recognizing function characterized by changing a clipping range based on the subject. 3. A subject recognizing means for recognizing a subject from a photographed image, and a gaze detecting means for detecting a gaze of a photographer. When the subject recognizing means requires learning, the gaze detection is applied to the learning input. A camera having a subject recognizing function, characterized by using means. 4. The camera according to claim 1, wherein an object recognition function is set to AF, AE,
A camera characterized in that it is used for at least one of zoom and tracking functions. 5. A method for recognizing a subject in a camera, comprising: a step A for detecting a line of sight of a photographer; a step B for cutting out a recognition range of the subject based on the detection result in the step A; And a step C of performing a subject recognition process in the range. 6. A method for recognizing a subject in a camera, comprising: a step A for detecting a line of sight of a photographer; and a step B for performing learning in a subject recognition process using the detection result in the step A. Characteristic object recognition method.